ISO/INSPIRE Metadata Support: Difference between revisions

From GRASS-Wiki
Jump to navigation Jump to search
No edit summary
m (→‎Requirements and installation: new dependency discovered)
(44 intermediate revisions by 5 users not shown)
Line 1: Line 1:
{{toModify}}
 


== ISO 19115:2003- Geographic information -- Metadata ==
== ISO 19115:2003- Geographic information -- Metadata ==
ISO 19115:2003(ISO) defines the scheme required for description of geographic information and services. It provides information about the identification, the extent, the quality, the spatial and temporal scheme, spatial reference, and distribution of digital geographic data.[http://www.iso.org/iso/catalogue_detail.htm?csnumber=26020 more]
[http://www.iso.org/iso/catalogue_detail.htm?csnumber=26020 ISO 19115:2003] defines the scheme required for description of geographic information and services. It provides information about the identification, the extent, the quality, the spatial and temporal scheme, spatial reference, and distribution of digital geographic data.
 
== Catalogue Service for the Web ==
[http://www.opengeospatial.org/standards/cat OGC Catalogue Services] support the ability to publish and search collections of descriptive information (metadata) for data, services, and related information objects. Metadata in catalogues represent resource characteristics that can be queried and presented for evaluation and further processing by both humans and software. Catalogue services are required to support the discovery and binding to registered information resources within an information community.  
 
= About =
Wx.metadata which was developed during Google Summer of Code 2014/2015 is currently available in GRASS 7 add-ons.
* The package for creating and editing metadata according to EN ISO 19115, described here, consists of few modules. The modules r.info.iso, v.info.iso and t.info.iso are command line based and support the conversion from the current native metadata management (r.info, v.info,t.info) into ISO based metadata. The GUI based module - g.gui.metadata offers advanced tools for management of metadata.


== About wx.metadata ==
New metadata management is currently available in GRASS add-ons. Package for creating and editing metadata according to EN ISO 19115, described here, consists of tree modules. Modules r.info.iso and v.info.iso are command line based and support conversion  from current native metadata management (r.info, v.info) into ISO based metadata. GUI based module - g.gui.metadata offers advanced tools for management of metadata.
[[Image:Wxmetadata-editing.png|center|600px]]
[[Image:Wxmetadata-editing.png|center|600px]]
== Dependencies and installation ==
=== Dependencies ===
Current development of metadata management has dependence on external libraries:


* OWSLib development version ()
* The package also includes the module g.gui.cswbrowser for searching and browsing metadata catalog (csw). For advanced users there are a few modules for establishing and handling pycsw server:
<pre>
* db.csw.admin - module for administration of pycsw server
* db.csw.run - module for starting server on localhost
* db.csw.harvest - module for harvesting metadata between two csw.
 
= Requirements and installation =
* [http://grass.osgeo.org/download/software/#g71x GRASS GIS 7.1 ]
wx.metadata package has dependence on external libraries:
 
* [http://pycsw.org/ pycsw] package - development version(OGC CSW server implementation written in Python):
<source lang="bash">
git clone https://github.com/geopython/pycsw.git && cd pycsw
sudo pip install -e . && sudo pip install -r requirements-standalone.txt && cd ..
</source>
 
* [https://geopython.github.io/OWSLib/ OWSLib] package - development version (OGC CSW client implementation written in Python):
<source lang="bash">
git clone git://github.com/geopython/OWSLib.git
git clone git://github.com/geopython/OWSLib.git
cd OWSLib && sudo python setup.py install
cd OWSLib && sudo python setup.py install && cd ..
</pre>
</source>
 
* [http://jinja.pocoo.org/docs/dev/ Jinja2] (one of the most used template engines for Python)
<source lang="bash">
git clone https://github.com/mitsuhiko/jinja2.git
cd jinja2 && sudo python setup.py install && cd ..
</source>
 
* [http://pygments.org/ Pygments] for highlighting text(for Ubuntu below)
<source lang="bash">
sudo apt-get install python-pygments
</source>
* Additionally, python-reportlab is required. The name of the package could vary for your distro.
 
To install, when in GRASS, type
<source lang="bash">
g.extension wx.metadata
</source>
 
= Creating and editing metadata =
== Naming of metadata files and storage ==
The default location for exported metadata files is ''metadata'' directory in the map's mapset. For raster maps, the prefix derived from the current nomenclature is ''cell'', for vector maps ''vector''. File ends with .xml suffix.


* Jinja templates
For example, the default metadata file name for vector map "roads" is ''vector_roads.xml''.
<pre>
git clone git://github.com/mitsuhiko/jinja2.git
cd jinja2 && sudo python setup.py install
</pre>


=== Installation of metadata management ===
== Metadata profile ==
Currently the wx.metadata package is available in GRASS Add-ons.
The metadata profile defines attributes, which can be edited in this editor. The editor offers the already mentioned two profiles (Basic and [http://inspire.ec.europa.eu/index.cfm INSPIRE]). The editor also supports loading of customized profiles. See Creation of custom ISO based profile.
<pre>
GRASS> g.extension wx.metadata
</pre>


== Modules and usage ==
<big>Currently there are three build-in ISO profiles available: </big>
=== Naming of metadata files and storage ===
* '''The basic profile''' is substituted from intersection between items stored in GRASS native metadata format and INSPIRE profile. The intersect (subset) includes all available GRASS metadata. Metadata which cannot be assigned to ISO based attributes are stored in metadata attribute sabstractact.  
The metadata product fully supports XML Default location of exported metadata uses  folder “metadata” in  GRASS Location. In case the metadata file name is not changed, the name is build from prefix according to characteristic of maps. For raster maps prefix is derived from current nomenclature - 'cell' and for vector map – 'vector' by the same way. File name is derived from the  the name of the map. Postfix-extension is '.xml'.  


example: vector map - roads
* '''The INSPIRE profile''' fulfills the [http://inspire.ec.europa.eu/index.cfm/pageid/101 criteria of INSPIRE profile]. In case if is not possible to get the values from native GRASS metadata, these are filled with text string '$NULL'. This rule is applied for both profiles.
vector_roads.xml


=== Metadata profile ===
* '''The temporal profile''' is based on the GRASS basic profile and includes additional specific attributes for Temporal framework.
The metadata profile defines attributes, which can be edited in this editor.  The editor offers the already mentioned two profiles(Basic and Inspire). Editor also support loading of customized profiles. See Creation of custom ISO based profile.


== Modules r.info.iso and v.info.iso==  
== Modules r.info.iso, v.info.iso t.info.iso ==
Command line based modules implements  conversion of native GRASS metadata to ISO based XML metadata files. Identical configuration applies for both modules. Products of these modules differ partly in certain metadata attributes related  to differences between  raster (r.info) and vector (v.info) maps metadata.  
Command line based modules implements  conversion of native GRASS metadata to ISO based XML metadata files. Identical configuration applies for both modules. Products of these modules differ partly in certain metadata attributes related  to differences between  raster (r.info) and vector (v.info) maps metadata.  
* Currently are available two build-in ISO profile:
'''The Basic profile''' is substituted from intersection between GRASS native metadata and INSPIRE profile. The intersect (subset) includes all available GRASS metadata. Metadata which cannot be  assigned to ISO based attributes are stored in metadata attribute - abstract.
'''The Inspire profile''' fulfills the criteria of [http://inspire.ec.europa.eu/documents/Metadata/INSPIRE_MD_IR_and_ISO_v1_2_20100616.pdf INSPIRE] profile.  Values which are not able to get from native GRASS metadata are filled by text string '$NULL'. This rule applies for both profiles.


'''Parameters of modules:'''
'''Parameters of modules:'''
Line 51: Line 75:
* output - path to output XML metadata file.  Default = according to naming&storage rules
* output - path to output XML metadata file.  Default = according to naming&storage rules
== Module g.gui.metadata  ==
== Metadata editor- g.gui.metadata  ==
Metadata editor (editor) includes graphical interface for conversion metadata from r.info and v.info to ISO based metadata, editing metadata files and creating metadata templates. In addition, the graphical module allows validating of  INSPIRE and Basic profile.
The metadata editor (editor) includes a graphical interface for converting metadata from r.info and v.info to ISO based metadata, editing metadata files and creating metadata templates. In addition, the graphical module allows validating of  INSPIRE and Basic profile.
 
=== Information about metadata item ===
Tooltip function helps to fill in the form correctly.
[[Image:Wxmetadata-tooltip.png|center|350]]


=== Creation of metadata ===
=== Creation of metadata ===
Editor offers two editing modes. The first one allows to create metadata from selected GRASS maps from an active location. The second one allows editing of external metadata file.
The editor offers two editing modes. The first one allows to create metadata from selected GRASS maps from an active location. The second one allows editing of external metadata file.  
 
=== Fetching values from database backend ===
The editor automatically fetches keywords from SQLite GRASS GIS backend. Three well know dictionaries are added by default. The user can manually add dictionaries by sql commands.
Dictionaries are stored in the table ''metadata_themes''
 
 
{| border="1" class="wikitable"
|+ teble metadta_themes
! Attribut
| title
| keyword
| date_iso
| date_type
|-
! Date type
| TEXT || TEXT|| TEXT|| TEXT
|-
! Example
|Example || GEMET - Concepts, version || 2010-01-13 || publication
 
|}
 
[[Image:Keywords.png|center|350]]


==== GRASS metadata map editor ====
=== Exporting metadata reports to format PDF ===
This function allows to export metadata of GRASS maps to report in pdf format. The report template is based on INSPIRE profile. Unknown values of metadata attributes are filled by text $UNKNOWN.
[[Image:Pdfexp.png|center|350]]
 
{{YouTube|V7tgOOuRyuA}}
 
 
=== GRASS metadata map editor ===
[[Image:Wxmetadata-mapEdit.png|center|150px]]
[[Image:Wxmetadata-mapEdit.png|center|150px]]


Line 63: Line 121:
* basic
* basic
* inspire
* inspire
* load custom – this option allows to load metadata profile from file. For example in case of use of predefined template. With this selection the editor make intersection of GRASS Basic profile(all available metadata for selected map) and loaded custom profile.
* load custom – allows to load metadata profile from file. For example in case of using predefined template. The intersection of GRASS Basic profile(all available metadata for selected map) and loaded custom profile will proceed with this selection.


{{YouTube|7V5NyVdO05o}}
{{YouTube|7V5NyVdO05o}}


==== External metadata editor ====
=== External metadata editor ===
[[Image:Wxmetadata-externalEdit.png|center|150]]
[[Image:Wxmetadata-externalEdit.png|center|150]]
In this case to load metadata profile file and metadata file is necessary for editing. This option does not support any connection with GRASS maps metadata. Of course, it is possible to load exported metadata from the past.
This editor mode allows to load own template and xml file for editing. This option does not support any connection with GRASS maps metadata.


{{YouTube|npMAavbwfi0}}
{{YouTube|npMAavbwfi0}}


=== Defining templates ===
=== Defining templates ===
This function allows to create pattern of metadata profile. In other words it support defining fixed values to template. With using created templates, the editor does not initialize texts inputs of defined values. These attributes is able to see only in Tree Browser. Direct link on this function is multiple editing mode (see below).  
This function allows to create pattern of metadata profile. In other words it support defining fixed values to template. With using created template, the editor does not initialize texts inputs of defined values. These attributes is able to see only in Tree Browser in editor. Direct link on this function is multiple editing mode (see below).
* If check-box of metadata attribute is checked, the OWSLib object in template will be replaced by value. Unchecked attributes will not change this 'part' of the template.
* The yellow background of text fields indicates metadata attributes, which are not possible get from GRASS map information.
[[Image:Wxmetadata-template.png|center|600px]]
[[Image:Wxmetadata-template.png|center|600px]]


==== Usage ====
=== Usage ===
* If check-box of metadata attribute is checked, the OWSLib object in template will be replaced by value. Unchecked attributes will not change this 'part' of the template.
* If the check-box of metadata attribute is checked, the OWSLib object in template will be replaced by value. Unchecked attributes will not change this 'part' of the template.
example: Result of checked attribute
 
example: Template result of attribute which is checked  
<pre>
<pre>
     <gmd:dateStamp>
     <gmd:dateStamp>
Line 86: Line 147:
</pre>
</pre>


example: Result of unchecked attribute
example: Template result of attribute which is unchecked  
<pre>
<pre>
     <gmd:dateStamp>
     <gmd:dateStamp>
Line 95: Line 156:
</pre>
</pre>


* The yellow background of text fields indicates metadata attributes, which are not possible parsed from GRASS map information.
Video shows record of simple practice of defining custom template
Video shows record of simple practice of defining own template
{{YouTube|Pu2sQoAJXfo}}
{{YouTube|Pu2sQoAJXfo}}


=== Editing metadata of multiple map selection ===
=== Editing metadata of multiple map selection ===
Advantages of this editing mode is in work using of defined template. This option can be used by selection of multiple maps in data catalog.   
The advantages of this editing mode is in work using of defined template. This option can be used by selection of multiple maps in data catalog.   


=== Validation of metadata profile ===
=== Validation of metadata profile ===
On the right side of editor there is a validator of metadata. Currently, validator is able to validate two build-in profiles. For selection of profile “Load custom” is set by default in the inspire profile.
On the right side of editor is a validator of metadata. Currently, validator is able to validate two build-in profiles. For selection of profile “Load custom” is set by default the inspire profile.


=== Updating GRASS native metadata ===
=== Updating GRASS native metadata ===
The g.gui.metadata module allows to update native GRASS metadata with using modules r.support and v.support. The intersection between ISO based metadata and r|v.support is limited by available parameters for these modules.
The g.gui.metadata module allows updating native GRASS metadata with using modules r.support and v.support. The intersection between ISO based metadata and r|v.support is limited by available parameters for these modules.
 
 
=== Publishing to csw server ===
Editor allows direct publishing of edited metadata by CSW(Catalog Service for Web) standard to remote catalogue server. Note that the remote catalogue must allows transaction.
[[File:Csw publisher.png|650px|center|publishing of metadata directly to remote csw server]]


for vector maps(v.support):
=== Notes ===
* organization =
* EPSG code must by added in url format. Information of projection is added only if EPSG code is known or if GDAL function for identifying from WKT is succesful.
<source lang="xml">
<gmd:referenceSystemInfo>
  <gmd:MD_ReferenceSystem>
    <gmd:referenceSystemIdentifier>
      <gmd:RS_Identifier>
        <gmd:code>
          <gco:CharacterString>http://www.opengis.net/def/crs/EPSG/0/3035</gco:CharacterString>
        </gmd:code>
        <gmd:codeSpace>
          <gco:CharacterString>INSPIRE RS registry</gco:CharacterString>
        </gmd:codeSpace>
      </gmd:RS_Identifier>
    </gmd:referenceSystemIdentifier>
  </gmd:MD_ReferenceSystem>
</gmd:referenceSystemInfo>
</source>


== Creation of custom ISO based profile ==
= Creation of custom ISO based profile =
Creation of own custom ISO based profile is with using  Jinja template syntax and some addition rules.
Creation of own custom ISO based profile is with using  Jinja template syntax and some addition rules.
The main item of template is composite from two braces.
The main item of template is composite from two braces.
Both of them are fully according to syntax of Jinja template. First holds information about statements of loop or condition and also simple Python object.  Second braces according to Jinja template syntax is represented by comment {# #}. In this braces additional information for generating graphical interface of editor are stored. In addition also  information about individual metadata attributes (name, example, description, multiplicity, etc) is included.
Both of them are fully according to syntax of Jinja template. First holds information about statements of loop or condition and also simple Python object.  Second braces according to Jinja template syntax is represented by comment {# #}. In this braces additional information for generating graphical interface of editor are stored. In addition also  information about individual metadata attributes (name, example, description, multiplicity, etc) is included.


=== The first braces ===
== The first braces ==
Contain object of OWSLib which holds the metadata value. This values can be object which is  describing  single variable or list of variables.  Value of some attribute can be defined many times(according to ISO). This attributes must be defined in block of template with starting statements and ending mark of block. Only allowed loop statement is FOR(see jinja2 manual). The XML based text of block will by duplicated according to number of items in the list.
Contain object of OWSLib which holds the metadata value. This values can be object which is  describing  single variable or list of variables.  Value of some attribute can be defined many times(according to ISO). This attributes must be defined in block of template with starting statements and ending mark of block. Only allowed loop statement is FOR(see jinja2 manual). The XML based text of block will by duplicated according to number of items in the list.


==== Initialize of OWSLib objects ====
== Initialize of OWSLib objects ==
Most object are initialized in configuration of file which is stored in wx.metadat/config/init_md.txt
Most object are initialized in configuration of file which is stored in wx.metadat/config/init_md.txt
example: Default configuration file for the build-in Basic and Inspire profile:
example: Default configuration file for the build-in Basic and Inspire profile:
Line 131: Line 212:
# on the left side of equation is obligatory prefix 'self.md'. Postfix(name of variable) is open. On right side is  OWSLib object
# on the left side of equation is obligatory prefix 'self.md'. Postfix(name of variable) is open. On right side is  OWSLib object


==== Single element ====
== Single element ==
In case if the value of metadata element is  declared only once:  
In case if the value of metadata element is  declared only once:  
example: Identifier
example: Identifier
Line 143: Line 224:
</pre>
</pre>


==== Loop or Condition statements ====
== Loop or Condition statements ==
Loops and condition are defined in by starting braces  and ending braces '{%  -%}' .
Loops and condition are defined in by starting braces  and ending braces '{%  -%}' .
Start of block:
Start of block:
Line 175: Line 256:
{% endfor -%}
{% endfor -%}
</pre>  
</pre>  
=== The second braces ===
== The second braces ==
The second braces(by Jinja syntax comment) contains these parameters:
The second braces(by Jinja syntax comment) contains these parameters:
* tag - just copy of string from first braces
* tag - just copy of string from first braces
Line 209: Line 290:
  type ="string" , group= "Quality and Validity" ,  multiline= True #}
  type ="string" , group= "Quality and Validity" ,  multiline= True #}
</pre>
</pre>
Video tutorial shows practise of creation custom ISO profile.
{{YouTube|ahl41mFD5w0}}


== Syntax of Jinja template system and editor custom parser ==
== Syntax of Jinja template system and editor custom parser ==
Line 217: Line 295:
* after '{# ','{%' must be white space. Also before '#}' '-%}'
* after '{# ','{%' must be white space. Also before '#}' '-%}'


== Examples of template ==
[http://trac.osgeo.org/grass/browser/grass-addons/grass7/gui/wxpython/wx.metadata/templates?order=date&desc=1 INSPIRE and Basic]
Video tutorial shows practice of creation custom ISO profile.
{{YouTube|ahl41mFD5w0}}
= Tools for working with metadata catalogue (csw) =
== Browsing and searching- g.gui.cswbrowser ==
Module g.gui.cswbrowser(Browser) based graphical interface allows managing csw connections, setting filter and searching, browsing results and adding services to GRASS GIS.
=== Connection manager ===
Connection manager allows:
* adding
* removing
* loading list of connections from file.
By default the Browser is initialized by list of few well known csw providers of csw. Button "Service info" shows metadata of the service.
User can manage connection in two ways:
* First one is with using hidden storage provided by [http://wxpython.org/Phoenix/docs/html/ConfigBase.html ​wx.Config]. This case is without accessing default connection file.
* After prompt user also can add/remove connection from default .xml file.
[[File:ConnectionMgr.png|center|600px|Connection manager in g.gui.cswbrowser]]
=== Searching ===
Before fetching data from selected csw server is proper to set up searching filter. Searching can be restricted by:
* DCMI Type (Collection , Dataset , Event , Image , InteractiveResource , MovingImage , PhysicalObject , Service , Software , Sound , StillImage , Text)
* Area(bounding box)
** current GRASS GIS computation region
** manually
* Keywords
** basic filter allows to use basic or advance keyword filtering. In the simple case user can define single keywords or multiple keywords with button "+". Logic operator between keywords is AND(&).
** second, advanced is based on OGC list of expressions which means that can be set filter with logic relations between keywords or sets of keywords. Dialogue for settings keywords text string is under "Advanced" check-box. Syntax of constraints is based on python list syntax. Each keywords must be in braces
[[File:Find1.png|center|600px|g.gui.cswbrowser, searching page]]
==== Advanced keywords filtering ====
On the left panel of Browser is placed check-box named "Advance". The keywords advanced dialogue allows to use basic bool operations.
*OR condition
    a || b || c
    ["a","b","c"]
*AND condition
    a && b && c
    [ ["a","b","c"] ]
*composition
    (a && b) || c || d || e
    [["a","b"],["c"],["d"],["e"]] or [["a","b"],"c","d","e"]
=== Browsing results===
On the bottom of Browser is a controller for browsing results. The data from server are fetched by number of items per one transaction. Default is a 20 items. After selection of item in the result table, metadata of record should be displayed. Browser automatically enables buttons on the top of Browser if WMS, WCS or WFS services are available.
=== Setting up csw server ===
== CSW server management ==
wx.metadata package includes tools for establishing and handling pycsw server. Pycsw stuff consist:
*db.csw.admin
*db.csw.run
*db.csw.harvest
For handling pycsw server can be use standard pythons scripts from pycsw install directory or mentioned three modules.
== Installation and configuration csw server ==
* Installation pycsw package - development version(OGC CSW server implementation written in Python):
e.g.to default path taken from configure file
<source lang="bash">
$ cd /var/www/pycsw
</source>
*deploying pycsw
**<source lang="bash">
$ git clone https://github.com/geopython/pycsw.git && cd pycsw
</source>
**<source lang="bash">
$ pip install -e . && pip install -r requirements-standalone.txt
</source>
* Create and adjust a configuration file:
**<source lang="bash">
$ cp default-sample.cfg default.cfg
</source>
* adjust parameters in ''default.cfg''
** '''server.home''' - path to installation directory
** '''repository.database''' - path to SQLite database e.g. $GISDBASE/WGS84/PERMANENT/sqlite/sqlite.db
** set '''server.url''' to: http://localhost:8000/
*Setup the database(creates tables, indexes, etc.)
**with unix command line(in pycsw install dir)
***<source lang="bash">
$ pycsw-admin.py -c load_records -f default.cfg -p /path/to/xml/
</source>
**with module db.csw.admin from GRASS GIS. Flag -s and parameter configure with path to configure file. Path to database is automatically taken from configure file.
***<source lang="bash">
db.csw.admin -s configure=/var/www/html/pycsw/default.cfg
</source>
*Load records:
**with unix command line(in pycsw install dir)
***<source lang="bash">
pycsw-admin.py -c load_records -f default.cfg -p /path/to/xml/
</source>
**db.csw.admin module with parameter -r for load recursively
***<source lang="bash">
db.csw.admin -r configure=/var/www/html/pycsw/default.cfg load_records=path/to/xml
</source>
*Run the server
**with unix command line(in pycsw install dir)
***<source lang="bash">
python csw.wsgi
</source>
**db.csw.admin module with parameter path and port
***<source lang="bash">
db.csw.run path=/var/www/html/pycsw port=8000
</source>


== Harvesting csw catalogue ==
Module db.csw.harvest allows to copy data from source csw server to destination csw server


== See also ==
= See also =
* [[GRASS Metadata Management]]
* [[GRASS Metadata Management]]
* [http://trac.osgeo.org/grass/wiki/GSoC/2014/MetadataForGRASS Development page]
* [http://trac.osgeo.org/grass/wiki/GSoC/2014/MetadataForGRASS Development page 2014]
* [https://trac.osgeo.org/grass/wiki/GSoC/2015/ImprovedMetadata Development page 2015]


[[Category:Development]]
[[Category:Development]]
[[Category:Metadata]]
[[Category:Metadata]]
[[Category:INSPIRE]]
[[Category:INSPIRE]]

Revision as of 10:33, 6 February 2016


ISO 19115:2003- Geographic information -- Metadata

ISO 19115:2003 defines the scheme required for description of geographic information and services. It provides information about the identification, the extent, the quality, the spatial and temporal scheme, spatial reference, and distribution of digital geographic data.

Catalogue Service for the Web

OGC Catalogue Services support the ability to publish and search collections of descriptive information (metadata) for data, services, and related information objects. Metadata in catalogues represent resource characteristics that can be queried and presented for evaluation and further processing by both humans and software. Catalogue services are required to support the discovery and binding to registered information resources within an information community.

About

Wx.metadata which was developed during Google Summer of Code 2014/2015 is currently available in GRASS 7 add-ons.

  • The package for creating and editing metadata according to EN ISO 19115, described here, consists of few modules. The modules r.info.iso, v.info.iso and t.info.iso are command line based and support the conversion from the current native metadata management (r.info, v.info,t.info) into ISO based metadata. The GUI based module - g.gui.metadata offers advanced tools for management of metadata.
  • The package also includes the module g.gui.cswbrowser for searching and browsing metadata catalog (csw). For advanced users there are a few modules for establishing and handling pycsw server:
* db.csw.admin - module for administration of pycsw server
* db.csw.run - module for starting server on localhost
* db.csw.harvest - module for harvesting metadata between two csw.

Requirements and installation

wx.metadata package has dependence on external libraries:

  • pycsw package - development version(OGC CSW server implementation written in Python):
git clone https://github.com/geopython/pycsw.git && cd pycsw
sudo pip install -e . && sudo pip install -r requirements-standalone.txt && cd ..
  • OWSLib package - development version (OGC CSW client implementation written in Python):
git clone git://github.com/geopython/OWSLib.git
cd OWSLib && sudo python setup.py install && cd ..
  • Jinja2 (one of the most used template engines for Python)
git clone https://github.com/mitsuhiko/jinja2.git
cd jinja2 && sudo python setup.py install && cd ..
  • Pygments for highlighting text(for Ubuntu below)
sudo apt-get install python-pygments
  • Additionally, python-reportlab is required. The name of the package could vary for your distro.

To install, when in GRASS, type

g.extension wx.metadata

Creating and editing metadata

Naming of metadata files and storage

The default location for exported metadata files is metadata directory in the map's mapset. For raster maps, the prefix derived from the current nomenclature is cell, for vector maps vector. File ends with .xml suffix.

For example, the default metadata file name for vector map "roads" is vector_roads.xml.

Metadata profile

The metadata profile defines attributes, which can be edited in this editor. The editor offers the already mentioned two profiles (Basic and INSPIRE). The editor also supports loading of customized profiles. See Creation of custom ISO based profile.

Currently there are three build-in ISO profiles available:

  • The basic profile is substituted from intersection between items stored in GRASS native metadata format and INSPIRE profile. The intersect (subset) includes all available GRASS metadata. Metadata which cannot be assigned to ISO based attributes are stored in metadata attribute sabstractact.
  • The INSPIRE profile fulfills the criteria of INSPIRE profile. In case if is not possible to get the values from native GRASS metadata, these are filled with text string '$NULL'. This rule is applied for both profiles.
  • The temporal profile is based on the GRASS basic profile and includes additional specific attributes for Temporal framework.

Modules r.info.iso, v.info.iso t.info.iso

Command line based modules implements conversion of native GRASS metadata to ISO based XML metadata files. Identical configuration applies for both modules. Products of these modules differ partly in certain metadata attributes related to differences between raster (r.info) and vector (v.info) maps metadata.

Parameters of modules:

  • map - selection of map for metadata creation
  • profile - selection of metadata profile (basic or inspire)
  • output - path to output XML metadata file. Default = according to naming&storage rules

Metadata editor- g.gui.metadata

The metadata editor (editor) includes a graphical interface for converting metadata from r.info and v.info to ISO based metadata, editing metadata files and creating metadata templates. In addition, the graphical module allows validating of INSPIRE and Basic profile.

Information about metadata item

Tooltip function helps to fill in the form correctly.

350
350

Creation of metadata

The editor offers two editing modes. The first one allows to create metadata from selected GRASS maps from an active location. The second one allows editing of external metadata file.

Fetching values from database backend

The editor automatically fetches keywords from SQLite GRASS GIS backend. Three well know dictionaries are added by default. The user can manually add dictionaries by sql commands. Dictionaries are stored in the table metadata_themes


teble metadta_themes
Attribut title keyword date_iso date_type
Date type TEXT TEXT TEXT TEXT
Example Example GEMET - Concepts, version 2010-01-13 publication
350
350

Exporting metadata reports to format PDF

This function allows to export metadata of GRASS maps to report in pdf format. The report template is based on INSPIRE profile. Unknown values of metadata attributes are filled by text $UNKNOWN.

350
350



GRASS metadata map editor

Combo box for selection of ISO profile:

  • basic
  • inspire
  • load custom – allows to load metadata profile from file. For example in case of using predefined template. The intersection of GRASS Basic profile(all available metadata for selected map) and loaded custom profile will proceed with this selection.


External metadata editor

150
150

This editor mode allows to load own template and xml file for editing. This option does not support any connection with GRASS maps metadata.


Defining templates

This function allows to create pattern of metadata profile. In other words it support defining fixed values to template. With using created template, the editor does not initialize texts inputs of defined values. These attributes is able to see only in Tree Browser in editor. Direct link on this function is multiple editing mode (see below).

  • If check-box of metadata attribute is checked, the OWSLib object in template will be replaced by value. Unchecked attributes will not change this 'part' of the template.
  • The yellow background of text fields indicates metadata attributes, which are not possible get from GRASS map information.

Usage

  • If the check-box of metadata attribute is checked, the OWSLib object in template will be replaced by value. Unchecked attributes will not change this 'part' of the template.

example: Template result of attribute which is checked

    <gmd:dateStamp>
        <gco:Date>2013-01-01</gco:Date>
    </gmd:dateStamp>

example: Template result of attribute which is unchecked

    <gmd:dateStamp>
        <gco:Date>{{ md.datestamp }}{# tag="md.datestamp" , name="Metadata date" ,\
 ref= "Part B 10.2" , desc= "Date that the metadata was created.",type = "date" ,
 example = "2005-03-27" , multi= 0 , group= "Metadata" , multiline= False #}</gco:Date>
    </gmd:dateStamp>

Video shows record of simple practice of defining custom template

Editing metadata of multiple map selection

The advantages of this editing mode is in work using of defined template. This option can be used by selection of multiple maps in data catalog.

Validation of metadata profile

On the right side of editor is a validator of metadata. Currently, validator is able to validate two build-in profiles. For selection of profile “Load custom” is set by default the inspire profile.

Updating GRASS native metadata

The g.gui.metadata module allows updating native GRASS metadata with using modules r.support and v.support. The intersection between ISO based metadata and r|v.support is limited by available parameters for these modules.


Publishing to csw server

Editor allows direct publishing of edited metadata by CSW(Catalog Service for Web) standard to remote catalogue server. Note that the remote catalogue must allows transaction.

publishing of metadata directly to remote csw server
publishing of metadata directly to remote csw server

Notes

  • EPSG code must by added in url format. Information of projection is added only if EPSG code is known or if GDAL function for identifying from WKT is succesful.
<gmd:referenceSystemInfo>
  <gmd:MD_ReferenceSystem>
    <gmd:referenceSystemIdentifier>
      <gmd:RS_Identifier>
        <gmd:code>
          <gco:CharacterString>http://www.opengis.net/def/crs/EPSG/0/3035</gco:CharacterString>
        </gmd:code>
        <gmd:codeSpace>
          <gco:CharacterString>INSPIRE RS registry</gco:CharacterString>
        </gmd:codeSpace>
      </gmd:RS_Identifier>
    </gmd:referenceSystemIdentifier>
  </gmd:MD_ReferenceSystem>
</gmd:referenceSystemInfo>

Creation of custom ISO based profile

Creation of own custom ISO based profile is with using Jinja template syntax and some addition rules. The main item of template is composite from two braces. Both of them are fully according to syntax of Jinja template. First holds information about statements of loop or condition and also simple Python object. Second braces according to Jinja template syntax is represented by comment {# #}. In this braces additional information for generating graphical interface of editor are stored. In addition also information about individual metadata attributes (name, example, description, multiplicity, etc) is included.

The first braces

Contain object of OWSLib which holds the metadata value. This values can be object which is describing single variable or list of variables. Value of some attribute can be defined many times(according to ISO). This attributes must be defined in block of template with starting statements and ending mark of block. Only allowed loop statement is FOR(see jinja2 manual). The XML based text of block will by duplicated according to number of items in the list.

Initialize of OWSLib objects

Most object are initialized in configuration of file which is stored in wx.metadat/config/init_md.txt example: Default configuration file for the build-in Basic and Inspire profile:

self.md.identification = MD_DataIdentification() 
self.md.dataquality = DQ_DataQuality() 
self.md.distribution = MD_Distribution() 
self.md.identification.extent = EX_Extent() 
self.md.identification.extent.boundingBox = EX_GeographicBoundingBox()
  1. on the left side of equation is obligatory prefix 'self.md'. Postfix(name of variable) is open. On right side is OWSLib object

Single element

In case if the value of metadata element is declared only once: example: Identifier

{{ md.identifier }}

example: Lineage

{{ md.dataquality.lineage }}

Loop or Condition statements

Loops and condition are defined in by starting braces and ending braces '{% -%}' . Start of block:

example: for statement

{% for co in md.contact -%}

Editor allows to use Python 'zip' function and 'len' function. Note- in Jinja syntax is Python len(length) function defined specifically.

example: zip

{% for (u,d) in zip(md.identification.uom,md.identification.distance) -%}

example: len

{% if md.identification.denominators|length > 0 -%}

End of block is defined by:

example: if statement

{% endif -%}

example: for statement

{% endfor -%}

The second braces

The second braces(by Jinja syntax comment) contains these parameters:

  • tag - just copy of string from first braces
  • object – for reading and holding metadata in python objects is used OWSLib library.

Most of object are initialized by configuration file wx.metadata/config/init_md.txt. Some metadata attributes are necessary initialized directly in this(object) parameter. Best example is the metadata contact object(organization,email) which can be initialized for many times.

example: responsible party

{% for co in md.contact -%}{# object="CI_ResponsibleParty()" #}
{{ co.organization }}{# object="CI_ResponsibleParty()" #}
{{ co.email }}{# object="CI_ResponsibleParty()" #}
{{ co.role }}{# object="CI_ResponsibleParty()" #}
{% endfor -%}
  • name – free text string describing the name of metadata item
  • ref – free additional information. By default should be reference of ISO attributes
  • type – data type characteristic. Input text field validator for 'email' and 'date' is implemented.
  • desc – free text which should describe or define the characteristic of metadata item
  • example – free text with example of the value for metadata item
  • group- label of the notebook list(tab) in editor.
  • inbox- label of the box which includes attributes from the block of Jinja template. All items from one block of Jinja template must be in same box(same name).
  • multi- 0 or 1. In case 0, the item cannot be duplicated= text field widget does not contain button for duplicating itself. Note that only OWSLib list based variables can be used with duplication button.
  • inboxmulti- 0 or 1. In case 0, the items in box cannot be duplicated = box with items not dispose with button for cloning self.
  • multiline – True = initialize text field with multilines editing. Suitable for longer texts input

example of second braces:

{# tag="md.dataquality.lineage" , name="Lineage" , ref= "Part B 6.1" ,/
 desc= "General explanation of the data producers knowledge about the lineage of a dataset."/
 ,example = "Product 1 scenes correspond to the path/row of the Landsat orbit.." ,/
 type ="string" , group= "Quality and Validity" ,  multiline= True #}

Syntax of Jinja template system and editor custom parser

  • both of braces of one metadata element must be on same line
  • after '{# ','{%' must be white space. Also before '#}' '-%}'

Examples of template

INSPIRE and Basic

Video tutorial shows practice of creation custom ISO profile.


Tools for working with metadata catalogue (csw)

Browsing and searching- g.gui.cswbrowser

Module g.gui.cswbrowser(Browser) based graphical interface allows managing csw connections, setting filter and searching, browsing results and adding services to GRASS GIS.

Connection manager

Connection manager allows:

  • adding
  • removing
  • loading list of connections from file.

By default the Browser is initialized by list of few well known csw providers of csw. Button "Service info" shows metadata of the service. User can manage connection in two ways:

  • First one is with using hidden storage provided by ​wx.Config. This case is without accessing default connection file.
  • After prompt user also can add/remove connection from default .xml file.
Connection manager in g.gui.cswbrowser
Connection manager in g.gui.cswbrowser

Searching

Before fetching data from selected csw server is proper to set up searching filter. Searching can be restricted by:

  • DCMI Type (Collection , Dataset , Event , Image , InteractiveResource , MovingImage , PhysicalObject , Service , Software , Sound , StillImage , Text)
  • Area(bounding box)
    • current GRASS GIS computation region
    • manually
  • Keywords
    • basic filter allows to use basic or advance keyword filtering. In the simple case user can define single keywords or multiple keywords with button "+". Logic operator between keywords is AND(&).
    • second, advanced is based on OGC list of expressions which means that can be set filter with logic relations between keywords or sets of keywords. Dialogue for settings keywords text string is under "Advanced" check-box. Syntax of constraints is based on python list syntax. Each keywords must be in braces
g.gui.cswbrowser, searching page
g.gui.cswbrowser, searching page

Advanced keywords filtering

On the left panel of Browser is placed check-box named "Advance". The keywords advanced dialogue allows to use basic bool operations.

  • OR condition
    a || b || c
    ["a","b","c"]
  • AND condition
    a && b && c
    [ ["a","b","c"] ]
  • composition
    (a && b) || c || d || e
    [["a","b"],["c"],["d"],["e"]] or [["a","b"],"c","d","e"]

Browsing results

On the bottom of Browser is a controller for browsing results. The data from server are fetched by number of items per one transaction. Default is a 20 items. After selection of item in the result table, metadata of record should be displayed. Browser automatically enables buttons on the top of Browser if WMS, WCS or WFS services are available.

Setting up csw server

CSW server management

wx.metadata package includes tools for establishing and handling pycsw server. Pycsw stuff consist:

  • db.csw.admin
  • db.csw.run
  • db.csw.harvest

For handling pycsw server can be use standard pythons scripts from pycsw install directory or mentioned three modules.

Installation and configuration csw server

  • Installation pycsw package - development version(OGC CSW server implementation written in Python):

e.g.to default path taken from configure file

$ cd /var/www/pycsw
  • deploying pycsw
    • $ git clone https://github.com/geopython/pycsw.git && cd pycsw
      
    • $ pip install -e . && pip install -r requirements-standalone.txt
      
  • Create and adjust a configuration file:
    • $ cp default-sample.cfg default.cfg
      
  • adjust parameters in default.cfg
    • server.home - path to installation directory
    • repository.database - path to SQLite database e.g. $GISDBASE/WGS84/PERMANENT/sqlite/sqlite.db
    • set server.url to: http://localhost:8000/
  • Setup the database(creates tables, indexes, etc.)
    • with unix command line(in pycsw install dir)
      • $ pycsw-admin.py -c load_records -f default.cfg -p /path/to/xml/
        
    • with module db.csw.admin from GRASS GIS. Flag -s and parameter configure with path to configure file. Path to database is automatically taken from configure file.
      • db.csw.admin -s configure=/var/www/html/pycsw/default.cfg
        
  • Load records:
    • with unix command line(in pycsw install dir)
      • pycsw-admin.py -c load_records -f default.cfg -p /path/to/xml/
        
    • db.csw.admin module with parameter -r for load recursively
      • db.csw.admin -r configure=/var/www/html/pycsw/default.cfg load_records=path/to/xml
        
  • Run the server
    • with unix command line(in pycsw install dir)
      • python csw.wsgi
        
    • db.csw.admin module with parameter path and port
      • db.csw.run path=/var/www/html/pycsw port=8000
        

Harvesting csw catalogue

Module db.csw.harvest allows to copy data from source csw server to destination csw server

See also