GRASS-Wiki - User contributions [en]

ISO/INSPIRE Metadata Support

2023-07-19T19:14:45Z

⚠️Tmszi: Update install additional Python packages under Gentoo GNU/Linux and update dependency.py file URL

== ISO 19115:2003- Geographic information -- Metadata ==
[http://www.iso.org/iso/catalogue_detail.htm?csnumber=26020 ISO 19115:2003] defines the scheme required for description of geographic information and services. It provides information about the identification, the extent, the quality, the spatial and temporal scheme, spatial reference, and distribution of digital geographic data.

== Catalogue Service for the Web ==
[http://www.opengeospatial.org/standards/cat OGC Catalogue Services] support the ability to publish and search collections of descriptive information (metadata) for data, services, and related information objects. Metadata in catalogues represent resource characteristics that can be queried and presented for evaluation and further processing by both humans and software. Catalogue services are required to support the discovery and binding to registered information resources within an information community.

= About =
Wx.metadata which was developed during Google Summer of Code 2014/2015 is currently available in GRASS 7 add-ons: https://github.com/OSGeo/grass-addons/tree/master/grass7/gui/wxpython/wx.metadata
* The package for creating and editing metadata according to EN ISO 19115, described here, consists of few modules. The modules {{AddonCmd|r.info.iso}}, {{AddonCmd|v.info.iso}} and {{AddonCmd|t.info.iso}} are command line based and support the conversion from the current native metadata management ({{cmd|r.info}}, {{cmd|v.info}}, {{cmd|t.info}}) into ISO based metadata. The GUI based module - {{AddonCmd|g.gui.metadata}} - offers advanced tools for management of metadata.

[[Image:Wxmetadata-editing.png|center|600px]]

* The package also includes the module {{AddonCmd|g.gui.cswbrowser}} for searching and browsing metadata catalog (csw). For advanced users there are a few modules for establishing and handling pycsw server:
* {{AddonCmd|db.csw.admin}} - module for administration of pycsw server
* {{AddonCmd|db.csw.run}} - module for starting server on localhost
* {{AddonCmd|db.csw.harvest}} - module for harvesting metadata between two CSW.

= Requirements and installation =

== Dependencies ==
'''Fedora Linux + Centos:'''

<source lang="bash">
# install dependencies
pip3 install jinja2 lxml owslib pycsw pyexcel_ods3 pygments reportlab sqlalchemy validators

# osmsm JavaScript lib map static image generator which use OpenStreetMap data
# https://github.com/jperelli/osm-static-maps
sudo npm i -g osm-static-maps
osmsm --help

# verify dependencies
wget https://raw.githubusercontent.com/OSGeo/grass-addons/grass8/src/gui/wxpython/wx.metadata/mdlib/dependency.py
python3 dependency.py
rm -f dependency.py
</source>

'''Gentoo GNU/Linux:'''

<source lang="bash">
sudo emerge -av dev-python/jinja dev-python/lxml dev-python/owslib dev-python/pygments dev-python/reportlab dev-python/sqlalchemy dev-python/validators
# Create Python virtual enviroment
python -m venv --system-site-packages $HOME/venvs/wx_metadata
# Activate created Python environment
source $HOME/venvs/wx_metadata/bin/activate
# Install additional Python packages
pip install pycsw pyexcel_ods3

# osmsm JavaScript lib map static image generator which use OpenStreetMap data
# https://github.com/jperelli/osm-static-maps
sudo npm i -g osm-static-maps
osmsm --help

# verify dependencies
wget https://raw.githubusercontent.com/OSGeo/grass-addons/grass8/src/gui/wxpython/wx.metadata/mdlib/dependency.py
python3 dependency.py
rm -f dependency.py
</source>

'''Ubuntu 20.04 LTS Focal Fossa GNU/Linux:'''

<source lang="bash">
sudo apt-get install python3-jinja2 python3-lxml python3-owslib python3-pycsw python3-pygments python3-reportlab python3-sqlalchemy
pip3 install --user pyexcel_ods3 validators

# osmsm JavaScript lib map static image generator which use OpenStreetMap data
# https://github.com/jperelli/osm-static-maps
sudo npm i -g osm-static-maps
osmsm --help

# verify dependencies
wget https://raw.githubusercontent.com/OSGeo/grass-addons/grass8/src/gui/wxpython/wx.metadata/mdlib/dependency.py
python3 dependency.py
rm -f dependency.py
</source>

'''Dependencies, from source code:'''
* Of course, have [https://grass.osgeo.org/download/ GRASS GIS 7.x]

wx.metadata package has several dependencies on external libraries:

* [https://pycsw.org/ pycsw] package - development version(OGC CSW server implementation written in Python):
<source lang="bash">
git clone https://github.com/geopython/pycsw.git && cd pycsw
sudo pip install -e . && sudo pip install -r requirements-standalone.txt && cd ..
</source>

* [https://geopython.github.io/OWSLib/ OWSLib] package - development version (OGC CSW client implementation written in Python):
<source lang="bash">
git clone git://github.com/geopython/OWSLib.git
cd OWSLib && sudo python setup.py install && cd ..
</source>

* [https://jinja.pocoo.org/docs/dev/ Jinja2] (one of the most used template engines for Python)
<source lang="bash">
git clone https://github.com/mitsuhiko/jinja2.git
cd jinja2 && sudo python setup.py install && cd ..
</source>

* [https://pygments.org/ Pygments] for highlighting text(for Ubuntu below)
<source lang="bash">
sudo apt-get install python-pygments
</source>

* Additionally, python-reportlab is required. The name of the package could vary for your distro.

== Addon installation ==

To install the addon, run in a GRASS GIS session:
<source lang="bash">
g.extension wx.metadata
</source>

= Creating and editing metadata =
== Naming of metadata files and storage ==
The default location for exported metadata files is ''metadata'' directory in the map's mapset. For raster maps, the prefix derived from the current nomenclature is ''cell'', for vector maps ''vector''. File ends with .xml suffix.

For example, the default metadata file name for vector map "roads" is ''vector_roads.xml''.

== Metadata profile ==
The metadata profile defines attributes, which can be edited in this editor. The editor offers the already mentioned two profiles (Basic and [https://inspire.ec.europa.eu/index.cfm INSPIRE]). The editor also supports loading of customized profiles. See Creation of custom ISO based profile.

<big>Currently there are three build-in ISO profiles available: </big>
* '''The basic profile''' is substituted from intersection between items stored in GRASS native metadata format and INSPIRE profile. The intersect (subset) includes all available GRASS metadata. Metadata which cannot be assigned to ISO based attributes are stored in metadata attribute ''sabstractact''.

* '''The INSPIRE profile''' fulfills the [https://inspire.ec.europa.eu/index.cfm/pageid/101 criteria of INSPIRE profile]. In case if is not possible to get the values from native GRASS metadata, these are filled with text string '$NULL'. This rule is applied for both profiles.

* '''The temporal profile''' is based on the GRASS basic profile and includes additional specific attributes for Temporal framework.

== Modules r.info.iso, v.info.iso t.info.iso ==
Command line based modules implements conversion of native GRASS metadata to ISO based XML metadata files. Identical configuration applies for both modules. Products of these modules differ partly in certain metadata attributes related to differences between raster ({{cmd|r.info}}) and vector ({{cmd|v.info}}) maps metadata.

'''Parameters of modules:'''
* map - selection of map for metadata creation
* profile - selection of metadata profile (basic or inspire)
* output - path to output XML metadata file. Default = according to naming & storage rules

== Metadata editor: g.gui.metadata ==
The metadata editor ([https://github.com/OSGeo/grass-addons/blob/master/grass7/gui/wxpython/wx.metadata/g.gui.metadata/g.gui.metadata.html addon on GitHub]) includes a graphical interface for converting metadata from {{cmd|r.info}} and {{cmd|v.info}} to ISO based metadata, editing metadata files and creating metadata templates. In addition, the graphical module allows validating of INSPIRE and Basic profile.

=== Information about metadata item ===
Tooltip function helps to fill in the form correctly.
[[Image:Wxmetadata-tooltip.png|center|350]]

=== Creation of metadata ===
The editor offers two editing modes. The first one allows to create metadata from selected GRASS maps from an active location. The second one allows editing of external metadata file.

=== Fetching values from database backend ===
The editor automatically fetches keywords from SQLite GRASS GIS backend. Three well know dictionaries are added by default. The user can manually add dictionaries by SQL commands.
Dictionaries are stored in the table ''metadata_themes''

{| border="1" class="wikitable"
|+ table metadata_themes
! Attribute
| title
| keyword
| date_iso
| date_type
|-
! Date type
| TEXT || TEXT|| TEXT|| TEXT
|-
! Example
|Example || GEMET - Concepts, version || 2010-01-13 || publication

|}

[[Image:Keywords.png|center|350]]

=== Exporting metadata reports to format PDF ===
This function allows to export metadata of GRASS maps to report in PDF format. The report template is based on INSPIRE profile. Unknown values of metadata attributes are filled by text $UNKNOWN.
[[Image:Pdfexp.png|center|350]]

{{YouTube|V7tgOOuRyuA}}

=== GRASS metadata map editor ===
[[Image:Wxmetadata-mapEdit.png|center|150px]]

Combo box for selection of ISO profile:
* basic
* inspire
* load custom – allows to load metadata profile from file. For example in case of using predefined template. The intersection of GRASS Basic profile (all available metadata for selected map) and loaded custom profile will proceed with this selection.

{{YouTube|7V5NyVdO05o}}

=== External metadata editor ===
[[Image:Wxmetadata-externalEdit.png|center|150]]
This editor mode allows to load own template and xml file for editing. This option does not support any connection with GRASS maps metadata.

{{YouTube|npMAavbwfi0}}

=== Defining templates ===
This function allows to create pattern of metadata profile. In other words it support defining fixed values to template. With using created template, the editor does not initialize texts inputs of defined values. These attributes is able to see only in Tree Browser in editor. Direct link on this function is multiple editing mode (see below).
* If check-box of metadata attribute is checked, the OWSLib object in template will be replaced by value. Unchecked attributes will not change this 'part' of the template.
* The yellow background of text fields indicates metadata attributes, which are not possible get from GRASS map information.
[[Image:Wxmetadata-template.png|center|600px]]

=== Usage ===
* If the check-box of metadata attribute is checked, the OWSLib object in template will be replaced by value. Unchecked attributes will not change this 'part' of the template.

example: Template result of attribute which is checked
<pre>
<gmd:dateStamp>
<gco:Date>2013-01-01</gco:Date>
</gmd:dateStamp>
</pre>

example: Template result of attribute which is unchecked
<pre>
<gmd:dateStamp>
<gco:Date>{{ md.datestamp }}{# tag="md.datestamp" , name="Metadata date" ,\
ref= "Part B 10.2" , desc= "Date that the metadata was created.",type = "date" ,
example = "2005-03-27" , multi= 0 , group= "Metadata" , multiline= False #}</gco:Date>
</gmd:dateStamp>
</pre>

Video shows record of simple practice of defining custom template
{{YouTube|Pu2sQoAJXfo}}

=== Editing metadata of multiple map selection ===
The advantages of this editing mode is in work using of defined template. This option can be used by selection of multiple maps in data catalog.

=== Validation of metadata profile ===
On the right side of editor is a validator of metadata. Currently, validator is able to validate two build-in profiles. For selection of profile “Load custom” is set by default the inspire profile.

=== Updating GRASS native metadata ===
The {{AddonCmd|g.gui.metadata}} module allows updating native GRASS metadata with using modules {{cmd|r.support}} and {{cmd|v.support}}. The intersection between ISO based metadata and r|v.support is limited by available parameters for these modules.

=== Publishing to csw server ===
Editor allows direct publishing of edited metadata by CSW(Catalog Service for Web) standard to remote catalogue server. Note that the remote catalogue must allows transaction.
[[File:Csw publisher.png|650px|center|publishing of metadata directly to remote csw server]]

=== Notes ===
* EPSG code must by added in url format. Information of projection is added only if EPSG code is known or if GDAL function for identifying from WKT is successful.
<source lang="xml">
<gmd:referenceSystemInfo>
<gmd:MD_ReferenceSystem>
<gmd:referenceSystemIdentifier>
<gmd:RS_Identifier>
<gmd:code>
<gco:CharacterString>http://www.opengis.net/def/crs/EPSG/0/3035</gco:CharacterString>
</gmd:code>
<gmd:codeSpace>
<gco:CharacterString>INSPIRE RS registry</gco:CharacterString>
</gmd:codeSpace>
</gmd:RS_Identifier>
</gmd:referenceSystemIdentifier>
</gmd:MD_ReferenceSystem>
</gmd:referenceSystemInfo>
</source>

= Creation of custom ISO based profile =
Creation of own custom ISO based profile is with using Jinja template syntax and some addition rules.
The main item of template is composite from two braces.
Both of them are fully according to syntax of Jinja template. First holds information about statements of loop or condition and also simple Python object. Second braces according to Jinja template syntax is represented by comment {# #}. In this braces additional information for generating graphical interface of editor are stored. In addition also information about individual metadata attributes (name, example, description, multiplicity, etc) is included.

== The first braces ==
Contain object of OWSLib which holds the metadata value. This values can be object which is describing single variable or list of variables. Value of some attribute can be defined￹ many times (according to ISO). This attributes must be defined in block of template with starting statements and ending mark of block. Only allowed loop statement is FOR (see jinja2 manual). The XML based text of block will by duplicated according to number of items in the list.

== Initialize of OWSLib objects ==
Most object are initialized in configuration of file which is stored in wx.metadat/config/init_md.txt

example: Default configuration file for the build-in Basic and Inspire profile:
<pre>
self.md.identification = MD_DataIdentification()
self.md.dataquality = DQ_DataQuality()
self.md.distribution = MD_Distribution()
self.md.identification.extent = EX_Extent()
self.md.identification.extent.boundingBox = EX_GeographicBoundingBox()
</pre>
# on the left side of equation is obligatory prefix 'self.md'. Postfix (name of variable) is open. On right side is OWSLib object

== Single element ==
In case if the value of metadata element is declared only once:
example: Identifier
<pre>
{{ md.identifier }}
</pre>

example: Lineage
<pre>
{{ md.dataquality.lineage }}
</pre>

== Loop or Condition statements ==
Loops and condition are defined in by starting braces and ending braces '{% -%}' .

Start of block:

example: for statement
<pre>
{% for co in md.contact -%}
</pre>

Editor allows to use Python 'zip' function and 'len' function. Note- in Jinja syntax is Python len(length) function defined specifically.

example: zip
<pre>
{% for (u,d) in zip(md.identification.uom,md.identification.distance) -%}
</pre>

example: len
<pre>
{% if md.identification.denominators|length > 0 -%}
</pre>

End of block is defined by:

example: if statement
<pre>
{% endif -%}
</pre>

example: for statement
<pre>
{% endfor -%}
</pre>

== The second braces ==
The second braces (by Jinja syntax comment) contains these parameters:
* tag - just copy of string from first braces
* object – for reading and holding metadata in python objects is used OWSLib library.

Most of object are initialized by configuration file wx.metadata/config/init_md.txt. Some metadata attributes are necessary initialized directly in this (object) parameter. Best example is the metadata contact object(organization,email) which can be initialized for many times.

example: responsible party
<pre>
{% for co in md.contact -%}{# object="CI_ResponsibleParty()" #}
{{ co.organization }}{# object="CI_ResponsibleParty()" #}
{{ co.email }}{# object="CI_ResponsibleParty()" #}
{{ co.role }}{# object="CI_ResponsibleParty()" #}
{% endfor -%}
</pre>

* name – free text string describing the name of metadata item
* ref – free additional information. By default should be reference of ISO attributes
* type – data type characteristic. Input text field validator for 'email' and 'date' is implemented.
* desc – free text which should describe or define the characteristic of metadata item
* example – free text with example of the value for metadata item
* group- label of the notebook list(tab) in editor.
* inbox- label of the box which includes attributes from the block of Jinja template. All items from one block of Jinja template must be in same box(same name).
* multi- 0 or 1. In case 0, the item cannot be duplicated= text field widget does not contain button for duplicating itself. Note that only OWSLib list based variables can be used with duplication button.
* inboxmulti- 0 or 1. In case 0, the items in box cannot be duplicated = box with items not dispose with button for cloning self.
* multiline – True = initialize text field with multilines editing. Suitable for longer texts input

example of second braces:
<pre>
{# tag="md.dataquality.lineage" , name="Lineage" , ref= "Part B 6.1" ,/
desc= "General explanation of the data producers knowledge about the lineage of a dataset."/
,example = "Product 1 scenes correspond to the path/row of the Landsat orbit.." ,/
type ="string" , group= "Quality and Validity" , multiline= True #}
</pre>

== Syntax of Jinja template system and editor custom parser ==
* both of braces of one metadata element must be on same line
* after '{# ','{%' must be white space. Also before '#}' '-%}'

== Examples of template ==

This video tutorial shows practice of creation custom ISO profile.
{{YouTube|ahl41mFD5w0}}

= Tools for working with metadata catalogue (csw) =
== Browsing and searching- g.gui.cswbrowser ==
Module {{AddonCmd|g.gui.cswbrowser}} (Browser) based graphical interface allows managing csw connections, setting filter and searching, browsing results and adding services to GRASS GIS.

=== Connection manager ===
Connection manager allows:
* adding
* removing
* loading list of connections from file.

By default the Browser is initialized by list of few well known csw providers of csw. Button "Service info" shows metadata of the service.

User can manage connection in two ways:
* First one is with using hidden storage provided by [http://wxpython.org/Phoenix/docs/html/ConfigBase.html wx.Config]. This case is without accessing default connection file.
* After prompt user also can add/remove connection from default .xml file.
[[File:ConnectionMgr.png|center|600px|Connection manager in g.gui.cswbrowser]]

=== Searching ===
Before fetching data from selected csw server is proper to set up searching filter. Searching can be restricted by:
* DCMI Type (Collection , Dataset , Event , Image , InteractiveResource , MovingImage , PhysicalObject , Service , Software , Sound , StillImage , Text)
* Area(bounding box)
** current GRASS GIS computation region
** manually
* Keywords
** basic filter allows to use basic or advance keyword filtering. In the simple case user can define single keywords or multiple keywords with button "+". Logic operator between keywords is AND(&).
** second, advanced is based on OGC list of expressions which means that can be set filter with logic relations between keywords or sets of keywords. Dialogue for settings keywords text string is under "Advanced" check-box. Syntax of constraints is based on python list syntax. Each keywords must be in braces
[[File:Find1.png|center|600px|g.gui.cswbrowser, searching page]]

==== Advanced keywords filtering ====
On the left panel of Browser is placed check-box named "Advance". The keywords advanced dialogue allows to use basic bool operations.

*OR condition
a || b || c
["a","b","c"]
*AND condition
a && b && c
[ ["a","b","c"] ]
*composition
(a && b) || c || d || e
[["a","b"],["c"],["d"],["e"]] or [["a","b"],"c","d","e"]

=== Browsing results===
On the bottom of Browser is a controller for browsing results. The data from server are fetched by number of items per one transaction. Default is a 20 items. After selection of item in the result table, metadata of record should be displayed. Browser automatically enables buttons on the top of Browser if WMS, WCS or WFS services are available.

=== Setting up csw server ===

== CSW server management ==
wx.metadata package includes tools for establishing and handling pycsw server. Pycsw stuff consists of:
* {{AddonCmd|db.csw.admin}}
* {{AddonCmd|db.csw.run}}
* {{AddonCmd|db.csw.harvest}}

For handling pycsw server can be use standard pythons scripts from pycsw install directory or mentioned three modules.

== Installation and configuration csw server ==
* Installation pycsw package - development version(OGC CSW server implementation written in Python):
e.g.to default path taken from configure file
<source lang="bash">
$ cd /var/www/pycsw
</source>

*deploying pycsw
**<source lang="bash">
$ git clone https://github.com/geopython/pycsw.git && cd pycsw
</source>
**<source lang="bash">
$ pip install -e . && pip install -r requirements-standalone.txt
</source>
* Create and adjust a configuration file:
**<source lang="bash">
$ cp default-sample.cfg default.cfg
</source>

* adjust parameters in ''default.cfg''
** '''server.home''' - path to installation directory
** '''repository.database''' - path to SQLite database e.g. $GISDBASE/WGS84/PERMANENT/sqlite/sqlite.db
** set '''server.url''' to: http://localhost:8000/

*Setup the database(creates tables, indexes, etc.)
**with unix command line(in pycsw install dir)
***<source lang="bash">
$ pycsw-admin.py -c load_records -f default.cfg -p /path/to/xml/
</source>
**with module db.csw.admin from GRASS GIS. Flag -s and parameter configure with path to configure file. Path to database is automatically taken from configure file.
***<source lang="bash">
db.csw.admin -s configure=/var/www/html/pycsw/default.cfg
</source>

*Load records:
**with unix command line(in pycsw install dir)
***<source lang="bash">
pycsw-admin.py -c load_records -f default.cfg -p /path/to/xml/
</source>
**db.csw.admin module with parameter -r for load recursively
***<source lang="bash">
db.csw.admin -r configure=/var/www/html/pycsw/default.cfg load_records=path/to/xml
</source>

*Run the server
**with unix command line(in pycsw install dir)
***<source lang="bash">
python csw.wsgi
</source>
**db.csw.admin module with parameter path and port
***<source lang="bash">
db.csw.run path=/var/www/html/pycsw port=8000
</source>

== Harvesting csw catalogue ==
Module {{AddonCmd|db.csw.harvest}} allows to copy data from source csw server to destination csw server.

= See also =
* [[GRASS Metadata Management]]
* [http://trac.osgeo.org/grass/wiki/GSoC/2014/MetadataForGRASS Development page 2014]
* [https://trac.osgeo.org/grass/wiki/GSoC/2015/ImprovedMetadata Development page 2015]

[[Category:Development]]
[[Category:Metadata]]
[[Category:INSPIRE]]

ISO/INSPIRE Metadata Support

2023-05-01T18:11:10Z

⚠️Tmszi: Add osmsm JavaScript lib map static image generator which use OpenStreetMap data as dependency for installation

== ISO 19115:2003- Geographic information -- Metadata ==
[http://www.iso.org/iso/catalogue_detail.htm?csnumber=26020 ISO 19115:2003] defines the scheme required for description of geographic information and services. It provides information about the identification, the extent, the quality, the spatial and temporal scheme, spatial reference, and distribution of digital geographic data.

== Catalogue Service for the Web ==
[http://www.opengeospatial.org/standards/cat OGC Catalogue Services] support the ability to publish and search collections of descriptive information (metadata) for data, services, and related information objects. Metadata in catalogues represent resource characteristics that can be queried and presented for evaluation and further processing by both humans and software. Catalogue services are required to support the discovery and binding to registered information resources within an information community.

= About =
Wx.metadata which was developed during Google Summer of Code 2014/2015 is currently available in GRASS 7 add-ons: https://github.com/OSGeo/grass-addons/tree/master/grass7/gui/wxpython/wx.metadata
* The package for creating and editing metadata according to EN ISO 19115, described here, consists of few modules. The modules {{AddonCmd|r.info.iso}}, {{AddonCmd|v.info.iso}} and {{AddonCmd|t.info.iso}} are command line based and support the conversion from the current native metadata management ({{cmd|r.info}}, {{cmd|v.info}}, {{cmd|t.info}}) into ISO based metadata. The GUI based module - {{AddonCmd|g.gui.metadata}} - offers advanced tools for management of metadata.

[[Image:Wxmetadata-editing.png|center|600px]]

* The package also includes the module {{AddonCmd|g.gui.cswbrowser}} for searching and browsing metadata catalog (csw). For advanced users there are a few modules for establishing and handling pycsw server:
* {{AddonCmd|db.csw.admin}} - module for administration of pycsw server
* {{AddonCmd|db.csw.run}} - module for starting server on localhost
* {{AddonCmd|db.csw.harvest}} - module for harvesting metadata between two CSW.

= Requirements and installation =

== Dependencies ==
'''Fedora Linux + Centos:'''

<source lang="bash">
# install dependencies
pip3 install jinja2 lxml owslib pycsw pyexcel_ods3 pygments reportlab sqlalchemy validators

# osmsm JavaScript lib map static image generator which use OpenStreetMap data
# https://github.com/jperelli/osm-static-maps
sudo npm i -g osm-static-maps
osmsm --help

# verify dependencies
wget https://raw.githubusercontent.com/OSGeo/grass-addons/master/grass7/gui/wxpython/wx.metadata/mdlib/dependency.py
python3 dependency.py
rm -f dependency.py
</source>

'''Gentoo GNU/Linux:'''

<source lang="bash">
sudo emerge -av dev-python/jinja dev-python/lxml dev-python/owslib dev-python/pygments dev-python/reportlab dev-python/sqlalchemy
pip install --user pycsw pyexcel_ods3 validators

# osmsm JavaScript lib map static image generator which use OpenStreetMap data
# https://github.com/jperelli/osm-static-maps
sudo npm i -g osm-static-maps
osmsm --help

# verify dependencies
wget https://raw.githubusercontent.com/OSGeo/grass-addons/master/grass7/gui/wxpython/wx.metadata/mdlib/dependency.py
python3 dependency.py
rm -f dependency.py
</source>

'''Ubuntu 20.04 LTS Focal Fossa GNU/Linux:'''

<source lang="bash">
sudo apt-get install python3-jinja2 python3-lxml python3-owslib python3-pycsw python3-pygments python3-reportlab python3-sqlalchemy
pip3 install --user pyexcel_ods3 validators

# osmsm JavaScript lib map static image generator which use OpenStreetMap data
# https://github.com/jperelli/osm-static-maps
sudo npm i -g osm-static-maps
osmsm --help

# verify dependencies
wget https://raw.githubusercontent.com/OSGeo/grass-addons/master/grass7/gui/wxpython/wx.metadata/mdlib/dependency.py
python3 dependency.py
rm -f dependency.py
</source>

'''Dependencies, from source code:'''
* Of course, have [https://grass.osgeo.org/download/ GRASS GIS 7.x]

wx.metadata package has several dependencies on external libraries:

* [https://pycsw.org/ pycsw] package - development version(OGC CSW server implementation written in Python):
<source lang="bash">
git clone https://github.com/geopython/pycsw.git && cd pycsw
sudo pip install -e . && sudo pip install -r requirements-standalone.txt && cd ..
</source>

* [https://geopython.github.io/OWSLib/ OWSLib] package - development version (OGC CSW client implementation written in Python):
<source lang="bash">
git clone git://github.com/geopython/OWSLib.git
cd OWSLib && sudo python setup.py install && cd ..
</source>

* [https://jinja.pocoo.org/docs/dev/ Jinja2] (one of the most used template engines for Python)
<source lang="bash">
git clone https://github.com/mitsuhiko/jinja2.git
cd jinja2 && sudo python setup.py install && cd ..
</source>

* [https://pygments.org/ Pygments] for highlighting text(for Ubuntu below)
<source lang="bash">
sudo apt-get install python-pygments
</source>

* Additionally, python-reportlab is required. The name of the package could vary for your distro.

== Addon installation ==

To install the addon, run in a GRASS GIS session:
<source lang="bash">
g.extension wx.metadata
</source>

= Creating and editing metadata =
== Naming of metadata files and storage ==
The default location for exported metadata files is ''metadata'' directory in the map's mapset. For raster maps, the prefix derived from the current nomenclature is ''cell'', for vector maps ''vector''. File ends with .xml suffix.

For example, the default metadata file name for vector map "roads" is ''vector_roads.xml''.

== Metadata profile ==
The metadata profile defines attributes, which can be edited in this editor. The editor offers the already mentioned two profiles (Basic and [https://inspire.ec.europa.eu/index.cfm INSPIRE]). The editor also supports loading of customized profiles. See Creation of custom ISO based profile.

<big>Currently there are three build-in ISO profiles available: </big>
* '''The basic profile''' is substituted from intersection between items stored in GRASS native metadata format and INSPIRE profile. The intersect (subset) includes all available GRASS metadata. Metadata which cannot be assigned to ISO based attributes are stored in metadata attribute ''sabstractact''.

* '''The INSPIRE profile''' fulfills the [https://inspire.ec.europa.eu/index.cfm/pageid/101 criteria of INSPIRE profile]. In case if is not possible to get the values from native GRASS metadata, these are filled with text string '$NULL'. This rule is applied for both profiles.

* '''The temporal profile''' is based on the GRASS basic profile and includes additional specific attributes for Temporal framework.

== Modules r.info.iso, v.info.iso t.info.iso ==
Command line based modules implements conversion of native GRASS metadata to ISO based XML metadata files. Identical configuration applies for both modules. Products of these modules differ partly in certain metadata attributes related to differences between raster ({{cmd|r.info}}) and vector ({{cmd|v.info}}) maps metadata.

'''Parameters of modules:'''
* map - selection of map for metadata creation
* profile - selection of metadata profile (basic or inspire)
* output - path to output XML metadata file. Default = according to naming & storage rules

== Metadata editor: g.gui.metadata ==
The metadata editor ([https://github.com/OSGeo/grass-addons/blob/master/grass7/gui/wxpython/wx.metadata/g.gui.metadata/g.gui.metadata.html addon on GitHub]) includes a graphical interface for converting metadata from {{cmd|r.info}} and {{cmd|v.info}} to ISO based metadata, editing metadata files and creating metadata templates. In addition, the graphical module allows validating of INSPIRE and Basic profile.

=== Information about metadata item ===
Tooltip function helps to fill in the form correctly.
[[Image:Wxmetadata-tooltip.png|center|350]]

=== Creation of metadata ===
The editor offers two editing modes. The first one allows to create metadata from selected GRASS maps from an active location. The second one allows editing of external metadata file.

=== Fetching values from database backend ===
The editor automatically fetches keywords from SQLite GRASS GIS backend. Three well know dictionaries are added by default. The user can manually add dictionaries by SQL commands.
Dictionaries are stored in the table ''metadata_themes''

{| border="1" class="wikitable"
|+ table metadata_themes
! Attribute
| title
| keyword
| date_iso
| date_type
|-
! Date type
| TEXT || TEXT|| TEXT|| TEXT
|-
! Example
|Example || GEMET - Concepts, version || 2010-01-13 || publication

|}

[[Image:Keywords.png|center|350]]

=== Exporting metadata reports to format PDF ===
This function allows to export metadata of GRASS maps to report in PDF format. The report template is based on INSPIRE profile. Unknown values of metadata attributes are filled by text $UNKNOWN.
[[Image:Pdfexp.png|center|350]]

{{YouTube|V7tgOOuRyuA}}

=== GRASS metadata map editor ===
[[Image:Wxmetadata-mapEdit.png|center|150px]]

Combo box for selection of ISO profile:
* basic
* inspire
* load custom – allows to load metadata profile from file. For example in case of using predefined template. The intersection of GRASS Basic profile (all available metadata for selected map) and loaded custom profile will proceed with this selection.

{{YouTube|7V5NyVdO05o}}

=== External metadata editor ===
[[Image:Wxmetadata-externalEdit.png|center|150]]
This editor mode allows to load own template and xml file for editing. This option does not support any connection with GRASS maps metadata.

{{YouTube|npMAavbwfi0}}

=== Defining templates ===
This function allows to create pattern of metadata profile. In other words it support defining fixed values to template. With using created template, the editor does not initialize texts inputs of defined values. These attributes is able to see only in Tree Browser in editor. Direct link on this function is multiple editing mode (see below).
* If check-box of metadata attribute is checked, the OWSLib object in template will be replaced by value. Unchecked attributes will not change this 'part' of the template.
* The yellow background of text fields indicates metadata attributes, which are not possible get from GRASS map information.
[[Image:Wxmetadata-template.png|center|600px]]

=== Usage ===
* If the check-box of metadata attribute is checked, the OWSLib object in template will be replaced by value. Unchecked attributes will not change this 'part' of the template.

example: Template result of attribute which is checked
<pre>
<gmd:dateStamp>
<gco:Date>2013-01-01</gco:Date>
</gmd:dateStamp>
</pre>

example: Template result of attribute which is unchecked
<pre>
<gmd:dateStamp>
<gco:Date>{{ md.datestamp }}{# tag="md.datestamp" , name="Metadata date" ,\
ref= "Part B 10.2" , desc= "Date that the metadata was created.",type = "date" ,
example = "2005-03-27" , multi= 0 , group= "Metadata" , multiline= False #}</gco:Date>
</gmd:dateStamp>
</pre>

Video shows record of simple practice of defining custom template
{{YouTube|Pu2sQoAJXfo}}

=== Editing metadata of multiple map selection ===
The advantages of this editing mode is in work using of defined template. This option can be used by selection of multiple maps in data catalog.

=== Validation of metadata profile ===
On the right side of editor is a validator of metadata. Currently, validator is able to validate two build-in profiles. For selection of profile “Load custom” is set by default the inspire profile.

=== Updating GRASS native metadata ===
The {{AddonCmd|g.gui.metadata}} module allows updating native GRASS metadata with using modules {{cmd|r.support}} and {{cmd|v.support}}. The intersection between ISO based metadata and r|v.support is limited by available parameters for these modules.

=== Publishing to csw server ===
Editor allows direct publishing of edited metadata by CSW(Catalog Service for Web) standard to remote catalogue server. Note that the remote catalogue must allows transaction.
[[File:Csw publisher.png|650px|center|publishing of metadata directly to remote csw server]]

=== Notes ===
* EPSG code must by added in url format. Information of projection is added only if EPSG code is known or if GDAL function for identifying from WKT is successful.
<source lang="xml">
<gmd:referenceSystemInfo>
<gmd:MD_ReferenceSystem>
<gmd:referenceSystemIdentifier>
<gmd:RS_Identifier>
<gmd:code>
<gco:CharacterString>http://www.opengis.net/def/crs/EPSG/0/3035</gco:CharacterString>
</gmd:code>
<gmd:codeSpace>
<gco:CharacterString>INSPIRE RS registry</gco:CharacterString>
</gmd:codeSpace>
</gmd:RS_Identifier>
</gmd:referenceSystemIdentifier>
</gmd:MD_ReferenceSystem>
</gmd:referenceSystemInfo>
</source>

= Creation of custom ISO based profile =
Creation of own custom ISO based profile is with using Jinja template syntax and some addition rules.
The main item of template is composite from two braces.
Both of them are fully according to syntax of Jinja template. First holds information about statements of loop or condition and also simple Python object. Second braces according to Jinja template syntax is represented by comment {# #}. In this braces additional information for generating graphical interface of editor are stored. In addition also information about individual metadata attributes (name, example, description, multiplicity, etc) is included.

== The first braces ==
Contain object of OWSLib which holds the metadata value. This values can be object which is describing single variable or list of variables. Value of some attribute can be defined￹ many times (according to ISO). This attributes must be defined in block of template with starting statements and ending mark of block. Only allowed loop statement is FOR (see jinja2 manual). The XML based text of block will by duplicated according to number of items in the list.

== Initialize of OWSLib objects ==
Most object are initialized in configuration of file which is stored in wx.metadat/config/init_md.txt

example: Default configuration file for the build-in Basic and Inspire profile:
<pre>
self.md.identification = MD_DataIdentification()
self.md.dataquality = DQ_DataQuality()
self.md.distribution = MD_Distribution()
self.md.identification.extent = EX_Extent()
self.md.identification.extent.boundingBox = EX_GeographicBoundingBox()
</pre>
# on the left side of equation is obligatory prefix 'self.md'. Postfix (name of variable) is open. On right side is OWSLib object

== Single element ==
In case if the value of metadata element is declared only once:
example: Identifier
<pre>
{{ md.identifier }}
</pre>

example: Lineage
<pre>
{{ md.dataquality.lineage }}
</pre>

== Loop or Condition statements ==
Loops and condition are defined in by starting braces and ending braces '{% -%}' .

Start of block:

example: for statement
<pre>
{% for co in md.contact -%}
</pre>

Editor allows to use Python 'zip' function and 'len' function. Note- in Jinja syntax is Python len(length) function defined specifically.

example: zip
<pre>
{% for (u,d) in zip(md.identification.uom,md.identification.distance) -%}
</pre>

example: len
<pre>
{% if md.identification.denominators|length > 0 -%}
</pre>

End of block is defined by:

example: if statement
<pre>
{% endif -%}
</pre>

example: for statement
<pre>
{% endfor -%}
</pre>

== The second braces ==
The second braces (by Jinja syntax comment) contains these parameters:
* tag - just copy of string from first braces
* object – for reading and holding metadata in python objects is used OWSLib library.

Most of object are initialized by configuration file wx.metadata/config/init_md.txt. Some metadata attributes are necessary initialized directly in this (object) parameter. Best example is the metadata contact object(organization,email) which can be initialized for many times.

example: responsible party
<pre>
{% for co in md.contact -%}{# object="CI_ResponsibleParty()" #}
{{ co.organization }}{# object="CI_ResponsibleParty()" #}
{{ co.email }}{# object="CI_ResponsibleParty()" #}
{{ co.role }}{# object="CI_ResponsibleParty()" #}
{% endfor -%}
</pre>

* name – free text string describing the name of metadata item
* ref – free additional information. By default should be reference of ISO attributes
* type – data type characteristic. Input text field validator for 'email' and 'date' is implemented.
* desc – free text which should describe or define the characteristic of metadata item
* example – free text with example of the value for metadata item
* group- label of the notebook list(tab) in editor.
* inbox- label of the box which includes attributes from the block of Jinja template. All items from one block of Jinja template must be in same box(same name).
* multi- 0 or 1. In case 0, the item cannot be duplicated= text field widget does not contain button for duplicating itself. Note that only OWSLib list based variables can be used with duplication button.
* inboxmulti- 0 or 1. In case 0, the items in box cannot be duplicated = box with items not dispose with button for cloning self.
* multiline – True = initialize text field with multilines editing. Suitable for longer texts input

example of second braces:
<pre>
{# tag="md.dataquality.lineage" , name="Lineage" , ref= "Part B 6.1" ,/
desc= "General explanation of the data producers knowledge about the lineage of a dataset."/
,example = "Product 1 scenes correspond to the path/row of the Landsat orbit.." ,/
type ="string" , group= "Quality and Validity" , multiline= True #}
</pre>

== Syntax of Jinja template system and editor custom parser ==
* both of braces of one metadata element must be on same line
* after '{# ','{%' must be white space. Also before '#}' '-%}'

== Examples of template ==

This video tutorial shows practice of creation custom ISO profile.
{{YouTube|ahl41mFD5w0}}

= Tools for working with metadata catalogue (csw) =
== Browsing and searching- g.gui.cswbrowser ==
Module {{AddonCmd|g.gui.cswbrowser}} (Browser) based graphical interface allows managing csw connections, setting filter and searching, browsing results and adding services to GRASS GIS.

=== Connection manager ===
Connection manager allows:
* adding
* removing
* loading list of connections from file.

By default the Browser is initialized by list of few well known csw providers of csw. Button "Service info" shows metadata of the service.

User can manage connection in two ways:
* First one is with using hidden storage provided by [http://wxpython.org/Phoenix/docs/html/ConfigBase.html wx.Config]. This case is without accessing default connection file.
* After prompt user also can add/remove connection from default .xml file.
[[File:ConnectionMgr.png|center|600px|Connection manager in g.gui.cswbrowser]]

=== Searching ===
Before fetching data from selected csw server is proper to set up searching filter. Searching can be restricted by:
* DCMI Type (Collection , Dataset , Event , Image , InteractiveResource , MovingImage , PhysicalObject , Service , Software , Sound , StillImage , Text)
* Area(bounding box)
** current GRASS GIS computation region
** manually
* Keywords
** basic filter allows to use basic or advance keyword filtering. In the simple case user can define single keywords or multiple keywords with button "+". Logic operator between keywords is AND(&).
** second, advanced is based on OGC list of expressions which means that can be set filter with logic relations between keywords or sets of keywords. Dialogue for settings keywords text string is under "Advanced" check-box. Syntax of constraints is based on python list syntax. Each keywords must be in braces
[[File:Find1.png|center|600px|g.gui.cswbrowser, searching page]]

==== Advanced keywords filtering ====
On the left panel of Browser is placed check-box named "Advance". The keywords advanced dialogue allows to use basic bool operations.

*OR condition
a || b || c
["a","b","c"]
*AND condition
a && b && c
[ ["a","b","c"] ]
*composition
(a && b) || c || d || e
[["a","b"],["c"],["d"],["e"]] or [["a","b"],"c","d","e"]

=== Browsing results===
On the bottom of Browser is a controller for browsing results. The data from server are fetched by number of items per one transaction. Default is a 20 items. After selection of item in the result table, metadata of record should be displayed. Browser automatically enables buttons on the top of Browser if WMS, WCS or WFS services are available.

=== Setting up csw server ===

== CSW server management ==
wx.metadata package includes tools for establishing and handling pycsw server. Pycsw stuff consists of:
* {{AddonCmd|db.csw.admin}}
* {{AddonCmd|db.csw.run}}
* {{AddonCmd|db.csw.harvest}}

For handling pycsw server can be use standard pythons scripts from pycsw install directory or mentioned three modules.

== Installation and configuration csw server ==
* Installation pycsw package - development version(OGC CSW server implementation written in Python):
e.g.to default path taken from configure file
<source lang="bash">
$ cd /var/www/pycsw
</source>

*deploying pycsw
**<source lang="bash">
$ git clone https://github.com/geopython/pycsw.git && cd pycsw
</source>
**<source lang="bash">
$ pip install -e . && pip install -r requirements-standalone.txt
</source>
* Create and adjust a configuration file:
**<source lang="bash">
$ cp default-sample.cfg default.cfg
</source>

* adjust parameters in ''default.cfg''
** '''server.home''' - path to installation directory
** '''repository.database''' - path to SQLite database e.g. $GISDBASE/WGS84/PERMANENT/sqlite/sqlite.db
** set '''server.url''' to: http://localhost:8000/

*Setup the database(creates tables, indexes, etc.)
**with unix command line(in pycsw install dir)
***<source lang="bash">
$ pycsw-admin.py -c load_records -f default.cfg -p /path/to/xml/
</source>
**with module db.csw.admin from GRASS GIS. Flag -s and parameter configure with path to configure file. Path to database is automatically taken from configure file.
***<source lang="bash">
db.csw.admin -s configure=/var/www/html/pycsw/default.cfg
</source>

*Load records:
**with unix command line(in pycsw install dir)
***<source lang="bash">
pycsw-admin.py -c load_records -f default.cfg -p /path/to/xml/
</source>
**db.csw.admin module with parameter -r for load recursively
***<source lang="bash">
db.csw.admin -r configure=/var/www/html/pycsw/default.cfg load_records=path/to/xml
</source>

*Run the server
**with unix command line(in pycsw install dir)
***<source lang="bash">
python csw.wsgi
</source>
**db.csw.admin module with parameter path and port
***<source lang="bash">
db.csw.run path=/var/www/html/pycsw port=8000
</source>

== Harvesting csw catalogue ==
Module {{AddonCmd|db.csw.harvest}} allows to copy data from source csw server to destination csw server.

= See also =
* [[GRASS Metadata Management]]
* [http://trac.osgeo.org/grass/wiki/GSoC/2014/MetadataForGRASS Development page 2014]
* [https://trac.osgeo.org/grass/wiki/GSoC/2015/ImprovedMetadata Development page 2015]

[[Category:Development]]
[[Category:Metadata]]
[[Category:INSPIRE]]

GRASS Debugging

2020-11-26T16:36:11Z

⚠️Tmszi: Extend wxPython GUI debugging subsection with gdb debugger example

The following hints assume to work on a working-copy of the {{twiki|DownloadSource#SubversionGRASSmainsourcecoderepository|GRASS GIS source code} directory.

Additionally it is good to have set the debugging symbols during compile-time. Do not strip the libraries or use optimization in the compile. see {{src|INSTALL}} and {{src|doc/debugging.txt}} in the source code for more information.

== Reporting bugs ==

* Bugs should be reported in the GRASS GIS [https://github.com/OSGeo/grass/issues bug and wish tracker].

* Please use [[GRASS_messages_translation#How_to_change_the_language|untranslated messages]] in your bug reports. The following command will disable the use of translated messages:
export LC_ALL=C

* Users should read the [[Mailing list etiquette]] page before posting questions to the <tt>grass-dev</tt> mailing list.

== Patches ==

* For instructions on creating and applying patches the see [[Patches]] wiki page.

== Setting GRASS-environment variables (numbers: 1-5) ==

{{cmd|g.gisenv}} set="DEBUG=1"

* A higher debug level means you see more messages.
* These messages are always present regardless of compiler settings.
* A setting of "5" is the most verbose, "0" is no debug messages.

== Common problems ==

* Check that the region settings are not totally overwhelming for the raster engine. While there is no set upper limit, anything bigger than 20000x20000 is considered large, and anything an order of mangnitude bigger than that will most likely end with out of memory or large file problems, especially on 32-bit platforms built without LFS support. Many modules ''will'' continue to work on these huge datasets, but processing time may make it not worth the while. See [[Memory issues]] for more tips.
{{cmd|g.region}} -p

== Watching top ==

''top'' is a nice command line utility for watching a process's progression. While it won't give you many hard metrics it will give you an idea of what is happening.
To make the view a little more stable, with ''top'' running from the command line press "M" to sort by memory usage.

== Return codes ==

All GRASS modules send out a return code of "0" if they exit successfully and "1" if they have exited with an error. This is typically accompanied by an "ERROR: " message.

On UNIX (Linux, Mac OS, and [[Cygwin]]) directly after a command line module ends you can type "<tt>echo $?</tt>" and it will tell you the exit code. In a MS-Windows DOS box the equivalent command is "<tt>echo %errorlevel%</tt>".

Note that some modules (like {{cmd|g.findfile}}) exit with a status of 1 if it (for example) could not find a file, which may be exactly the test the module was run to perform. In that way the return code can be used in script logic. Thus the exit code only reports EXIT_SUCCESS or EXIT_FAILURE, but a failure can happen for any number of reasons, expected or unexpected, and is not a reason to panic in and of itself.

== Using a C debugger on MS Windows ==

You'll need binaries built with debugging symbols [todo], then you can use tools like [http://support.microsoft.com/default.aspx?scid=kb;EN-US;308538 Dr. Watson] and [http://msdn.microsoft.com/en-us/windows/hardware/gg463016 windbg.exe] from Microsoft.
Some hints for how this could work can be found [http://www.cruisersforum.com/forums/f134/opencpn-version-2-4-beta-build-423-a-59409-4.html#post677590 here].

== DLL dependencies ==

Sometimes it is needed to disentangle DLL dependencies. A reasonable tool is "Dependency Walker" (free as in free lunch, not software libero): http://www.dependencywalker.com

== Using GDB ==

The [https://www.gnu.org/software/gdb/ GNU Debugger] may be used to diagnose '''<tt>Segmentation Faults</tt>''' and other weirdness.

=== Compile Time Setup ===

To add debugging information into the built binary, first run
make distclean

Then configure the source code with ''-g'' being added to the CFLAGS arguments:

CFLAGS="-ggdb -Wall -Werror-implicit-function-declaration" ./configure ...

Do '''not''' use ''-O'' for optimization and do '''not''' strip the binaries with LDFLAGS="-s".

Eventually compile the source code and start the GRSS GIS session for debugging (see below).

=== Using gdb on command line ===

* Running a program inside [https://en.wikipedia.org/wiki/Gdb GDB] works (installation of GDB required :-)
: (e.g. on [https://www.debian.org Debian GNU/Linux]: apt-get install gdb)

* Use the exact module name on the command line (at the GRASS prompt) without arguments. Put any command line arguments on the <tt>(gdb)</tt> command line after the word ''run''.

gdb `which v.in.ogr`
run out=some_map dsn="PG:dbname=postgis user=me" olayer=postgislayer

* when it crashes, type "<tt>bt full</tt>" for a full backtrace
* type "<tt>l</tt>" to list where in the source code it got to
* type "<tt>frame 2</tt>" to switch to the second level function (see the backtrace), there you can again type "<tt>l</tt>" to see where it got up to.

Alternatively, you can add arguments to gdb directly on the command line:

gdb --args v.in.ogr out=bla dsn="PG:dbname=postgis user=me" olayer=postgislayer

Additionally, you can tell it to run the process right away:

gdb -ex run --args v.in.ogr out=bla dsn="PG:dbname=postgis user=me" olayer=postgislayer

* ''Attaching to child process'':
: use "<tt>attach <pid></tt>" to attach to an existing process (needed for DBMI debugging etc).
: For details, see this [http://lists.osgeo.org/pipermail/grass-dev/2007-October/033732.html hint]

=== Saving a core dump for later analysis ===

On Linux, often if a program exits with a "<tt>Segmentation Fault</tt>" the program will exit leaving nothing behind. It is however possible to dump the state of the program (the memory "core") to a file. Ubuntu & Mac users may have seen with some crashing programs the long list of memory address computerese which it then asks if you want to report back to the authors.

On Linux, you can check if you are set to "dump the core" to a file with the ulimit command:
ulimit -c

a response of "<tt>0</tt>" means no core file will be written. Core files can be very large (depending on how much system RAM the program was using at the time it crashed), so is usually kept off. You can temporarily change the limit in a command terminal to dump the whole thing with:
ulimit -c unlimited
which will save a file called "core" in the present directory (if you have permission to write there). You can then analyze the core at a later time with:
gdb `which g.module` /path/to/core/file

It is important to note that you must analyze the ''core'' file against the exact version of the executable you were running at the time, and that if you send the ''core'' file to someone else they will be able to see the data you had loaded in the program at the time (if you are a consultant, you client might not like that).



=== Using gdb within GNU Emacs ===

Notes:
:* 'C' is usually key 'Ctrl'
:* 'M' is usually key 'Alt'

* start [https://en.wikipedia.org/wiki/Emacs GNU Emacs] within GRASS session, e.g.

emacs general/manage/cmd/remove.c

[[Image:Emacs_gdb-1a.png|center|640px]]

* create second buffer by <tt>C-x 2</tt>

[[Image:Emacs_gdb-2.png|center|640px]]

* move cursor to the second buffer by <tt>C-x o</tt>, then <tt>M-x gdb</tt>, Enter
* type module name which you would like to debug, e.g. <tt>g.remove</tt>, Enter

* now you can use gdb inside GNU Emacs

[[Image:Emacs_gdb-3.png|center|640px]]

=== Using DDD (gdb graphical frontend) ===
* debugging a program inside [https://en.wikipedia.org/wiki/Ddd DDD] is rather easy (installation of ddd and gdb required), (on Debian GNU/Linux: <tt>apt-get install gdb ddd</tt>; on Fedora: <tt>dnf install ddd</tt>). Start debugger with GRASS command (e.g., {{cmd|v.in.ogr}}):

ddd v.in.ogr

[[File:Ddd_startup.png|400px|thumb|center|ddd startup]]

* Run (from main menu PROGRAM): "out=test dsn="PG:dbname=postgis user=me" olayer=postgislayer"

[[File:Ddd_run1.png|400px|thumb|center|ddd define arguments]]

* run with parameters
* when it crashes, use the ''UP'' menu item to trace back
* reach the line where it crashed
* set a breakpoint, then run again to stop before the crash

[[File:Ddd_run2.png|400px|thumb|center|ddd going up]]

* right mouse button on variables permits to display them etc.

[[File:Ddd_display_vars.png|400px|thumb|center|ddd display variables]]

* figure out why it crashed there. This requires PATIENCE. But you will nearly save the world if you identify the problem :-)

=== Using kdbg (gdb graphical frontend) ===

* kdbg is not unlike DDD, but it's a KDE application.
* Use is similar to DDD.
* Start with (within GRASS):
kdbg `which g.module`
* Fill in command line arguments with menu item Execution->Arguments.
* Open the View->Locals window.
* Click the Run icon (or Execution->Run) and see where it breaks.
* Set pause-points by clicking a red stop-sign to the left of the "+" on a line of the source code. From there you can step through instructions.
* Explore the values of variables in the locals window.
* You can select an individual variable to watch with
View->Watched expressions
: Type "data" in the text box at the top to view the entire structure of some variable called "data". If it is a structure you can then expand it and click on an interior variable. Clicking on its name makes it come up in the text box. Hit enter from the text box and it will be added to the list of watched expressions. You can edit a Y to X (then enter again) to quickly add another similarly named variable. e.g.:
(data->header).Ycoordinate

== Using ldd ==

On Linux ''ldd'' will show the shared library dependencies for a program or library. For example:

For a module:
GRASS> ldd `which v.in.ogr`
For a library:
$ ldd /usr/src/grass/grass78/dist.x86_64-pc-linux-gnu/lib/libgrass_gis.so

The output may look like this:
linux-vdso.so.1 (0x00007ffe1cf03000)
libgrass_datetime.7.8.so => /usr/src/grass/grass78/dist.x86_64-pc-linux-gnulib/libgrass_datetime.7.8.so (0x00007fae7a6cf000)
libz.so.1 => /lib64/libz.so.1 (0x00007fae7a6b5000)
libzstd.so.1 => /lib64/libzstd.so.1 (0x00007fae7a5df000)
libm.so.6 => /lib64/libm.so.6 (0x00007fae7a499000)
libc.so.6 => /lib64/libc.so.6 (0x00007fae7a2cf000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fae7a2ad000)
/lib64/ld-linux-x86-64.so.2 (0x00007fae7a769000)

''ldd'' is good for spotting if the right libraries are used (in the correct path etc).

* On Mac OS "otool -L `which v.in.ogr`" almost does the same job.

* For MS Windows the [https://www.dependencywalker.com/ Dependency Walker] program (depends.exe) might help.

=== Library search path ===

If a support library like libgdal can't be found, and you are sure it is installed, make sure the library search path is set correctly.
For example in Linux if GDAL was installed to <tt>/usr/local</tt>, add the line "<tt>/usr/local/lib</tt>" to the /etc/ld.so.conf file, and as root run <tt>ldconfig</tt> to rebuild the library links and cache.

Alternatively you can add the library path to the $LD_LIBRARY_PATH environmental variable.

* [https://www.cprogramming.com/tutorial/shared-libraries-linux-gcc.html Read more about how shared libraries work on Linux]

== Using nm and objdump ==

''nm'' will show a list symbols contained in a library or a program. Use this if you want to check which functions it calls. It will not work if the binary has been stripped.

For example:
GRASS> nm $GISBASE/lib/libgrass_gis.so
GRASS> nm `which r.digit`

If the binaries have been stripped you can still use the -D option to show dynamic symbols instead of normal symbols:
$ nm -D /bin/ls

* see also the {{src|tools/sql.sh}} script in the GRASS souce code, which will run "nm" on every object file, library and executable to find dependencies. The result is stored in a PostgreSQL database.

''objdump'' will work even if the binaries have been stripped.
For example:
$ objdump -T /usr/lib/libX11.so
$ objdump -R /usr/lib/libX11.so

== Using LD_DEBUG ==

To see in which order the various libraries are called, run the command with <tt>LD_DEBUG=files</tt>.
For example:

LD_DEBUG=files r.info elevation

== Using strace ==

Running the command through strace could give you another hint what is going wrong somewhere. It shows the command execution at low level.

For example:
strace v.in.ogr out=your_map dsn="PG:dbname=postgis user=me" olayer=postgislayer

== Using Valgrind ==

* [https://valgrind.org Valgrind] is a tool to check for memory leaks and memory errors.

;Examples

* analysis of ''heap usage''
CMD="v.in.ascii -zt z=3 in=lidaratm2_250k.txt out=lidaratm2_250k fs=,"
valgrind --tool=massif $CMD --o

* analysis of ''memory leaks'' (include <code>--trace-children=yes</code> to profile e.g. the DBF driver or other children)

** Full memcheck tool, including non-orphaned, but left over, allocated memory:
valgrind --tool=memcheck --leak-check=yes --show-reachable=yes $CMD --o

;Notes

* The default is to send Valgrind output to stderr. Add <code>--log-file=<filename></code> to save output to a file, but note (if it matters) that you will lose any interleaved GRASS debug messages from the log that way.

* Valgrind can create PostScript output charts. To convert PostScript to PNG:
: ('pstoimg' is part of the latex2html Debian package)

* If you get something like this (huge value for "still reachable"):
<pre>
> ==13217== LEAK SUMMARY:
> ==13217== definitely lost: 964 bytes in 39 blocks
> ==13217== indirectly lost: 72 bytes in 10 blocks
> ==13217== possibly lost: 79 bytes in 4 blocks
> ==13217== still reachable: 9,025,026,330 bytes in 188,201,312 blocks
</pre>
then please run valgrind again with <tt>--show-reachable=yes</tt> as you will want to find out where these 9,025,026,330 bytes (in this example) have been allocated.

pstoimg -aaliastext -flip r90 -out massif.550.png -scale 1.3 -crop a massif.550.ps

* '''Graphical user interfaces''':
** https://kcachegrind.github.io/html/Home.html
** https://invent.kde.org/sdk/massif-visualizer
** https://wiki.gnome.org/Attic/MassifG

== Using Electric Fence ==

* [https://en.wikipedia.org/wiki/Electric_Fence Electric Fence] checks for bad memory writes, ie specific malloc() over-runs and under-runs.

('''''Insert howto here''''')

Example:
CMD="gdalwarp -tps -srcnodata 255 -dstnodata 255 \
-of GTiff -rcs ${TMP}.tif ${OUTFILE}_warp"
LD_PRELOAD=libefence.so.0.0 $CMD

== Using a profiling tool ==

A profiling tool may be used to identify the bottlenecks in long running processes.

=== Kcachegrind ===

[https://kcachegrind.github.io/html/Home.html kcachegrind] is a visualization tool for [https://valgrind.org Valgrind]'s callgrind profiling tool (formerly known as the calltree skin). It is very simple to use. Install the GraphViz package as well.

* The [https://valgrind.org/docs/manual/cl-manual.html callgrind chapter] in the Valgrind user manual

Example:
# Spearfish dataset
g.region raster=elevation.dem
v.random n=10000 out=rand10k
v.db.addtable rand10k column="elev INT"
CMD="v.what.rast rand10k rast=elevation.dem column=elev"
valgrind --tool=callgrind --trace-children=yes $CMD

This creates a file containing profiling information which kcachegrind can load. We include <code>--trace-children</code> in the above example so we can profile the DBF driver process as well as the main {{cmd|v.what.rast}} process.

kcachegrind callgrind.out.12345

In the above example from the two process costs we can see that 99.5% is taken by the dbf process and 0.5% is taken by the v.what.rast process. Within the dbf process almost 50% is taken by <tt>G_debug()</tt> and <tt>G_strcasecmp()</tt>.

=== gprof ===

* [https://sourceware.org/binutils/docs/gprof/ GNU gprof] user guide
* [http://linuxgazette.net/100/vinayak.html Profiling programs using gprof] - article from the Linux Gazzette, March 2004 (#100)

Compile GRASS with <code>-pg</code> flag. E.g.

CFLAGS='-pg' CXXFLAGS='-pg' LDFLAGS='-pg' ./configure --disable-shared

; Example

Run the module

v.select -tg ain=geodetic_swwake_pts bin=urbanarea out=x ope=overlap --o

Use gprof to generate human readable output

gprof ../../dist.i686-pc-linux-gnu/bin/v.select gmon.out > profile.txt

== Using Mudflap ==

GCC built in feature, that "instruments all risky pointer/array dereferencing operations, some standard library string/heap functions, and some other associated constructs with range/validity tests. Modules so instrumented should be immune to buffer overflows, invalid heap use, and some other classes of C/C++ programming errors. The instrumentation relies on a separate runtime library (libmudflap), which will be linked into a program if -fmudflap -lmudflap is given at link time." [http://gcc.gnu.org/wiki/Mudflap_Pointer_Debugging GCC Wiki]

* Requires GCC with mudflap support (GCC 4.x with mudflap USE flag on Gentoo, separate library "libmudflap*" in other distros);
* Add -fmudflap and -lmudflap options ('''''Is this correct way? It works...'''''):
** in include/Make/Platform.make add -lmudflap to LD_SEARCH_FLAGS;
** in include/Make/Platform.make add -fmudflap to CFLAGS1;
* Disable mudflap during compilation
export MUDFLAP_OPTIONS='-mode-nop -viol-nop'
* Compile GRASS (<tt>make clean && make</tt>). For best results use NO optimisations for compilation (<tt>-O0</tt>);
* Use GRASS.
** To disable warnings
*:<pre>export MUDFLAP_OPTIONS='-mode-nop -viol-nop'</pre>
** To enable warnings
*:<pre>export MUDFLAP_OPTIONS='-mode-check -viol-nop -check-initialization'</pre>

More information about debugging with mudflap, can be found [http://gcc.fyxm.net/summit/2003/mudflap.pdf here].

== Skimming the ChangeLog for changes ==

* use the tool {{src|tools/gitlog2changelog.py}} for generating a local changelog-file, by simply running:
** make changelog



== Shell script debugging ==

Add "-x" to the first line of the shell script:

original:
#!/bin/sh

change to:
#!/bin/sh -x

alternatively add `set -x` to somewhere near the start of the script.

Find the script file to edit with
which script

== Python script debugging ==

=== Python debugging with pdb ===

See also:
* pdb Background: https://docs.python.org/3/library/pdb.html
* pdb Tutorial: https://fedoramagazine.org/getting-started-python-debugger/

1. Edit the GRASS python script and add this line to the import section:
<source lang="python">
import pdb
</source>

2. Edit and add this line within the code where to debug:
<source lang="python">
pdb.set_trace()
</source>

3. Run the python script normally, it will stop where pdb.set_trace() was added:
r.unpack x60030_2000.green.histo2000_g.pack
<source lang="python">
GRASS 7.8 (patUTM32):~ > r.unpack x60030_2000.green.histo2000_g.pack
> /home/neteler/software/grass78/dist.x86_64-unknown-linux-gnu/scripts/r.unpack(93)main()
-> diff_result_1 = diff_result_2 = None
</source>

3. Now you can either (s)tep through or (n)ext or (r)run the rest (c: continue, q: quit)
<source lang="python">
(Pdb) s
</source>

While "s" will decend in functions, "n" will go to the next line. See the manual for [https://docs.python.org/2/library/pdb.html#debugger-commands Debugger Commands]

4. To print the content of variables:
<source lang="python">
(Pdb) s
> /usr/lib64/python3.8/posixpath.py(73)join()
-> path = a
(Pdb) a
a = /grassdata/patUTM32/hist_matching
p = ('..', 'PERMANENT', 'PROJ_INFO')
(Pdb)
</source>

Now will likely be able to better find the issues in your Python script.

See also: https://docs.python.org/3/library/pdb.html

=== wxPython GUI debugging ===

'''A) Debug messages:'''

[[wxGUI]] prints debugging messages in five levels (1-5). To redirect these messages to the terminal, run

* in GRASS 7.x

g.gisenv set=WX_DEBUG=1

'''B) Using a debugger:'''

For real debugging of wxgui.py ([https://github.com/OSGeo/grass/blob/master/general/g.gui/main.c source code]) with gdb or other tools, so you can use something like:

cd grass/src
python gui/wxpython/wxgui.py

or:

python $GISBASE/gui/wxpython/wxgui.py

or with gdb e.g.:

<source lang="bash">
# GRASS GIS session terminal
d.mon start=wx0

# Examine running wxGUI components by htop program and get pid
# Run in another terminal or in terminal tab
gdb --pid $(pgrep -f /gui/wxpython/mapdisp/main.py)
continue

# And then interact with the wxGUI component (d.mon)
# If wxGUI component crash then you get backtrace with gdb bt command
bt
</source>

Well, but that's just how to run the Python code directly.

=== Python debugging with gdb ===

See https://wiki.python.org/moin/DebuggingWithGdb

=== Komodo dbgp.client.brk() ===

* See http://docs.activestate.com/komodo/5.0/debugpython.html
from dbgp.client import brk

=== Profiling python scripts ===

* See https://docs.python.org/library/profile.html

== PROJ debugging ==

See also: https://proj.org/usage/environmentvars.html?highlight=debugging

For unix with a style shell use export:
export PROJ_DEBUG=5
cs2cs ...

[[Category:Development]]

ISO/INSPIRE Metadata Support

2020-09-30T03:48:35Z

⚠️Tmszi: Update Gentoo, Ubuntu GNU/Linux distribution py libs dependencies

== ISO 19115:2003- Geographic information -- Metadata ==
[http://www.iso.org/iso/catalogue_detail.htm?csnumber=26020 ISO 19115:2003] defines the scheme required for description of geographic information and services. It provides information about the identification, the extent, the quality, the spatial and temporal scheme, spatial reference, and distribution of digital geographic data.

== Catalogue Service for the Web ==
[http://www.opengeospatial.org/standards/cat OGC Catalogue Services] support the ability to publish and search collections of descriptive information (metadata) for data, services, and related information objects. Metadata in catalogues represent resource characteristics that can be queried and presented for evaluation and further processing by both humans and software. Catalogue services are required to support the discovery and binding to registered information resources within an information community.

= About =
Wx.metadata which was developed during Google Summer of Code 2014/2015 is currently available in GRASS 7 add-ons: https://github.com/OSGeo/grass-addons/tree/master/grass7/gui/wxpython/wx.metadata
* The package for creating and editing metadata according to EN ISO 19115, described here, consists of few modules. The modules {{AddonCmd|r.info.iso}}, {{AddonCmd|v.info.iso}} and {{AddonCmd|t.info.iso}} are command line based and support the conversion from the current native metadata management ({{cmd|r.info}}, {{cmd|v.info}}, {{cmd|t.info}}) into ISO based metadata. The GUI based module - {{AddonCmd|g.gui.metadata}} - offers advanced tools for management of metadata.

[[Image:Wxmetadata-editing.png|center|600px]]

* The package also includes the module {{AddonCmd|g.gui.cswbrowser}} for searching and browsing metadata catalog (csw). For advanced users there are a few modules for establishing and handling pycsw server:
* {{AddonCmd|db.csw.admin}} - module for administration of pycsw server
* {{AddonCmd|db.csw.run}} - module for starting server on localhost
* {{AddonCmd|db.csw.harvest}} - module for harvesting metadata between two CSW.

= Requirements and installation =

== Dependencies ==
'''Fedora Linux + Centos:'''

<source lang="bash">
# install dependencies
pip3 install jinja2 lxml owslib pycsw pyexcel_ods3 pygments reportlab sqlalchemy validators

# verify dependencies
wget https://raw.githubusercontent.com/OSGeo/grass-addons/master/grass7/gui/wxpython/wx.metadata/mdlib/dependency.py
python3 dependency.py
rm -f dependency.py
</source>

'''Gentoo GNU/Linux:'''

<source lang="bash">
sudo emerge -av dev-python/jinja dev-python/lxml dev-python/owslib dev-python/pygments dev-python/reportlab dev-python/sqlalchemy
pip install --user pycsw pyexcel_ods3 validators

# verify dependencies
wget https://raw.githubusercontent.com/OSGeo/grass-addons/master/grass7/gui/wxpython/wx.metadata/mdlib/dependency.py
python3 dependency.py
rm -f dependency.py
</source>

'''Ubuntu 20.04 LTS Focal Fossa GNU/Linux:'''

<source lang="bash">
sudo apt-get install python3-jinja2 python3-lxml python3-owslib python3-pycsw python3-pygments python3-reportlab python3-sqlalchemy
pip3 install --user pyexcel_ods3 validators

# verify dependencies
wget https://raw.githubusercontent.com/OSGeo/grass-addons/master/grass7/gui/wxpython/wx.metadata/mdlib/dependency.py
python3 dependency.py
rm -f dependency.py
</source>

'''Dependencies, from source code:'''
* Of course, have [https://grass.osgeo.org/download/ GRASS GIS 7.x]

wx.metadata package has several dependencies on external libraries:

* [https://pycsw.org/ pycsw] package - development version(OGC CSW server implementation written in Python):
<source lang="bash">
git clone https://github.com/geopython/pycsw.git && cd pycsw
sudo pip install -e . && sudo pip install -r requirements-standalone.txt && cd ..
</source>

* [https://geopython.github.io/OWSLib/ OWSLib] package - development version (OGC CSW client implementation written in Python):
<source lang="bash">
git clone git://github.com/geopython/OWSLib.git
cd OWSLib && sudo python setup.py install && cd ..
</source>

* [https://jinja.pocoo.org/docs/dev/ Jinja2] (one of the most used template engines for Python)
<source lang="bash">
git clone https://github.com/mitsuhiko/jinja2.git
cd jinja2 && sudo python setup.py install && cd ..
</source>

* [https://pygments.org/ Pygments] for highlighting text(for Ubuntu below)
<source lang="bash">
sudo apt-get install python-pygments
</source>

* Additionally, python-reportlab is required. The name of the package could vary for your distro.

== Addon installation ==

To install the addon, run in a GRASS GIS session:
<source lang="bash">
g.extension wx.metadata
</source>

= Creating and editing metadata =
== Naming of metadata files and storage ==
The default location for exported metadata files is ''metadata'' directory in the map's mapset. For raster maps, the prefix derived from the current nomenclature is ''cell'', for vector maps ''vector''. File ends with .xml suffix.

For example, the default metadata file name for vector map "roads" is ''vector_roads.xml''.

== Metadata profile ==
The metadata profile defines attributes, which can be edited in this editor. The editor offers the already mentioned two profiles (Basic and [https://inspire.ec.europa.eu/index.cfm INSPIRE]). The editor also supports loading of customized profiles. See Creation of custom ISO based profile.

<big>Currently there are three build-in ISO profiles available: </big>
* '''The basic profile''' is substituted from intersection between items stored in GRASS native metadata format and INSPIRE profile. The intersect (subset) includes all available GRASS metadata. Metadata which cannot be assigned to ISO based attributes are stored in metadata attribute ''sabstractact''.

* '''The INSPIRE profile''' fulfills the [https://inspire.ec.europa.eu/index.cfm/pageid/101 criteria of INSPIRE profile]. In case if is not possible to get the values from native GRASS metadata, these are filled with text string '$NULL'. This rule is applied for both profiles.

* '''The temporal profile''' is based on the GRASS basic profile and includes additional specific attributes for Temporal framework.

== Modules r.info.iso, v.info.iso t.info.iso ==
Command line based modules implements conversion of native GRASS metadata to ISO based XML metadata files. Identical configuration applies for both modules. Products of these modules differ partly in certain metadata attributes related to differences between raster ({{cmd|r.info}}) and vector ({{cmd|v.info}}) maps metadata.

'''Parameters of modules:'''
* map - selection of map for metadata creation
* profile - selection of metadata profile (basic or inspire)
* output - path to output XML metadata file. Default = according to naming & storage rules

== Metadata editor: g.gui.metadata ==
The metadata editor ([https://github.com/OSGeo/grass-addons/blob/master/grass7/gui/wxpython/wx.metadata/g.gui.metadata/g.gui.metadata.html addon on GitHub]) includes a graphical interface for converting metadata from {{cmd|r.info}} and {{cmd|v.info}} to ISO based metadata, editing metadata files and creating metadata templates. In addition, the graphical module allows validating of INSPIRE and Basic profile.

=== Information about metadata item ===
Tooltip function helps to fill in the form correctly.
[[Image:Wxmetadata-tooltip.png|center|350]]

=== Creation of metadata ===
The editor offers two editing modes. The first one allows to create metadata from selected GRASS maps from an active location. The second one allows editing of external metadata file.

=== Fetching values from database backend ===
The editor automatically fetches keywords from SQLite GRASS GIS backend. Three well know dictionaries are added by default. The user can manually add dictionaries by SQL commands.
Dictionaries are stored in the table ''metadata_themes''

{| border="1" class="wikitable"
|+ table metadata_themes
! Attribute
| title
| keyword
| date_iso
| date_type
|-
! Date type
| TEXT || TEXT|| TEXT|| TEXT
|-
! Example
|Example || GEMET - Concepts, version || 2010-01-13 || publication

|}

[[Image:Keywords.png|center|350]]

=== Exporting metadata reports to format PDF ===
This function allows to export metadata of GRASS maps to report in PDF format. The report template is based on INSPIRE profile. Unknown values of metadata attributes are filled by text $UNKNOWN.
[[Image:Pdfexp.png|center|350]]

{{YouTube|V7tgOOuRyuA}}

=== GRASS metadata map editor ===
[[Image:Wxmetadata-mapEdit.png|center|150px]]

Combo box for selection of ISO profile:
* basic
* inspire
* load custom – allows to load metadata profile from file. For example in case of using predefined template. The intersection of GRASS Basic profile (all available metadata for selected map) and loaded custom profile will proceed with this selection.

{{YouTube|7V5NyVdO05o}}

=== External metadata editor ===
[[Image:Wxmetadata-externalEdit.png|center|150]]
This editor mode allows to load own template and xml file for editing. This option does not support any connection with GRASS maps metadata.

{{YouTube|npMAavbwfi0}}

=== Defining templates ===
This function allows to create pattern of metadata profile. In other words it support defining fixed values to template. With using created template, the editor does not initialize texts inputs of defined values. These attributes is able to see only in Tree Browser in editor. Direct link on this function is multiple editing mode (see below).
* If check-box of metadata attribute is checked, the OWSLib object in template will be replaced by value. Unchecked attributes will not change this 'part' of the template.
* The yellow background of text fields indicates metadata attributes, which are not possible get from GRASS map information.
[[Image:Wxmetadata-template.png|center|600px]]

=== Usage ===
* If the check-box of metadata attribute is checked, the OWSLib object in template will be replaced by value. Unchecked attributes will not change this 'part' of the template.

example: Template result of attribute which is checked
<pre>
<gmd:dateStamp>
<gco:Date>2013-01-01</gco:Date>
</gmd:dateStamp>
</pre>

example: Template result of attribute which is unchecked
<pre>
<gmd:dateStamp>
<gco:Date>{{ md.datestamp }}{# tag="md.datestamp" , name="Metadata date" ,\
ref= "Part B 10.2" , desc= "Date that the metadata was created.",type = "date" ,
example = "2005-03-27" , multi= 0 , group= "Metadata" , multiline= False #}</gco:Date>
</gmd:dateStamp>
</pre>

Video shows record of simple practice of defining custom template
{{YouTube|Pu2sQoAJXfo}}

=== Editing metadata of multiple map selection ===
The advantages of this editing mode is in work using of defined template. This option can be used by selection of multiple maps in data catalog.

=== Validation of metadata profile ===
On the right side of editor is a validator of metadata. Currently, validator is able to validate two build-in profiles. For selection of profile “Load custom” is set by default the inspire profile.

=== Updating GRASS native metadata ===
The {{AddonCmd|g.gui.metadata}} module allows updating native GRASS metadata with using modules {{cmd|r.support}} and {{cmd|v.support}}. The intersection between ISO based metadata and r|v.support is limited by available parameters for these modules.

=== Publishing to csw server ===
Editor allows direct publishing of edited metadata by CSW(Catalog Service for Web) standard to remote catalogue server. Note that the remote catalogue must allows transaction.
[[File:Csw publisher.png|650px|center|publishing of metadata directly to remote csw server]]

=== Notes ===
* EPSG code must by added in url format. Information of projection is added only if EPSG code is known or if GDAL function for identifying from WKT is successful.
<source lang="xml">
<gmd:referenceSystemInfo>
<gmd:MD_ReferenceSystem>
<gmd:referenceSystemIdentifier>
<gmd:RS_Identifier>
<gmd:code>
<gco:CharacterString>http://www.opengis.net/def/crs/EPSG/0/3035</gco:CharacterString>
</gmd:code>
<gmd:codeSpace>
<gco:CharacterString>INSPIRE RS registry</gco:CharacterString>
</gmd:codeSpace>
</gmd:RS_Identifier>
</gmd:referenceSystemIdentifier>
</gmd:MD_ReferenceSystem>
</gmd:referenceSystemInfo>
</source>

= Creation of custom ISO based profile =
Creation of own custom ISO based profile is with using Jinja template syntax and some addition rules.
The main item of template is composite from two braces.
Both of them are fully according to syntax of Jinja template. First holds information about statements of loop or condition and also simple Python object. Second braces according to Jinja template syntax is represented by comment {# #}. In this braces additional information for generating graphical interface of editor are stored. In addition also information about individual metadata attributes (name, example, description, multiplicity, etc) is included.

== The first braces ==
Contain object of OWSLib which holds the metadata value. This values can be object which is describing single variable or list of variables. Value of some attribute can be defined￹ many times (according to ISO). This attributes must be defined in block of template with starting statements and ending mark of block. Only allowed loop statement is FOR (see jinja2 manual). The XML based text of block will by duplicated according to number of items in the list.

== Initialize of OWSLib objects ==
Most object are initialized in configuration of file which is stored in wx.metadat/config/init_md.txt

example: Default configuration file for the build-in Basic and Inspire profile:
<pre>
self.md.identification = MD_DataIdentification()
self.md.dataquality = DQ_DataQuality()
self.md.distribution = MD_Distribution()
self.md.identification.extent = EX_Extent()
self.md.identification.extent.boundingBox = EX_GeographicBoundingBox()
</pre>
# on the left side of equation is obligatory prefix 'self.md'. Postfix (name of variable) is open. On right side is OWSLib object

== Single element ==
In case if the value of metadata element is declared only once:
example: Identifier
<pre>
{{ md.identifier }}
</pre>

example: Lineage
<pre>
{{ md.dataquality.lineage }}
</pre>

== Loop or Condition statements ==
Loops and condition are defined in by starting braces and ending braces '{% -%}' .

Start of block:

example: for statement
<pre>
{% for co in md.contact -%}
</pre>

Editor allows to use Python 'zip' function and 'len' function. Note- in Jinja syntax is Python len(length) function defined specifically.

example: zip
<pre>
{% for (u,d) in zip(md.identification.uom,md.identification.distance) -%}
</pre>

example: len
<pre>
{% if md.identification.denominators|length > 0 -%}
</pre>

End of block is defined by:

example: if statement
<pre>
{% endif -%}
</pre>

example: for statement
<pre>
{% endfor -%}
</pre>

== The second braces ==
The second braces (by Jinja syntax comment) contains these parameters:
* tag - just copy of string from first braces
* object – for reading and holding metadata in python objects is used OWSLib library.

Most of object are initialized by configuration file wx.metadata/config/init_md.txt. Some metadata attributes are necessary initialized directly in this (object) parameter. Best example is the metadata contact object(organization,email) which can be initialized for many times.

example: responsible party
<pre>
{% for co in md.contact -%}{# object="CI_ResponsibleParty()" #}
{{ co.organization }}{# object="CI_ResponsibleParty()" #}
{{ co.email }}{# object="CI_ResponsibleParty()" #}
{{ co.role }}{# object="CI_ResponsibleParty()" #}
{% endfor -%}
</pre>

* name – free text string describing the name of metadata item
* ref – free additional information. By default should be reference of ISO attributes
* type – data type characteristic. Input text field validator for 'email' and 'date' is implemented.
* desc – free text which should describe or define the characteristic of metadata item
* example – free text with example of the value for metadata item
* group- label of the notebook list(tab) in editor.
* inbox- label of the box which includes attributes from the block of Jinja template. All items from one block of Jinja template must be in same box(same name).
* multi- 0 or 1. In case 0, the item cannot be duplicated= text field widget does not contain button for duplicating itself. Note that only OWSLib list based variables can be used with duplication button.
* inboxmulti- 0 or 1. In case 0, the items in box cannot be duplicated = box with items not dispose with button for cloning self.
* multiline – True = initialize text field with multilines editing. Suitable for longer texts input

example of second braces:
<pre>
{# tag="md.dataquality.lineage" , name="Lineage" , ref= "Part B 6.1" ,/
desc= "General explanation of the data producers knowledge about the lineage of a dataset."/
,example = "Product 1 scenes correspond to the path/row of the Landsat orbit.." ,/
type ="string" , group= "Quality and Validity" , multiline= True #}
</pre>

== Syntax of Jinja template system and editor custom parser ==
* both of braces of one metadata element must be on same line
* after '{# ','{%' must be white space. Also before '#}' '-%}'

== Examples of template ==

This video tutorial shows practice of creation custom ISO profile.
{{YouTube|ahl41mFD5w0}}

= Tools for working with metadata catalogue (csw) =
== Browsing and searching- g.gui.cswbrowser ==
Module {{AddonCmd|g.gui.cswbrowser}} (Browser) based graphical interface allows managing csw connections, setting filter and searching, browsing results and adding services to GRASS GIS.

=== Connection manager ===
Connection manager allows:
* adding
* removing
* loading list of connections from file.

By default the Browser is initialized by list of few well known csw providers of csw. Button "Service info" shows metadata of the service.

User can manage connection in two ways:
* First one is with using hidden storage provided by [http://wxpython.org/Phoenix/docs/html/ConfigBase.html wx.Config]. This case is without accessing default connection file.
* After prompt user also can add/remove connection from default .xml file.
[[File:ConnectionMgr.png|center|600px|Connection manager in g.gui.cswbrowser]]

=== Searching ===
Before fetching data from selected csw server is proper to set up searching filter. Searching can be restricted by:
* DCMI Type (Collection , Dataset , Event , Image , InteractiveResource , MovingImage , PhysicalObject , Service , Software , Sound , StillImage , Text)
* Area(bounding box)
** current GRASS GIS computation region
** manually
* Keywords
** basic filter allows to use basic or advance keyword filtering. In the simple case user can define single keywords or multiple keywords with button "+". Logic operator between keywords is AND(&).
** second, advanced is based on OGC list of expressions which means that can be set filter with logic relations between keywords or sets of keywords. Dialogue for settings keywords text string is under "Advanced" check-box. Syntax of constraints is based on python list syntax. Each keywords must be in braces
[[File:Find1.png|center|600px|g.gui.cswbrowser, searching page]]

==== Advanced keywords filtering ====
On the left panel of Browser is placed check-box named "Advance". The keywords advanced dialogue allows to use basic bool operations.

*OR condition
a || b || c
["a","b","c"]
*AND condition
a && b && c
[ ["a","b","c"] ]
*composition
(a && b) || c || d || e
[["a","b"],["c"],["d"],["e"]] or [["a","b"],"c","d","e"]

=== Browsing results===
On the bottom of Browser is a controller for browsing results. The data from server are fetched by number of items per one transaction. Default is a 20 items. After selection of item in the result table, metadata of record should be displayed. Browser automatically enables buttons on the top of Browser if WMS, WCS or WFS services are available.

=== Setting up csw server ===

== CSW server management ==
wx.metadata package includes tools for establishing and handling pycsw server. Pycsw stuff consists of:
* {{AddonCmd|db.csw.admin}}
* {{AddonCmd|db.csw.run}}
* {{AddonCmd|db.csw.harvest}}

For handling pycsw server can be use standard pythons scripts from pycsw install directory or mentioned three modules.

== Installation and configuration csw server ==
* Installation pycsw package - development version(OGC CSW server implementation written in Python):
e.g.to default path taken from configure file
<source lang="bash">
$ cd /var/www/pycsw
</source>

*deploying pycsw
**<source lang="bash">
$ git clone https://github.com/geopython/pycsw.git && cd pycsw
</source>
**<source lang="bash">
$ pip install -e . && pip install -r requirements-standalone.txt
</source>
* Create and adjust a configuration file:
**<source lang="bash">
$ cp default-sample.cfg default.cfg
</source>

* adjust parameters in ''default.cfg''
** '''server.home''' - path to installation directory
** '''repository.database''' - path to SQLite database e.g. $GISDBASE/WGS84/PERMANENT/sqlite/sqlite.db
** set '''server.url''' to: http://localhost:8000/

*Setup the database(creates tables, indexes, etc.)
**with unix command line(in pycsw install dir)
***<source lang="bash">
$ pycsw-admin.py -c load_records -f default.cfg -p /path/to/xml/
</source>
**with module db.csw.admin from GRASS GIS. Flag -s and parameter configure with path to configure file. Path to database is automatically taken from configure file.
***<source lang="bash">
db.csw.admin -s configure=/var/www/html/pycsw/default.cfg
</source>

*Load records:
**with unix command line(in pycsw install dir)
***<source lang="bash">
pycsw-admin.py -c load_records -f default.cfg -p /path/to/xml/
</source>
**db.csw.admin module with parameter -r for load recursively
***<source lang="bash">
db.csw.admin -r configure=/var/www/html/pycsw/default.cfg load_records=path/to/xml
</source>

*Run the server
**with unix command line(in pycsw install dir)
***<source lang="bash">
python csw.wsgi
</source>
**db.csw.admin module with parameter path and port
***<source lang="bash">
db.csw.run path=/var/www/html/pycsw port=8000
</source>

== Harvesting csw catalogue ==
Module {{AddonCmd|db.csw.harvest}} allows to copy data from source csw server to destination csw server.

= See also =
* [[GRASS Metadata Management]]
* [http://trac.osgeo.org/grass/wiki/GSoC/2014/MetadataForGRASS Development page 2014]
* [https://trac.osgeo.org/grass/wiki/GSoC/2015/ImprovedMetadata Development page 2015]

[[Category:Development]]
[[Category:Metadata]]
[[Category:INSPIRE]]

ISO/INSPIRE Metadata Support

2020-07-12T05:39:41Z

⚠️Tmszi: Add wx.metadata add-on dependencies installation note on Gentoo, Ubuntu GNU/Linux distribution

== ISO 19115:2003- Geographic information -- Metadata ==
[http://www.iso.org/iso/catalogue_detail.htm?csnumber=26020 ISO 19115:2003] defines the scheme required for description of geographic information and services. It provides information about the identification, the extent, the quality, the spatial and temporal scheme, spatial reference, and distribution of digital geographic data.

== Catalogue Service for the Web ==
[http://www.opengeospatial.org/standards/cat OGC Catalogue Services] support the ability to publish and search collections of descriptive information (metadata) for data, services, and related information objects. Metadata in catalogues represent resource characteristics that can be queried and presented for evaluation and further processing by both humans and software. Catalogue services are required to support the discovery and binding to registered information resources within an information community.

= About =
Wx.metadata which was developed during Google Summer of Code 2014/2015 is currently available in GRASS 7 add-ons: https://github.com/OSGeo/grass-addons/tree/master/grass7/gui/wxpython/wx.metadata
* The package for creating and editing metadata according to EN ISO 19115, described here, consists of few modules. The modules r.info.iso, v.info.iso and t.info.iso are command line based and support the conversion from the current native metadata management (r.info, v.info,t.info) into ISO based metadata. The GUI based module - g.gui.metadata offers advanced tools for management of metadata.

[[Image:Wxmetadata-editing.png|center|600px]]

* The package also includes the module g.gui.cswbrowser for searching and browsing metadata catalog (csw). For advanced users there are a few modules for establishing and handling pycsw server:
* db.csw.admin - module for administration of pycsw server
* db.csw.run - module for starting server on localhost
* db.csw.harvest - module for harvesting metadata between two csw.

= Requirements and installation =

'''Fedora Linux:'''

<source lang="bash">
sudo dnf install python3-sqlalchemy
sudo pip3 install pycsw
sudo pip3 install jinja2
</source>

'''Gentoo GNU/Linux:'''

<source lang="bash">
sudo emerge -av dev-python/owslib dev-python/reportlab dev-python/sqlalchemy dev-python/jinja dev-python/lxml dev-python/pygments
pip install --user pycsw
</source>

'''Ubuntu 20.04 LTS Focal Fossa GNU/Linux:'''

<source lang="bash">
sudo apt-get install python3-owslib python3-reportlab python3-sqlalchemy python3-jinja2 python3-lxml python3-pycsw python3-pygments
</source>

'''From source code:'''
* [http://grass.osgeo.org/download/software/ GRASS GIS 7.x ]
wx.metadata package has dependence on external libraries:

* [http://pycsw.org/ pycsw] package - development version(OGC CSW server implementation written in Python):
<source lang="bash">
git clone https://github.com/geopython/pycsw.git && cd pycsw
sudo pip install -e . && sudo pip install -r requirements-standalone.txt && cd ..
</source>

* [https://geopython.github.io/OWSLib/ OWSLib] package - development version (OGC CSW client implementation written in Python):
<source lang="bash">
git clone git://github.com/geopython/OWSLib.git
cd OWSLib && sudo python setup.py install && cd ..
</source>

* [http://jinja.pocoo.org/docs/dev/ Jinja2] (one of the most used template engines for Python)
<source lang="bash">
git clone https://github.com/mitsuhiko/jinja2.git
cd jinja2 && sudo python setup.py install && cd ..
</source>

* [http://pygments.org/ Pygments] for highlighting text(for Ubuntu below)
<source lang="bash">
sudo apt-get install python-pygments
</source>
* Additionally, python-reportlab is required. The name of the package could vary for your distro.

To install, when in GRASS, type
<source lang="bash">
g.extension wx.metadata
</source>

= Creating and editing metadata =
== Naming of metadata files and storage ==
The default location for exported metadata files is ''metadata'' directory in the map's mapset. For raster maps, the prefix derived from the current nomenclature is ''cell'', for vector maps ''vector''. File ends with .xml suffix.

For example, the default metadata file name for vector map "roads" is ''vector_roads.xml''.

== Metadata profile ==
The metadata profile defines attributes, which can be edited in this editor. The editor offers the already mentioned two profiles (Basic and [http://inspire.ec.europa.eu/index.cfm INSPIRE]). The editor also supports loading of customized profiles. See Creation of custom ISO based profile.

<big>Currently there are three build-in ISO profiles available: </big>
* '''The basic profile''' is substituted from intersection between items stored in GRASS native metadata format and INSPIRE profile. The intersect (subset) includes all available GRASS metadata. Metadata which cannot be assigned to ISO based attributes are stored in metadata attribute sabstractact.

* '''The INSPIRE profile''' fulfills the [http://inspire.ec.europa.eu/index.cfm/pageid/101 criteria of INSPIRE profile]. In case if is not possible to get the values from native GRASS metadata, these are filled with text string '$NULL'. This rule is applied for both profiles.

* '''The temporal profile''' is based on the GRASS basic profile and includes additional specific attributes for Temporal framework.

== Modules r.info.iso, v.info.iso t.info.iso ==
Command line based modules implements conversion of native GRASS metadata to ISO based XML metadata files. Identical configuration applies for both modules. Products of these modules differ partly in certain metadata attributes related to differences between raster (r.info) and vector (v.info) maps metadata.

'''Parameters of modules:'''
* map - selection of map for metadata creation
* profile - selection of metadata profile (basic or inspire)
* output - path to output XML metadata file. Default = according to naming&storage rules

== Metadata editor: g.gui.metadata ==
The metadata editor ([https://github.com/OSGeo/grass-addons/blob/master/grass7/gui/wxpython/wx.metadata/g.gui.metadata/g.gui.metadata.html addon on GitHub]) includes a graphical interface for converting metadata from r.info and v.info to ISO based metadata, editing metadata files and creating metadata templates. In addition, the graphical module allows validating of INSPIRE and Basic profile.

=== Information about metadata item ===
Tooltip function helps to fill in the form correctly.
[[Image:Wxmetadata-tooltip.png|center|350]]

=== Creation of metadata ===
The editor offers two editing modes. The first one allows to create metadata from selected GRASS maps from an active location. The second one allows editing of external metadata file.

=== Fetching values from database backend ===
The editor automatically fetches keywords from SQLite GRASS GIS backend. Three well know dictionaries are added by default. The user can manually add dictionaries by sql commands.
Dictionaries are stored in the table ''metadata_themes''

{| border="1" class="wikitable"
|+ teble metadta_themes
! Attribut
| title
| keyword
| date_iso
| date_type
|-
! Date type
| TEXT || TEXT|| TEXT|| TEXT
|-
! Example
|Example || GEMET - Concepts, version || 2010-01-13 || publication

|}

[[Image:Keywords.png|center|350]]

=== Exporting metadata reports to format PDF ===
This function allows to export metadata of GRASS maps to report in pdf format. The report template is based on INSPIRE profile. Unknown values of metadata attributes are filled by text $UNKNOWN.
[[Image:Pdfexp.png|center|350]]

{{YouTube|V7tgOOuRyuA}}

=== GRASS metadata map editor ===
[[Image:Wxmetadata-mapEdit.png|center|150px]]

Combo box for selection of ISO profile:
* basic
* inspire
* load custom – allows to load metadata profile from file. For example in case of using predefined template. The intersection of GRASS Basic profile(all available metadata for selected map) and loaded custom profile will proceed with this selection.

{{YouTube|7V5NyVdO05o}}

=== External metadata editor ===
[[Image:Wxmetadata-externalEdit.png|center|150]]
This editor mode allows to load own template and xml file for editing. This option does not support any connection with GRASS maps metadata.

{{YouTube|npMAavbwfi0}}

=== Defining templates ===
This function allows to create pattern of metadata profile. In other words it support defining fixed values to template. With using created template, the editor does not initialize texts inputs of defined values. These attributes is able to see only in Tree Browser in editor. Direct link on this function is multiple editing mode (see below).
* If check-box of metadata attribute is checked, the OWSLib object in template will be replaced by value. Unchecked attributes will not change this 'part' of the template.
* The yellow background of text fields indicates metadata attributes, which are not possible get from GRASS map information.
[[Image:Wxmetadata-template.png|center|600px]]

=== Usage ===
* If the check-box of metadata attribute is checked, the OWSLib object in template will be replaced by value. Unchecked attributes will not change this 'part' of the template.

example: Template result of attribute which is checked
<pre>
<gmd:dateStamp>
<gco:Date>2013-01-01</gco:Date>
</gmd:dateStamp>
</pre>

example: Template result of attribute which is unchecked
<pre>
<gmd:dateStamp>
<gco:Date>{{ md.datestamp }}{# tag="md.datestamp" , name="Metadata date" ,\
ref= "Part B 10.2" , desc= "Date that the metadata was created.",type = "date" ,
example = "2005-03-27" , multi= 0 , group= "Metadata" , multiline= False #}</gco:Date>
</gmd:dateStamp>
</pre>

Video shows record of simple practice of defining custom template
{{YouTube|Pu2sQoAJXfo}}

=== Editing metadata of multiple map selection ===
The advantages of this editing mode is in work using of defined template. This option can be used by selection of multiple maps in data catalog.

=== Validation of metadata profile ===
On the right side of editor is a validator of metadata. Currently, validator is able to validate two build-in profiles. For selection of profile “Load custom” is set by default the inspire profile.

=== Updating GRASS native metadata ===
The g.gui.metadata module allows updating native GRASS metadata with using modules r.support and v.support. The intersection between ISO based metadata and r|v.support is limited by available parameters for these modules.

=== Publishing to csw server ===
Editor allows direct publishing of edited metadata by CSW(Catalog Service for Web) standard to remote catalogue server. Note that the remote catalogue must allows transaction.
[[File:Csw publisher.png|650px|center|publishing of metadata directly to remote csw server]]

=== Notes ===
* EPSG code must by added in url format. Information of projection is added only if EPSG code is known or if GDAL function for identifying from WKT is succesful.
<source lang="xml">
<gmd:referenceSystemInfo>
<gmd:MD_ReferenceSystem>
<gmd:referenceSystemIdentifier>
<gmd:RS_Identifier>
<gmd:code>
<gco:CharacterString>http://www.opengis.net/def/crs/EPSG/0/3035</gco:CharacterString>
</gmd:code>
<gmd:codeSpace>
<gco:CharacterString>INSPIRE RS registry</gco:CharacterString>
</gmd:codeSpace>
</gmd:RS_Identifier>
</gmd:referenceSystemIdentifier>
</gmd:MD_ReferenceSystem>
</gmd:referenceSystemInfo>
</source>

= Creation of custom ISO based profile =
Creation of own custom ISO based profile is with using Jinja template syntax and some addition rules.
The main item of template is composite from two braces.
Both of them are fully according to syntax of Jinja template. First holds information about statements of loop or condition and also simple Python object. Second braces according to Jinja template syntax is represented by comment {# #}. In this braces additional information for generating graphical interface of editor are stored. In addition also information about individual metadata attributes (name, example, description, multiplicity, etc) is included.

== The first braces ==
Contain object of OWSLib which holds the metadata value. This values can be object which is describing single variable or list of variables. Value of some attribute can be defined￹￹ many times(according to ISO). This attributes must be defined in block of template with starting statements and ending mark of block. Only allowed loop statement is FOR(see jinja2 manual). The XML based text of block will by duplicated according to number of items in the list.

== Initialize of OWSLib objects ==
Most object are initialized in configuration of file which is stored in wx.metadat/config/init_md.txt
example: Default configuration file for the build-in Basic and Inspire profile:
<pre>
self.md.identification = MD_DataIdentification()
self.md.dataquality = DQ_DataQuality()
self.md.distribution = MD_Distribution()
self.md.identification.extent = EX_Extent()
self.md.identification.extent.boundingBox = EX_GeographicBoundingBox()
</pre>
# on the left side of equation is obligatory prefix 'self.md'. Postfix(name of variable) is open. On right side is OWSLib object

== Single element ==
In case if the value of metadata element is declared only once:
example: Identifier
<pre>
{{ md.identifier }}
</pre>

example: Lineage
<pre>
{{ md.dataquality.lineage }}
</pre>

== Loop or Condition statements ==
Loops and condition are defined in by starting braces and ending braces '{% -%}' .
Start of block:

example: for statement
<pre>
{% for co in md.contact -%}
</pre>

Editor allows to use Python 'zip' function and 'len' function. Note- in Jinja syntax is Python len(length) function defined specifically.

example: zip
<pre>
{% for (u,d) in zip(md.identification.uom,md.identification.distance) -%}
</pre>

example: len
<pre>
{% if md.identification.denominators|length > 0 -%}
</pre>

End of block is defined by:

example: if statement
<pre>
{% endif -%}
</pre>

example: for statement
<pre>
{% endfor -%}
</pre>
== The second braces ==
The second braces(by Jinja syntax comment) contains these parameters:
* tag - just copy of string from first braces
* object – for reading and holding metadata in python objects is used OWSLib library.

Most of object are initialized by configuration file wx.metadata/config/init_md.txt. Some metadata attributes are necessary initialized directly in this(object) parameter. Best example is the metadata contact object(organization,email) which can be initialized for many times.

example: responsible party
<pre>
{% for co in md.contact -%}{# object="CI_ResponsibleParty()" #}
{{ co.organization }}{# object="CI_ResponsibleParty()" #}
{{ co.email }}{# object="CI_ResponsibleParty()" #}
{{ co.role }}{# object="CI_ResponsibleParty()" #}
{% endfor -%}
</pre>

* name – free text string describing the name of metadata item
* ref – free additional information. By default should be reference of ISO attributes
* type – data type characteristic. Input text field validator for 'email' and 'date' is implemented.
* desc – free text which should describe or define the characteristic of metadata item
* example – free text with example of the value for metadata item
* group- label of the notebook list(tab) in editor.
* inbox- label of the box which includes attributes from the block of Jinja template. All items from one block of Jinja template must be in same box(same name).
* multi- 0 or 1. In case 0, the item cannot be duplicated= text field widget does not contain button for duplicating itself. Note that only OWSLib list based variables can be used with duplication button.
* inboxmulti- 0 or 1. In case 0, the items in box cannot be duplicated = box with items not dispose with button for cloning self.
* multiline – True = initialize text field with multilines editing. Suitable for longer texts input

example of second braces:
<pre>
{# tag="md.dataquality.lineage" , name="Lineage" , ref= "Part B 6.1" ,/
desc= "General explanation of the data producers knowledge about the lineage of a dataset."/
,example = "Product 1 scenes correspond to the path/row of the Landsat orbit.." ,/
type ="string" , group= "Quality and Validity" , multiline= True #}
</pre>

== Syntax of Jinja template system and editor custom parser ==
* both of braces of one metadata element must be on same line
* after '{# ','{%' must be white space. Also before '#}' '-%}'

== Examples of template ==
[http://trac.osgeo.org/grass/browser/grass-addons/grass7/gui/wxpython/wx.metadata/templates?order=date&desc=1 INSPIRE and Basic]

Video tutorial shows practice of creation custom ISO profile.
{{YouTube|ahl41mFD5w0}}

= Tools for working with metadata catalogue (csw) =
== Browsing and searching- g.gui.cswbrowser ==
Module g.gui.cswbrowser(Browser) based graphical interface allows managing csw connections, setting filter and searching, browsing results and adding services to GRASS GIS.
=== Connection manager ===
Connection manager allows:
* adding
* removing
* loading list of connections from file.
By default the Browser is initialized by list of few well known csw providers of csw. Button "Service info" shows metadata of the service.
User can manage connection in two ways:
* First one is with using hidden storage provided by [http://wxpython.org/Phoenix/docs/html/ConfigBase.html wx.Config]. This case is without accessing default connection file.
* After prompt user also can add/remove connection from default .xml file.
[[File:ConnectionMgr.png|center|600px|Connection manager in g.gui.cswbrowser]]

=== Searching ===
Before fetching data from selected csw server is proper to set up searching filter. Searching can be restricted by:
* DCMI Type (Collection , Dataset , Event , Image , InteractiveResource , MovingImage , PhysicalObject , Service , Software , Sound , StillImage , Text)
* Area(bounding box)
** current GRASS GIS computation region
** manually
* Keywords
** basic filter allows to use basic or advance keyword filtering. In the simple case user can define single keywords or multiple keywords with button "+". Logic operator between keywords is AND(&).
** second, advanced is based on OGC list of expressions which means that can be set filter with logic relations between keywords or sets of keywords. Dialogue for settings keywords text string is under "Advanced" check-box. Syntax of constraints is based on python list syntax. Each keywords must be in braces
[[File:Find1.png|center|600px|g.gui.cswbrowser, searching page]]
==== Advanced keywords filtering ====
On the left panel of Browser is placed check-box named "Advance". The keywords advanced dialogue allows to use basic bool operations.

*OR condition
a || b || c
["a","b","c"]
*AND condition
a && b && c
[ ["a","b","c"] ]
*composition
(a && b) || c || d || e
[["a","b"],["c"],["d"],["e"]] or [["a","b"],"c","d","e"]

=== Browsing results===
On the bottom of Browser is a controller for browsing results. The data from server are fetched by number of items per one transaction. Default is a 20 items. After selection of item in the result table, metadata of record should be displayed. Browser automatically enables buttons on the top of Browser if WMS, WCS or WFS services are available.

=== Setting up csw server ===

== CSW server management ==
wx.metadata package includes tools for establishing and handling pycsw server. Pycsw stuff consist:
*db.csw.admin
*db.csw.run
*db.csw.harvest

For handling pycsw server can be use standard pythons scripts from pycsw install directory or mentioned three modules.

== Installation and configuration csw server ==
* Installation pycsw package - development version(OGC CSW server implementation written in Python):
e.g.to default path taken from configure file
<source lang="bash">
$ cd /var/www/pycsw
</source>

*deploying pycsw
**<source lang="bash">
$ git clone https://github.com/geopython/pycsw.git && cd pycsw
</source>
**<source lang="bash">
$ pip install -e . && pip install -r requirements-standalone.txt
</source>
* Create and adjust a configuration file:
**<source lang="bash">
$ cp default-sample.cfg default.cfg
</source>

* adjust parameters in ''default.cfg''
** '''server.home''' - path to installation directory
** '''repository.database''' - path to SQLite database e.g. $GISDBASE/WGS84/PERMANENT/sqlite/sqlite.db
** set '''server.url''' to: http://localhost:8000/

*Setup the database(creates tables, indexes, etc.)
**with unix command line(in pycsw install dir)
***<source lang="bash">
$ pycsw-admin.py -c load_records -f default.cfg -p /path/to/xml/
</source>
**with module db.csw.admin from GRASS GIS. Flag -s and parameter configure with path to configure file. Path to database is automatically taken from configure file.
***<source lang="bash">
db.csw.admin -s configure=/var/www/html/pycsw/default.cfg
</source>

*Load records:
**with unix command line(in pycsw install dir)
***<source lang="bash">
pycsw-admin.py -c load_records -f default.cfg -p /path/to/xml/
</source>
**db.csw.admin module with parameter -r for load recursively
***<source lang="bash">
db.csw.admin -r configure=/var/www/html/pycsw/default.cfg load_records=path/to/xml
</source>

*Run the server
**with unix command line(in pycsw install dir)
***<source lang="bash">
python csw.wsgi
</source>
**db.csw.admin module with parameter path and port
***<source lang="bash">
db.csw.run path=/var/www/html/pycsw port=8000
</source>

== Harvesting csw catalogue ==
Module db.csw.harvest allows to copy data from source csw server to destination csw server

= See also =
* [[GRASS Metadata Management]]
* [http://trac.osgeo.org/grass/wiki/GSoC/2014/MetadataForGRASS Development page 2014]
* [https://trac.osgeo.org/grass/wiki/GSoC/2015/ImprovedMetadata Development page 2015]

[[Category:Development]]
[[Category:Metadata]]
[[Category:INSPIRE]]