Toolboxes

New toolbox approach discussed at the GRASS Community Code Sprint 2012

At the community sprint in Prague 2012 the toolbox approach with different repositories as well as the approach using "--with-toolbox/--without-toolbox" configure statements at compile time have been discussed. The major drawback of these solutions is the their complexity and the problem of cross dependencies between modules.

A new approach has been suggested using XML files to describe toolboxes and special directory structure to hide unneeded modules from the user.

The idea is to have a binary directory containing only modules that are listed in a XML file that defines what core modules should be available and what toolboxes should be used. The XML file will be used by the GUI to create menu listings for the selected modules. All modules that are deactivated in the XML file will be held in a directory that is not available in the users binary path, but in the binary search path of modules. Hence unneeded modules are invisible at the command line and in the GUI, but still available for modules that might call other modules.

New toolboxes and core modules can be simply added to the search path and the GUI by editing the XML definition and calling g.extension. The module g.extension will simply links/copy the hidden modules into the binary directory that is visible by the user or move deselected modules in the hidden directory. The GUI will always analyzes the XML toolbox configuration file for menu generation.

The configuration XML file will include a list of core modules for general, raster, image, voxel and vector processing as well as the definition of toolboxes. Toolboxes are defined by their type (raster, vector, ...) a short description and a list of modules. Modules can be defined in several different toolboxes.

The GRASS development team should provide several pre-defined XML toolbox configuration files as for hydrological modeling, vector processing, terrain modeling, image analysis, ..., and WPS server.

Using this approach the repository structure must not be changed. The build system has to be modified to use the XML toolbox file to decide where to copy the binary modules and scripts. g.extension has to be extended to link/copy selected modules in the users binary search path and to remove modules from them. GRASS will be build as usual.

Original approach

• See original post by Jarosław Jasiewicz

• Discussion about the suitability of the proposed change on the talk side of this wiki page; discussion of the possible method on this side of the page.

Hi all!

That rather radical ideas I present here are rather for future, at least for GRASS 8,
but I'd like present it now for long-term reflection.

Probably all notice that for over two years there is big increase in add-on repository
(including me). There are modules of different quality: from fully GRASS toolsets,
to shell or python scripts, from  actively developed tools to abandoned,
from all-purpose tools to very specialized etc. I also think that that activity
will be grown due to substitute shell script by python

Similar situation is in main GRASS branch: there are modules for all like conversion tools,
interpolation methods, georeferencing etc, and very specialized modules for very limited
group of users (like wild fire), there are also some modules out of date.

I'm not enthusiastic about moving new modules into main branch. Almost every module has
different coding style and it will lasting in future that GRASS would be difficult to maintain.
On the other hand some people complains that some interesting modules are only available as
add-ons (I assume for some reasons they cannot install it)

So my suggestion is to rearrange future GRASS form two layers (main branch/add-on) into
three layers architecture:

1) GRASS core layer: much limited limited than now, only GIS environment and basic,
all-puropse tools, slow changes, great stability
2) GRASS toolset layer: oficcial GRASS thematic tools and toolsets (like terrain analysis,
hydrological analysis, photo-interpretation, landscape analysis etc,) every toolset with its
maintainer, rapid development, new ready to use tools after quality control may appear here,
also some of current main branch tool shall be moved to that layer
3) GRASS community layer:  everything else like experimental, actively development new tools,
that what do not pass quality control, simple scripts, etc....

What benefits:
for developers and contributors: much clear situation and better publication path.
Toolset layer should be much more open for new tools than current GRASS main branch

for users: faster access to new tools.
There is no doubt that new tools are faster developed (less risk) than GRASS core
Binaries with toolsets could be maintained as separate apt/urpmi/pacman/yum/exe etc packages,
so it may appear in linux repository separetly form GRASS core.

There is only loose ideas. Most of them are of course taken from R (core/toolsets/rest of packages;
separate core and package development) but I think it is worth of some discuss ...

regards
Jarek

Proposed toolboxes

Toolbox is a topic-based set of modules used for solving common GIS analysis, like hydrology, networking, LRS, etc.

Managing toolboxes

g.extension is designed to manage locally installed GRASS Addons. It allows to install single addons modules and also toolboxes (flag -t). Note that toolbox support is experimental.

• list available toolboxes

g.extension -tl

List of available extensions (toolboxes):
Hydrology (HY)
* r.stream.basins
* r.stream.channel
* r.stream.distance
* r.stream.extract
* r.stream.order
* r.stream.segment
* r.stream.slope
* r.stream.snap
* r.stream.stats
...

• list available modules from Addons

g.extension -l

List of available extensions (modules):
...
r.stream.slope
...

• install new toolbox ("Hydrology")

g.extension -t extension=HY

• install single module from "Hydrology" toolbox

g.extension extension=r.stream.slope

Module g.extension also allows to list installed extensions

• toolboxes

g.extension -ta

List of installed extensions (toolboxes):
HY

• or modules

g.extension -a

List of installed extensions (modules):
r.stream.basins
r.stream.channel
r.stream.distance
r.stream.extract
r.stream.order
r.stream.segment
r.stream.slope
r.stream.snap
r.stream.stats

• remove locally installed toolbox

g.extension -ft extension=HY operation=remove

• remove single module

g.extension -f extension=r.stream.slope operation=remove

List of toolboxes

Toolbox name	Toolbox code	Description	Correlate toolbox
3D Raster	R3	All `r3.*` modules
3D Visualization	V3	nviz_cmd
Atmospheric tools	AT	Subset of image processing tool
Cartography	CA	ps.map, ps.output
Database	DB	All `db.*` modules	Vector analysis
Geostatistcal	GS	v.krige maybe other
GUI	GI	wxGUI
Hydrology	HY	r.watershed, r.terraflow, `r.stream.*` modules in addons, maybe other, r.flow, r.sim.water, r.sim.sediment	Raster analysis
Image Processing	IM	All `i.*` modules, selected r.* modules
Interoperability tools	IO	All the various import/export modules except for GDAL/OGR	Raster analysis, Vector analysis, Image Processing, Database
Landscape analysis	LA	All `r.li.*` modules, maybe other	Raster analysis
Lidar analysis	LI	All `v.lidar.*` modules, v.surf.bspline, r.in.xyz
Linear Reference Systems	LS	All `v.lrs.*` modules	Vector analysis
Network analysis	NA	All `v.net.*` modules, maybe other	Vector analysis
Raster analysis	RA	A lot of basic `r.*` modules (like r.buffer, r.reclass, r.fillnull...)
Terrain analysis	TA	Different kind of analysis: soil (r.sun,), terrain (r.slope.aspect, r.los), geomorphometry (r.param.scale)	Raster analysis
Vector analysis	VA	A lot of basic `v.*` modules (like v.buffer, v.centroid, v.overlay...)	Database

List of modules (trunk)

Display

Module	Core/Tools/Addons	Toolbox(es)
d.barscale	?
d.colorlist	?
d.colortable	?
d.correlate	?
d.erase	?
d.font	?
d.fontlist	?
d.geodesic	?
d.graph	?
d.grid	?
d.his	?
d.histogram	?
d.info	?
d.labels	?
d.legend	?
d.linegraph	?
d.nviz	?
d.path	?
d.polar	?
d.profile	?
d.rast	?
d.rast.arrow	?
d.rast.edit	?
d.rast.leg	?
d.rast.num	?
d.rgb	?
d.rhumbline	?
d.shadedmap	?
d.text	?
d.thematic.area	?
d.title	?
d.vect	?
d.vect.chart	?
d.vect.thematic	?
d.where	?

Database

Module	Core/Tools/Addons	Toolbox(es)
db.columns	?
db.connect	?
db.copy	?
db.createdb	?
db.databases	?
db.describe	?
db.drivers	?
db.dropcolumn	?
db.dropdb	?
db.droptable	?
db.droptable	?
db.execute	?
db.in.ogr	?
db.login	?
db.out.ogr	?
db.select	?
db.tables	?
db.test	?

General

Module	Core/Tools/Addons	Toolbox(es)
g.access	C
g.cairocomp	C
g.copy	C
g.dirseps	C
g.extension	C
g.filename	C
g.findetc	C
g.findfile	C
g.gisenv	C
g.gui	C
g.list	C
g.manual	C
g.mapset	C
g.mapsets	C
g.message	C
g.mkfontcap	C
g.mlist	C
g.mremove	C
g.parser	C
g.pnmcomp	C
g.ppmtopng	C
g.proj	C
g.region	C
g.remove	C
g.rename	C
g.tempfile	C
g.transform	C
g.version	C

Imagery

Module	Core/Tools/Addons	Toolbox(es)
i.albedo	T	IM
i.atcorr	C
i.biomass	T	IM
i.cca	T	IM
i.cluster	C
i.eb.eta	T	IM
i.eb.evapfr	T	IM
i.eb.h_sebal01	T	IM
i.eb.soilheatflux	T	IM
i.emissivity	T	IM
i.evapo.time	T	IM
i.fft	C
i.fusion.brovey	C
i.gensig	C
i.gensigset	C
i.group	C
i.his.rgb	C
i.ifft	C
i.image.mosaic	C
i.in.spotvgt	T	IO
i.landsat.rgb	C
i.latlong	T	IM
i.maxlik	C
i.modis.qc	T	IM
i.oif	T	IM
i.pca	C
i.rectify	C
i.rgb.his	C
i.smap	C
i.spectral	T	IM
i.sunhours	T	IM
i.target	C
i.tasscap	T	IM
i.vi	T	IM
i.zc	T	IM

Misc

Module	Core/Tools/Addons	Toolbox(es)
m.cogo	?
m.measure	?
m.proj	?

Postscript

Module	Core/Tools/Addons	Toolbox(es)
ps.map	C

Raster

Module	Core/Tools/Addons	Toolbox(es)
r.basins.fill	T	TA
r.bitpattern	T	IM
r.blend	C
r.buffer	C
r.buffer2	C
r.carve	T	TA
r.category	C
r.circle	T	RA
r.clump	C
r.coin	T	RA
r.colors	C
r.colors.out	T	RA
r.colors.stddev	T	RA
r.composite	T	IM
r.compress	C
r.contour	C
r.cost	T	RA
r.covar	T	RA
r.cross	T	RA
r.describe	C
r.distance	T	RA
r.drain	T	RA
r.external	C
r.external.out	C
r.fill.dir	T	HY
r.fillnulls	T	RA
r.flow	T	HY
r.grow	C
r.grow.distance	C
r.gwflow	T	HY
r.his	T	IM
r.horizon	T	TA
r.in.arc	T	IO
r.in.ascii	T	IO
r.in.aster	T	IO
r.in.bin	T	IO
r.in.gdal	C
r.in.gridatb	T	IO
r.in.mat	T	IO
r.in.png	T	IO
r.in.poly	T	IO
r.in.srtm	T	IO
r.in.wms	T	IO
r.in.xyz	C
r.info	C
r.kappa	T	RA
r.lake	T	HY
r.li.cwed	T	LA
r.li.dominance	T	LA
r.li.edgedensity	T	LA
r.li.mpa	T	LA
r.li.mps	T	LA
r.li.padcv	T	LA
r.li.padrange	T	LA
r.li.padsd	T	LA
r.li.patchdensity	T	LA
r.li.patchnum	T	LA
r.li.richness	T	LA
r.li.shannon	T	LA
r.li.shape	T	LA
r.li.simpson	T	LA
r.los	T	TA
r.mapcalc	C
r.mask	C
r.mfilter	T	RA
r.mode	T	RA
r.neighbors	C
r.null	C
r.out.arc	T	IO
r.out.ascii	T	IO
r.out.bin	T	IO
r.out.gdal	C
r.out.gridatb	T	IO
r.out.mat	T	IO
r.out.mpeg	T	IO
r.out.png	T	IO
r.out.pov	T	IO
r.out.ppm	T	IO
r.out.ppm3	T	IO
r.out.tiff	T	IO
r.out.vrml	T	IO
r.out.vtk	T	IO
r.out.xyz	C
r.param.scale	T	TA
r.patch	C
r.plane	T	RA
r.profile	T	TA
r.proj	C
r.quant	C
r.quantile	T	RA
r.random	T	RA
r.random.cells	T	RA
r.random.surface	T	RA
r.reclass	C
r.reclass.area	C
r.recode	C
r.region	C
r.regression.line	T	RA
r.report	C
r.resamp.bspline	C
r.resamp.filter	C
r.resamp.interp	C
r.resamp.rst	C
r.resamp.stats	C
r.resample	C
r.rescale	C
r.rescale.eq	C
r.ros	T	RA
r.series	C
r.shaded.relief	C
r.sim.sediment	T	HY
r.sim.water	T	HY
r.slope.aspect	T	TA
r.solute.transport	T	HY
r.spread	T	RA
r.spreadpath	T	RA
r.statistics	C
r.statistics2	C
r.statistics3	C
r.stats	C
r.sun	T	TA
r.sunmask	T	TA
r.support	C
r.support.stats	C
r.surf.area	T	TA
r.surf.contour	T	TA
r.surf.fractal	T	TA
r.surf.gauss	T	RA
r.surf.idw	T	RA
r.surf.idw2	T	RA
r.surf.random	T	RA
r.terraflow	T	HY
r.texture	T	RA
r.thin	C
r.tileset	?
r.timestamp	C
r.to.rast3	C
r.to.rast3elev	C
r.to.vect	C
r.topidx	T	HY
r.topmodel	T	HY
r.transect	T	RA
r.univar	C
r.uslek	T	HY
r.usler	T	HY
r.volume	T	RA
r.walk	T	TA
r.water.outlet	T	HY
r.watershed	T	HY
r.what	C
r.what.color	C

3D raster (voxel)

Module	Core/Tools/Addons	Toolbox(es)
r3.cross.rast	T	3D Raster
r3.gwflow	T	3D Raster
r3.in.ascii	T	3D Raster
r3.in.v5d	T	3D Raster
r3.info	T	3D Raster
r3.mapcalc	T	3D Raster
r3.mask	T	3D Raster
r3.mkdspf	T	3D Raster
r3.null	T	3D Raster
r3.out.ascii	T	3D Raster
r3.out.v5d	T	3D Raster
r3.out.vtk	T	3D Raster
r3.stats	T	3D Raster
r3.timestamp	T	3D Raster
r3.to.rast	T	3D Raster
r3.univar	T	3D Raster

Vector

Module	Core/Tools/Addons	Toolbox(es)
v.autokrige	?
v.buffer	C
v.build	C
v.build.all	C
v.build.polylines	C
v.category	C
v.centroids	C
v.class	C
v.clean	C
v.colors	C
v.convert	T	IO
v.convert.all	T	IO
v.db.addcolumn	?
v.db.addtable	?
v.db.connect	?
v.db.dropcolumn	?
v.db.droprow	?
v.db.droptable	?
v.db.join	?
v.db.reconnect.all	?
v.db.renamecolumn	?
v.db.select	?
v.db.univar	?
v.db.update	?
v.delaunay	?
v.dissolve	C
v.distance	?
v.drape	?
v.edit	C
v.external	C
v.extract	C
v.extrude	?
v.generalize	?
v.hull	?
v.in.ascii	T	IO
v.in.db	C
v.in.dxf	T	IO
v.in.e00	T	IO
v.in.geonames	T	IO
v.in.gns	T	IO
v.in.lines	T	IO
v.in.mapgen	T	IO
v.in.ogr	C
v.in.postgis	T	IO
v.in.region	C
v.in.sites	T	IO
v.in.sites.all	T	IO
v.in.wfs	T	IO
v.info	C
v.kcv	?
v.kernel	?
v.krige	?
v.label	?
v.lidar.correction	?
v.lidar.edgedetection	?
v.lidar.growing	?
v.lrs.create	?
v.lrs.label	?
v.lrs.segment	?
v.lrs.where	?
v.mkgrid	?
v.neighbors	?
v.net	?
v.net.alloc	?
v.net.allpairs	?
v.net.bridge	?
v.net.centrality	?
v.net.components	?
v.net.connectivity	?
v.net.distance	?
v.net.flow	?
v.net.iso	?
v.net.path	?
v.net.salesman	?
v.net.spanningtree	?
v.net.steiner	?
v.net.timetable	?
v.net.visibility	?
v.normal	?
v.out.ascii	T	IO
v.out.dxf	T	IO
v.out.gps	T	IO
v.out.ogr	C
v.out.postgis	T	IO
v.out.pov	T	IO
v.out.svg	T	IO
v.out.vtk	T	IO
v.outlier	?
v.overlay	?
v.parallel	?
v.patch	?
v.perturb	?
v.proj	C
v.qcount	?
v.random	?
v.rast.stats	?
v.reclass	?
v.report	?
v.sample	?
v.segment	?
v.select	?
v.split	?
v.support	C
v.surf.bspline	?
v.surf.idw	?
v.surf.rst	?
v.to.3d	?
v.to.db	?
v.to.points	?
v.to.rast	?
v.to.rast3	?
v.transform	?
v.type	?
v.univar	?
v.vol.rst	?
v.voronoi	?
v.what	?
v.what.rast	?
v.what.vect	?

Various

Module	Core/Tools/Addons	Toolbox(es)
nviz	?
nviz_cmd	?
wximgview	?
wxpyimgview	?
xganim	?
ximgview	?

List of selected AddOns modules

See GRASS AddOns for full list of available modules. Feel free to extend this list.

Module	Core/Tools/Addons	Toolbox(es)	Note
r.stream.channel	T	HY	√
r.stream.distance	T	HY	√
r.stream.extract	T	HY	√
r.stream.order	T	HY	√
r.stream.segment	T	HY	√
r.stream.slope	T	HY	√
r.stream.snap	T	HY	√
r.stream.stats	T	HY	√
r.seg	T	IM	It could be rename to `i.seg`
i.topocorr	T	IM
r.diversity	T	LA
Ps.output	?
...	?

How to implement

Proposal 1

• add html tag <param> in the description.html with these options:

<param name = "core"> for core module
<param name = "tool" value="tool_code/codes"> for toolbox
<param name = "addons" [value="tool_code/codes"]> for addons, value is optional

• create a script to generate a xml file for each toolbox
• add toolbox option in g.extension to read the xml and install the modules

Proposal 2

• modules (currently only Addons) are described (name, description, keywords, files) in metadata XML file called modules.xml - this file is auto-generated, see trac page for details
• similarly toolboxes are described (name, code, correlate modules, list of modules) in toolboxes.xml - this file is manually manageable via SVN

g.extension reads (from remote server http://grass.osgeo.org/addons/grass7) modules.xml when listing available modules (-l), and toolboxes.xml when listing available toolboxes (-lt). When installing toolbox using g.extension the module reads toolboxes.xml and installs in the loop given list of modules. When all modules are installed it also updates local metadata files - GRASS_ADDON_BASE/modules.xml and GRASS_ADDON_BASE/toolboxes.xml. These local metadata files are checked when listing locally installed toolboxes or single modules (-a or -ta).

Proposal 3

Use a modified version of pip (a tool for installing and managing Python packages)

With Pip you can add/change repository, compile C/Fortran module, manage dependencies, install from some version control system (use URLs like hg+http://domain/repo -- or prefix git+, svn+ etc), and using virtualenv