GRASS-Wiki - User contributions [en]

Documentation and Education Working Group

2023-10-30T17:47:21Z

⚠️El selvaje: /* Members */

Documentation and education working group is established to coordinate the development of documentation and education materials to meet the needs of a wide range of GRASS users.

== Scope ==
* coordinate development of documentation and education materials
* coordinate development of sample datasets and localized datasets
* decide on best formats and place for documentation, tutorials
* develop onboarding materials for contributors of documentation and education materials

== Members ==
Let us know who you are and what are your interests and topics you want to discuss and develop as part of this group.

; [https://wiki.osgeo.org/wiki/User:Helena Helena Mitasova] (working group coordinator)
: Professor at North Carolina State University
: Interested in wiki pages cleanup, manual pages updates, integration of GRASS in courses, localized data sets

; [https://wiki.osgeo.org/wiki/Anna_Petrasova Anna Petrasova]
: Researcher, educator and software developer at North Carolina State University
: Topics of interest include developing tutorials and workshops in Jupyter Notebooks, onboarding materials

; Eric Patton
:

; Florian Betz
:

; Hernán De Angelis
:

; Micha Silver
:

; Titus Kiprutto
:

; Paulo van Breugel
:

; Brendan Harmon
:

; [https://spatial-ecology.net/docs/build/html/COURSETRAINERS/trainers.html#giuseppe-amatulli-phd:Giuseppe Amatulli]
: Research scientist at School of the Environment, Yale University. Educator at Spatial Ecology
: Topics of interest include developing tutorials and workshops in GRASS

; Michael Barton
:

; Massimo Di Stefano
:

; Carlos Grohmann
:

To be added to this working group, please contact the group coordinator.

== Communication Channels ==
* [https://lists.osgeo.org/mailman/listinfo/grass-user grass-user] and [https://lists.osgeo.org/mailman/listinfo/grass-dev grass-dev] mailing lists
* [https://github.com/OSGeo/grass OSGeo/grass GitHub issues and pull requests] and [https://github.com/orgs/OSGeo/projects/3 GitHub project]
* [https://app.gitter.im/#/room/#grassgis_documentation:gitter.im grassgis/documentation gitter room]

[[Category:Community]]
[[Category:Working_Groups]]

Documentation and Education Working Group

2023-10-30T17:46:47Z

⚠️El selvaje: /* Members */

Documentation and education working group is established to coordinate the development of documentation and education materials to meet the needs of a wide range of GRASS users.

== Scope ==
* coordinate development of documentation and education materials
* coordinate development of sample datasets and localized datasets
* decide on best formats and place for documentation, tutorials
* develop onboarding materials for contributors of documentation and education materials

== Members ==
Let us know who you are and what are your interests and topics you want to discuss and develop as part of this group.

; [https://wiki.osgeo.org/wiki/User:Helena Helena Mitasova] (working group coordinator)
: Professor at North Carolina State University
: Interested in wiki pages cleanup, manual pages updates, integration of GRASS in courses, localized data sets

; [https://wiki.osgeo.org/wiki/Anna_Petrasova Anna Petrasova]
: Researcher, educator and software developer at North Carolina State University
: Topics of interest include developing tutorials and workshops in Jupyter Notebooks, onboarding materials

; Eric Patton
:

; Florian Betz
:

; Hernán De Angelis
:

; Micha Silver
:

; Titus Kiprutto
:

; Paulo van Breugel
:

; Brendan Harmon
:

; ["https://spatial-ecology.net/docs/build/html/COURSETRAINERS/trainers.html#giuseppe-amatulli-phd":Giuseppe Amatulli]
: Research scientist at School of the Environment, Yale University. Educator at Spatial Ecology
: Topics of interest include developing tutorials and workshops in GRASS

; Michael Barton
:

; Massimo Di Stefano
:

; Carlos Grohmann
:

To be added to this working group, please contact the group coordinator.

== Communication Channels ==
* [https://lists.osgeo.org/mailman/listinfo/grass-user grass-user] and [https://lists.osgeo.org/mailman/listinfo/grass-dev grass-dev] mailing lists
* [https://github.com/OSGeo/grass OSGeo/grass GitHub issues and pull requests] and [https://github.com/orgs/OSGeo/projects/3 GitHub project]
* [https://app.gitter.im/#/room/#grassgis_documentation:gitter.im grassgis/documentation gitter room]

[[Category:Community]]
[[Category:Working_Groups]]

Documentation and Education Working Group

2023-10-30T17:39:57Z

⚠️El selvaje: /* Members */

Documentation and education working group is established to coordinate the development of documentation and education materials to meet the needs of a wide range of GRASS users.

== Scope ==
* coordinate development of documentation and education materials
* coordinate development of sample datasets and localized datasets
* decide on best formats and place for documentation, tutorials
* develop onboarding materials for contributors of documentation and education materials

== Members ==
Let us know who you are and what are your interests and topics you want to discuss and develop as part of this group.

; [https://wiki.osgeo.org/wiki/User:Helena Helena Mitasova] (working group coordinator)
: Professor at North Carolina State University
: Interested in wiki pages cleanup, manual pages updates, integration of GRASS in courses, localized data sets

; [https://wiki.osgeo.org/wiki/Anna_Petrasova Anna Petrasova]
: Researcher, educator and software developer at North Carolina State University
: Topics of interest include developing tutorials and workshops in Jupyter Notebooks, onboarding materials

; Eric Patton
:

; Florian Betz
:

; Hernán De Angelis
:

; Micha Silver
:

; Titus Kiprutto
:

; Paulo van Breugel
:

; Brendan Harmon
:

; Giuseppe Amatulli
: Research scientist at School of the Environment, Yale University. Educator at Spatial Ecology
: Topics of interest include developing tutorials and workshops in GRASS

; Michael Barton
:

; Massimo Di Stefano
:

; Carlos Grohmann
:

To be added to this working group, please contact the group coordinator.

== Communication Channels ==
* [https://lists.osgeo.org/mailman/listinfo/grass-user grass-user] and [https://lists.osgeo.org/mailman/listinfo/grass-dev grass-dev] mailing lists
* [https://github.com/OSGeo/grass OSGeo/grass GitHub issues and pull requests] and [https://github.com/orgs/OSGeo/projects/3 GitHub project]
* [https://app.gitter.im/#/room/#grassgis_documentation:gitter.im grassgis/documentation gitter room]

[[Category:Community]]
[[Category:Working_Groups]]

GRASS GIS Performance

2021-02-27T14:00:05Z

⚠️El selvaje: add Float32 range-bound

== GRASS GIS Performance ==

GRASS GIS is noted for being ready for massive data analysis. This page contains an yet incomplete collection of performance indicators.

=== Architecture ===

GRASS GIS is fully 32bit and 64bit compliant. See also the [[Software requirements specification]].

=== Search strategies used in processing geodata ===

GRASS GIS makes heavy use of search trees in order to speed up computation:
* segment lib: btree2
* 2D splines (RST): quadtree
* 3D splines (RST): octree
* vector lib topology: R*-tree

See the [http://grass.osgeo.org/programming7/ Programmer's manual] for details.

=== Number of opened input files ===

There are only operating system constraints of the number of input files which can be opened simultaneously. Commonly the limit is 1024 files. In operating systems like Linux this limit can be overcome with the "ulimit" settings.

See also
* {{cmd|r.series}}
* {{cmd|r.series.accumulate}}

=== Memory management ===

Due to the modular architecture of GRASS GIS the overhead of the software itself is minimal.

'''Raster data operations''': where appropriate, modules offer a parameter to optimize caching ("memory" parameter).
# Pixel based operations: they have very low impact on memory usage.
# Moving window based operations: they have medium impact on memory usage.
# Full map operations (watersheds, cost surfaces, etc.): they have high impact on memory usage.
# Statistical operations: while univariate statistics have low impact on memory usage, quartiles and other aggregated statistics have medium impact on memory usage.

'''Vector data operations''':
# Vector point operations: memory consumption depends on the amount of points. LiDAR data processing is commonly demanding. For some operations the creation of topology can be skipped to reduce the memory footprint.
# Vector line operations: they have low impact on memory usage (depends on the amount of data).
# Vector area/faces operations: they have high impact on memory usage.
# Topological versus non-topological operations: a subset of vector modules is able to operate on point vector maps without topology which saves notably RAM usage.

'''Database operations''':
* Most operations are simply SQL transactions with low impact on memory usage.

'''See also'''
* Solving [[Memory issues]] when dealing with large amounts of data

=== Vector management ===

A "vector map" is a data layer consisting of a number of sparse features in geographic space. These might be data points (drill sites), lines (roads), polygons (park boundary), volumes (3D CAD structure), or some combination of all these. Typically each feature in the map will be tied to a set of attribute layers stored in a database (road names, site ID, geologic type, etc.). As a general rule these can exist in 2D or 3D space and are independent of the GIS's computation region.

See also {{cmd|vectorintro}} in the manual.

==== Vector geometry ====

In all GRASS GIS versions,

* with topology the feature limit is at time 2^31 - 1 (about 2 billion) features per vector map.
* ''TODO: add limit if topology creation is disabled at import for points (e.g., LiDAR points).''

==== Vector attribute management ====

Attributes are managed through a SQL interface (see also {{cmd|databaseintro}})

The default database backend is
* DBF files (tend to be slow) in GRASS GIS 6 ({{cmd|grass-dbf|version=64}})
* SQLite file (very fast compared to DBF in GRASS GIS 7 ({{cmd|grass-sqlite}})

Other SQL backends are offered as well including PostgreSQL, MySQL, etc.: see {{cmd|sql}} support in GRASS GIS.

'''Speed of DBF versus SQLite drivers''': attribute operations which take hours using the DBF backend just take seconds using the SQLite backend.

==== Maximum Number of Attribute Columns ====

The maximum number of attribute columns of a table connected to a vector map is defined by the capabilities of the the selected database backend (set with {{cmd|db.connect}}).

* '''DBF-Backend''': GRASS 4.x - 6.x use by default the DBF backend. While there is no explicitly stated maximum number of allowed attribute columns, Web sources report a maximum '''between 128 and 1023/24'''. Trials with GRASS 6.4.2 in 2012 result in write failure if > 2000 attribute columns are used. Export to DBF-based ESRI Shapefile provides a warning if more that '''255''' attributes are used: Other software tools may ignore all further attributes, hence a maximum of '''128''' columns may be prudent.
* '''SQLite-Backend''': GRASS 7.x uses by default the SQLite backend. The default maximum number of attribute columns is '''2000''' according to the [http://www.sqlite.org/limits.html#max_column specifications]. This number can be increased by compiling SQlite with changed settings.
* '''MySQL-Backend''': The default maximum number of attribute columns is '''4096''' according to the [http://wiki.postgresql.org/wiki/FAQ#What_is_the_maximum_size_for_a_row.2C_a_table.2C_and_a_database.3F specifications].
* '''PostgreSQL-Backend''': The default maximum number of attribute columns is '''250-1600''' according to the [http://dev.mysql.com/doc/refman/5.1/en/column-count-limit.html specifications] depending on column types.
* '''Oracle-Backend''': The default maximum number of attribute columns is '''1000''' according to the [http://docs.oracle.com/cd/B19306_01/server.102/b14237/limits003.htm specifications].

==== Maximum file size of the attributes file ====

* '''DBF-Backend''' (in GRASS 6 the default DB backend): ''to be added'' (2Gb? in case of LFS enabled?)
* '''SQLite-Backend''' (in GRASS 7 the default DB backend): The maximum file size of a SQLite db is '''140 TB''', independent of the architecture, i.e. Large File Support (LFS) is always there. Usually SQLite will hit the maximum file size limit of the underlying filesystem or disk hardware size limit long before it hits its own [http://www.sqlite.org/fileformat2.html internal size limit].

=== Raster management ===

A "raster map" is a data layer consisting of a gridded array of cells. It has a certain number of rows and columns, with a data point (or null value indicator) in each cell. These may exist as a 2D grid or as a 3D cube made up of many smaller cubes, i.e. a stack of 2D grids.

See also {{cmd|rasterintro}} in the manual.

==== Raster map precision types ====

* '''CELL DATA TYPE''': a raster map from INTEGER type (4 bytes, whole numbers only).
** In GRASS GIS, CELL is a 32 bit integer with a range from -2,147,483,647 to +2,147,483,647. The value -2,147,483,648 is reserved for NODATA.
* '''FCELL DATA TYPE''': a raster map from FLOAT type (4 bytes, 7-9 digits precision).
** In GRASS GIS, FCELL is a 32 bit float (Float32) with a range from -3.4E38 to 3.4E38. However, the integer precision can be only ensured between -16777216 and 16777216. If your raster overpass this range we strongly suggest to use DCELL, as Float64 data type.
* '''DCELL DATA TYPE''': a raster map from DOUBLE type (8 bytes, 15-17 digits precision).
** In GRASS GIS, DCELL is a 64 bit float (Float64) with a range from -1.79E308 to 1.79E308.
* '''NULL''': represents "no data" in raster maps, to be distinguished from 0 (zero) data value.

Aliases:
* '''INTEGER MAP''': see CELL DATA TYPE
* '''FLOAT MAP''': see FCELL DATA TYPE
* '''DOUBLE MAP''': see DCELL DATA TYPE

(reference in the [https://github.com/OSGeo/grass/blob/master/include/gis.h#L588 GRASS GIS source code])

See also [[GRASS raster semantics]]

== Large file support ==
=== Large raster data processing ===

GRASS GIS 7 supports the off_t type, hence it can address an enormous amount of raster data.

See also:
* [[Large raster data processing]]

==== Some '''benchmarks''' ====

* Import of [http://www.ecad.eu/ ECAD 6.0] Tmean dataset: 22650 layers in single netCDF file: import takes 300 Seconds while reading file via NFS (i.e. 75 maps per second)
* Calculation of watersheds, half basins, flow accumulation, drainage directions, and stream with {{cmd|r.watershed}} for an area of 90,000 rows x 100,000 cols (9,000,000,000 cells, metric) successfully done in 77.2 hours (Intel Xeon X5670, 2.93GHz)
* European DEM at 25m ([http://www.eea.europa.eu/data-and-maps/data/eu-dem#tab-original-data eudem_dem_3035_europe.tif], 24.1 GB GeoTIFF, 48 billion cells) processing:
** Import of this GeoTIFF file with r.in.gdal on a blade via NFS: a) '''77h''' without memory option (hence 40MB = GDAL's default cache), b) ''''1.5h'''' with memory=300 (hence using 300MB GDAL cache), c) ''''1.5h'''' with memory=2000 (hence using 2GB GDAL cache)
* r.neighbors with [https://trac.osgeo.org/grass/ticket/2676#comment:2 3.694261e+12 pixels] (rows: 440046 cols: 830958 cells)
* Import of Global Forest Loss map with rows=560000 * cols=1440000 = 8.064e+11 pixels (see {{trac|3365}}); map can be easily shown in GRASS GIS monitor
* r.stream.extract: the upper limit matrix cell number that can handle is about 1.15e+18 raster cells (1.15 "[https://en.wikipedia.org/wiki/Unit_prefix exa]"-cells. The number of detected stream segments must not be larger than 2,147,483,647 streams.
* ... add more

=== Large vector data processing ===

GRASS GIS 7 supports the off_t type, hence it can address an enormous amount of vector data.
Currently multi-billion vector points have been managed ([http://lists.osgeo.org/pipermail/grass-dev/2011-January/052996.html citation]) without topology (since not needed). In all GRASS versions, the limit with topology is at time 2^31 - 1 (about 2 billion) features per vector map.

See also:
* [[Large vector data processing]]

==== Some '''benchmarks''' ====

* ...
* ... add more

== Parallelization ==

In GRASS 7, a few modules have been experimentally parallelized with OpenMP. However, if data can be processed in chunks, GRASS GIS can be used on clusters.

* [[OpenMP/Benchmarks]]

Parallelized modules:

* v.surf.rst, r.sim.water, r.sun, ...

Note: As of 2020, there are still issues with openMP (it may lead to weird results or perform slower).

== Benchmarks ==

=== r.neighbors ===

$ g.region -pa n=228500 s=215000 w=630000 e=645000 res=0.5
$ r.random.surface output=random
$ time r.neighbors input=random output=avg,min,max method=average,minimum,maximum size=5
real 6m58.801s
user 6m45.132s
sys 0m6.864s

810,000,000 cells (27,000x30,000), 3 outputs (average, min, max), window size 5, one core, negligible use of RAM.

== User reports ==

* [[Case studies]]

== See also ==
* [[Parallel GRASS jobs]]
* [[Software requirements specification]]
* [[Performance comparison GRASS vs. ArcGIS]]
* [[OpenMP/Benchmarks|Benchmarks of OpenMP parallelized code]]
* [http://wiki.osgeo.org/wiki/GIS_workstation_setup_tips GIS workstation setup tips] (OSGeo Wiki page)

[[Category: FAQ]]
[[Category: Hardware]]
[[Category: massive data analysis]]
[[Category: Raster]]
[[Category: Vector]]

GRASS raster semantics

2021-02-27T13:59:23Z

⚠️El selvaje: adding Float32 range-bound

A quick summary c/o Glynn Clements:

== Raster map precision types ==

* '''CELL DATA TYPE''': a raster map from INTEGER type (4 bytes, whole numbers only).
** In GRASS GIS, CELL is a 32 bit integer with a range from -2,147,483,647 to +2,147,483,647. The value -2,147,483,648 is reserved for NODATA.
* '''FCELL DATA TYPE''': a raster map from FLOAT type (4 bytes, 7-9 digits precision).
** In GRASS GIS, FCELL is a 32 bit float (Float32) with a range from -3.4E38 to 3.4E38. However, the integer precision can be only ensured between -16777216 and 16777216. If your raster overpass this range we strongly suggest to use DCELL, as Float64 data type.
* '''DCELL DATA TYPE''': a raster map from DOUBLE type (8 bytes, 15-17 digits precision).
** In GRASS GIS, DCELL is a 64 bit float (Float64) with a range from -1.79E308 to 1.79E308.
* '''NULL''': represents "no data" in raster maps, to be distinguished from 0 (zero) data value.

Aliases:
* '''INTEGER MAP''': see CELL DATA TYPE
* '''FLOAT MAP''': see FCELL DATA TYPE
* '''DOUBLE MAP''': see DCELL DATA TYPE

(reference in the [https://github.com/OSGeo/grass/blob/master/include/gis.h#L588 GRASS GIS source code])

==Region Calculations==

The "current region" or "computational region" is the actual setting of the region boundaries and the actual raster resolution. Note that the computational region isn't limited to raster data; it may also affect some
vector operations.

The computational region's bounds describe a rectangle in two-dimensional space. For
raster operations, this rectangle is subdivided into a grid of
rectangular cells, so the region's bounds are aligned with the edges
of the outermost cells.

For details, see [[Computational region]].

==Cell Locations==

* [[Grid registration|GRASS uses cell-center registration not grid point registration]]
* The raster region defined with {{cmd|g.region}} refers to the outer-boundary edge of all cells contained within it.

Cells are areas, not points, so they don't have locations. Their
corners have locations, as do their centres.

A cell with array indices (i,j) (easting, northing) corresponds to the
rectangle:

{ (x,y) : west + i * ewres <= x < west + (i+1) * ewres,
north - (j+1) * nsres <= y < north - j * nsres }

whose centre is at:

(west + (i+1/2) * ewres, north - (j+1/2) * nsres)

[Subject to wrapping of longitude values in lat/lon locations.]

==Raster to Vector Conversions==

IIRC, {{cmd|r.to.vect}} uses the midpoints of the cell's edges (i.e. one
coordinate will be on a grid line, the other will be mid-way between
grid lines).

==Resampling==

(see also [[Interpolation#Resampling_methods_and_interpolation_in_GRASS]])

The built-in nearest-neighbour resampling of raster data calculates
the centre of each region cell, and takes the value of the raster cell
in which that point falls.

If the point falls exactly upon a grid line, the exact result will be
determined by the direction of any rounding error.

[One consequence of this is that downsampling by a factor which is an
even integer will always sample exactly on the boundary between cells,
meaning that the result is ill-defined.]

{{cmd|r.resample}} uses the built-in resampling, so it should produce
identical results.

{{cmd|r.resamp.interp}} method=nearest uses the same algorithm, but not the
same code, so it may not produce identical results in cases which are
decided by the rounding of floating-point numbers.

For method=bilinear and method=bicubic, the raster values are treated
as samples at each raster cell's centre, defining a piecewise-
continuous surface. The resulting raster values are obtained by
sampling the surface at each region cell's centre.

As the algorithm only interpolates, and doesn't extrapolate, a margin
of 0.5 (for bilinear) or 1.5 (for bicubic) cells is lost from the
extent of the original raster. Any samples taken within this margin
will be null.

AFAIK, {{cmd|r.resamp.rst}} behaves similarly, i.e. it computes a surface
assuming that the values are samples at each raster cell's centre, and
samples the surface at each region cell's centre.

For {{cmd|r.resamp.stats}} without -w, the value of each region cell is the
chosen aggregate of the values from all of the raster cells whose
centres fall within the bounds of the region cell.

With -w, the samples are weighted according to the proportion of the
raster cell which falls within the bounds of the region cell, so the
result is normally[1] unaffected by rounding error (a miniscule
difference in the position of the boundary results in the addition or
subtraction of a sample weighted by a miniscule factor).

[1] The min and max aggregates can't use weights, so -w has no effect
for those.

==General Rules==

For the most part, the interpretation is the "obvious" one, given:

# Cells are areas rather than points.
# Operations which need a point (e.g. interpolation) use the cell's centre.

== Handling of NULL values ==

=== General remarks ===

The NULL (no data) handling depends on the map precision type. See for details
* http://grass.osgeo.org/programming7/rasterlib.html#Null_no_data

=== {{cmd|r.mapcalc}} ===

Nearly all operators and functions return null if any of their
arguments are null.

The main exceptions are:

* "isnull(x)" returns 0 if x is non-null and 1 if x is null.

* "if(1,a,b)" returns a even if b is null.

* "if(0,a,b)" returns b even if a is null.

* "eval(a,b,c,...,x)" returns x even if any or all of a,b,c,... are null.

* "0 &&& x" and "x &&& 0" return 0 even if x is null.

* "1 ||| x" and "x ||| 1" return 1 even if x is null.

* "graph(x,x1,y1,...,xn,yn)" will return null if x is null or any of the x[i] are null, or if a "relevant" y[i] is null, but not where a y[i] which isn't used in the calculation is null.

[The &&& and ||| operators are a relatively recent addition; the older
&& and || operators follow the usual behaviour for nulls, i.e. they
return null if either operand is null.]

The behaviour is quite intuitive if you view null as representing an
unknown quantity. This explains why both "x == null()" and "x != null()"
are always null, regardless of whether x is null or non-null.

null() is an unknown value, so it's unknown whether or not it's equal
to any particular (known) value. If x is also null, then you have two
unknown values, and it's unknown whether or not the two are equal.

In many respects, the behaviour is similar to that of NaN in
floating-point arithmetic.

With the exception of the cases listed above, r.mapcalc doesn't try to
detect "special" cases where the result can be deduced even if one of
the operands is null. E.g. if x is null, "x * 0" is null rather than
zero, "x == x" is null rather than 1, "x != x" is null rather than 0,
etc.

If you have a "boolean" map where the values are either null or 1, you
normally want to convert null to 0 before performing any other
processing. You can either replace the nulls in an existing map using
e.g. "r.null null=0", or adjust the tests within r.mapcalc

== Arithmetic operations ==

r.mapcalc: Arithmetic operations on mixed types promote the lesser type to the greater type, where CELL < FCELL < DCELL:

<source lang="bash">
| CELL FCELL DCELL
------+------------------
CELL | CELL FCELL DCELL
FCELL | FCELL FCELL DCELL
DCELL | DCELL DCELL DCELL
</source>

==Developer Notes Rules==
From a programming perspective, the functions:

G_row_to_northing()
G_col_to_easting()
G_northing_to_row()
G_easting_to_col()

all transform floating-point values.

Passing integer row or column indices to the first two functions will
return the coordinates of the cell's top-left corner; for the centre
coordinates, pass row+0.5 and/or col+0.5.

Similarly, the last two functions will typically return non-integral
values; use floor() to discard the fractional part and obtain the row
or column index of the cell within which the point lies.

[[Category: FAQ]]
[[Category: Interpolation]]
[[Category:Raster]]

GRASS GIS Performance

2021-02-27T13:52:10Z

⚠️El selvaje: /* Raster map precision types */

== GRASS GIS Performance ==

GRASS GIS is noted for being ready for massive data analysis. This page contains an yet incomplete collection of performance indicators.

=== Architecture ===

GRASS GIS is fully 32bit and 64bit compliant. See also the [[Software requirements specification]].

=== Search strategies used in processing geodata ===

GRASS GIS makes heavy use of search trees in order to speed up computation:
* segment lib: btree2
* 2D splines (RST): quadtree
* 3D splines (RST): octree
* vector lib topology: R*-tree

See the [http://grass.osgeo.org/programming7/ Programmer's manual] for details.

=== Number of opened input files ===

There are only operating system constraints of the number of input files which can be opened simultaneously. Commonly the limit is 1024 files. In operating systems like Linux this limit can be overcome with the "ulimit" settings.

See also
* {{cmd|r.series}}
* {{cmd|r.series.accumulate}}

=== Memory management ===

Due to the modular architecture of GRASS GIS the overhead of the software itself is minimal.

'''Raster data operations''': where appropriate, modules offer a parameter to optimize caching ("memory" parameter).
# Pixel based operations: they have very low impact on memory usage.
# Moving window based operations: they have medium impact on memory usage.
# Full map operations (watersheds, cost surfaces, etc.): they have high impact on memory usage.
# Statistical operations: while univariate statistics have low impact on memory usage, quartiles and other aggregated statistics have medium impact on memory usage.

'''Vector data operations''':
# Vector point operations: memory consumption depends on the amount of points. LiDAR data processing is commonly demanding. For some operations the creation of topology can be skipped to reduce the memory footprint.
# Vector line operations: they have low impact on memory usage (depends on the amount of data).
# Vector area/faces operations: they have high impact on memory usage.
# Topological versus non-topological operations: a subset of vector modules is able to operate on point vector maps without topology which saves notably RAM usage.

'''Database operations''':
* Most operations are simply SQL transactions with low impact on memory usage.

'''See also'''
* Solving [[Memory issues]] when dealing with large amounts of data

=== Vector management ===

A "vector map" is a data layer consisting of a number of sparse features in geographic space. These might be data points (drill sites), lines (roads), polygons (park boundary), volumes (3D CAD structure), or some combination of all these. Typically each feature in the map will be tied to a set of attribute layers stored in a database (road names, site ID, geologic type, etc.). As a general rule these can exist in 2D or 3D space and are independent of the GIS's computation region.

See also {{cmd|vectorintro}} in the manual.

==== Vector geometry ====

In all GRASS GIS versions,

* with topology the feature limit is at time 2^31 - 1 (about 2 billion) features per vector map.
* ''TODO: add limit if topology creation is disabled at import for points (e.g., LiDAR points).''

==== Vector attribute management ====

Attributes are managed through a SQL interface (see also {{cmd|databaseintro}})

The default database backend is
* DBF files (tend to be slow) in GRASS GIS 6 ({{cmd|grass-dbf|version=64}})
* SQLite file (very fast compared to DBF in GRASS GIS 7 ({{cmd|grass-sqlite}})

Other SQL backends are offered as well including PostgreSQL, MySQL, etc.: see {{cmd|sql}} support in GRASS GIS.

'''Speed of DBF versus SQLite drivers''': attribute operations which take hours using the DBF backend just take seconds using the SQLite backend.

==== Maximum Number of Attribute Columns ====

The maximum number of attribute columns of a table connected to a vector map is defined by the capabilities of the the selected database backend (set with {{cmd|db.connect}}).

* '''DBF-Backend''': GRASS 4.x - 6.x use by default the DBF backend. While there is no explicitly stated maximum number of allowed attribute columns, Web sources report a maximum '''between 128 and 1023/24'''. Trials with GRASS 6.4.2 in 2012 result in write failure if > 2000 attribute columns are used. Export to DBF-based ESRI Shapefile provides a warning if more that '''255''' attributes are used: Other software tools may ignore all further attributes, hence a maximum of '''128''' columns may be prudent.
* '''SQLite-Backend''': GRASS 7.x uses by default the SQLite backend. The default maximum number of attribute columns is '''2000''' according to the [http://www.sqlite.org/limits.html#max_column specifications]. This number can be increased by compiling SQlite with changed settings.
* '''MySQL-Backend''': The default maximum number of attribute columns is '''4096''' according to the [http://wiki.postgresql.org/wiki/FAQ#What_is_the_maximum_size_for_a_row.2C_a_table.2C_and_a_database.3F specifications].
* '''PostgreSQL-Backend''': The default maximum number of attribute columns is '''250-1600''' according to the [http://dev.mysql.com/doc/refman/5.1/en/column-count-limit.html specifications] depending on column types.
* '''Oracle-Backend''': The default maximum number of attribute columns is '''1000''' according to the [http://docs.oracle.com/cd/B19306_01/server.102/b14237/limits003.htm specifications].

==== Maximum file size of the attributes file ====

* '''DBF-Backend''' (in GRASS 6 the default DB backend): ''to be added'' (2Gb? in case of LFS enabled?)
* '''SQLite-Backend''' (in GRASS 7 the default DB backend): The maximum file size of a SQLite db is '''140 TB''', independent of the architecture, i.e. Large File Support (LFS) is always there. Usually SQLite will hit the maximum file size limit of the underlying filesystem or disk hardware size limit long before it hits its own [http://www.sqlite.org/fileformat2.html internal size limit].

=== Raster management ===

A "raster map" is a data layer consisting of a gridded array of cells. It has a certain number of rows and columns, with a data point (or null value indicator) in each cell. These may exist as a 2D grid or as a 3D cube made up of many smaller cubes, i.e. a stack of 2D grids.

See also {{cmd|rasterintro}} in the manual.

==== Raster map precision types ====

* '''CELL DATA TYPE''': a raster map from INTEGER type (4 bytes, whole numbers only)
** in GRASS GIS, CELL is a 32 bit integer with a range from -2,147,483,647 to +2,147,483,647. The value -2,147,483,648 is reserved for NODATA.
* '''FCELL DATA TYPE''': a raster map from FLOAT type (4 bytes, 7-9 digits precision)
** in GRASS GIS, FCELL is a 32 bit float (Float32) with a range from -3.4E38 to 3.4E38. However, the integer precision can be only ensured between -16777216 and 16777216. If your raster overpass this range we strongly suggest to use DCELL, as Float64 data type.
* '''DCELL DATA TYPE''': a raster map from DOUBLE type (8 bytes, 15-17 digits precision)
** in GRASS GIS, DCELL is a 64 bit float (Float64) with a range from -3.4E38 to 3.4E38.
* '''NULL''': represents "no data" in raster maps, to be distinguished from 0 (zero) data value

Aliases:
* '''INTEGER MAP''': see CELL DATA TYPE
* '''FLOAT MAP''': see FCELL DATA TYPE
* '''DOUBLE MAP''': see DCELL DATA TYPE

(reference in the [https://github.com/OSGeo/grass/blob/master/include/gis.h#L588 GRASS GIS source code])

See also [[GRASS raster semantics]]

== Large file support ==
=== Large raster data processing ===

GRASS GIS 7 supports the off_t type, hence it can address an enormous amount of raster data.

See also:
* [[Large raster data processing]]

==== Some '''benchmarks''' ====

* Import of [http://www.ecad.eu/ ECAD 6.0] Tmean dataset: 22650 layers in single netCDF file: import takes 300 Seconds while reading file via NFS (i.e. 75 maps per second)
* Calculation of watersheds, half basins, flow accumulation, drainage directions, and stream with {{cmd|r.watershed}} for an area of 90,000 rows x 100,000 cols (9,000,000,000 cells, metric) successfully done in 77.2 hours (Intel Xeon X5670, 2.93GHz)
* European DEM at 25m ([http://www.eea.europa.eu/data-and-maps/data/eu-dem#tab-original-data eudem_dem_3035_europe.tif], 24.1 GB GeoTIFF, 48 billion cells) processing:
** Import of this GeoTIFF file with r.in.gdal on a blade via NFS: a) '''77h''' without memory option (hence 40MB = GDAL's default cache), b) ''''1.5h'''' with memory=300 (hence using 300MB GDAL cache), c) ''''1.5h'''' with memory=2000 (hence using 2GB GDAL cache)
* r.neighbors with [https://trac.osgeo.org/grass/ticket/2676#comment:2 3.694261e+12 pixels] (rows: 440046 cols: 830958 cells)
* Import of Global Forest Loss map with rows=560000 * cols=1440000 = 8.064e+11 pixels (see {{trac|3365}}); map can be easily shown in GRASS GIS monitor
* r.stream.extract: the upper limit matrix cell number that can handle is about 1.15e+18 raster cells (1.15 "[https://en.wikipedia.org/wiki/Unit_prefix exa]"-cells. The number of detected stream segments must not be larger than 2,147,483,647 streams.
* ... add more

=== Large vector data processing ===

GRASS GIS 7 supports the off_t type, hence it can address an enormous amount of vector data.
Currently multi-billion vector points have been managed ([http://lists.osgeo.org/pipermail/grass-dev/2011-January/052996.html citation]) without topology (since not needed). In all GRASS versions, the limit with topology is at time 2^31 - 1 (about 2 billion) features per vector map.

See also:
* [[Large vector data processing]]

==== Some '''benchmarks''' ====

* ...
* ... add more

== Parallelization ==

In GRASS 7, a few modules have been experimentally parallelized with OpenMP. However, if data can be processed in chunks, GRASS GIS can be used on clusters.

* [[OpenMP/Benchmarks]]

Parallelized modules:

* v.surf.rst, r.sim.water, r.sun, ...

Note: As of 2020, there are still issues with openMP (it may lead to weird results or perform slower).

== Benchmarks ==

=== r.neighbors ===

$ g.region -pa n=228500 s=215000 w=630000 e=645000 res=0.5
$ r.random.surface output=random
$ time r.neighbors input=random output=avg,min,max method=average,minimum,maximum size=5
real 6m58.801s
user 6m45.132s
sys 0m6.864s

810,000,000 cells (27,000x30,000), 3 outputs (average, min, max), window size 5, one core, negligible use of RAM.

== User reports ==

* [[Case studies]]

== See also ==
* [[Parallel GRASS jobs]]
* [[Software requirements specification]]
* [[Performance comparison GRASS vs. ArcGIS]]
* [[OpenMP/Benchmarks|Benchmarks of OpenMP parallelized code]]
* [http://wiki.osgeo.org/wiki/GIS_workstation_setup_tips GIS workstation setup tips] (OSGeo Wiki page)

[[Category: FAQ]]
[[Category: Hardware]]
[[Category: massive data analysis]]
[[Category: Raster]]
[[Category: Vector]]