GRASS GIS Performance - Revision history

Gregorywpower: /* Maximum Number of Attribute Columns */

2024-10-13T02:14:07Z

Maximum Number of Attribute Columns

← Older revision		Revision as of 02:14, 13 October 2024
Line 77:		Line 77:

	* '''DBF-Backend''': GRASS 4.x - 6.x use by default the DBF backend. While there is no explicitly stated maximum number of allowed attribute columns, Web sources report a maximum '''between 128 and 1023/24'''. Trials with GRASS 6.4.2 in 2012 result in write failure if > 2000 attribute columns are used. Export to DBF-based ESRI Shapefile provides a warning if more that '''255''' attributes are used: Other software tools may ignore all further attributes, hence a maximum of '''128''' columns may be prudent.		* '''DBF-Backend''': GRASS 4.x - 6.x use by default the DBF backend. While there is no explicitly stated maximum number of allowed attribute columns, Web sources report a maximum '''between 128 and 1023/24'''. Trials with GRASS 6.4.2 in 2012 result in write failure if > 2000 attribute columns are used. Export to DBF-based ESRI Shapefile provides a warning if more that '''255''' attributes are used: Other software tools may ignore all further attributes, hence a maximum of '''128''' columns may be prudent.
	* '''SQLite-Backend''': GRASS 7.x uses by default the SQLite backend. The default maximum number of attribute columns is '''2000''' according to the [~~http~~://www.sqlite.org/limits.html#max_column specifications]. This number can be increased by compiling SQlite with changed settings.		* '''SQLite-Backend''': GRASS 7.x uses by default the SQLite backend. The default maximum number of attribute columns is '''2000''' according to the [https://www.sqlite.org/limits.html#max_column specifications]. This number can be increased by compiling SQlite with changed settings.
	* '''MySQL-Backend''': The default maximum number of attribute columns is '''4096''' according to the [~~http~~://~~wiki~~.~~postgresql~~.~~org~~/~~wiki~~/~~FAQ#What_is_the_maximum_size_for_a_row~~.~~2C_a_table~~.~~2C_and_a_database.3F~~ specifications].		* '''MySQL-Backend''': The default maximum number of attribute columns is '''4096''' according to the [https://dev.mysql.com/doc/refman/5.1/en/column-count-limit.html specifications].
	* '''PostgreSQL-Backend''': The default maximum number of attribute columns is '''250-1600''' according to the [~~http~~://~~dev~~.~~mysql~~.~~com~~/~~doc~~/~~refman/5~~.~~1/en/column-count-limit~~.~~html~~ specifications] depending on column types.		* '''PostgreSQL-Backend''': The default maximum number of attribute columns is '''250-1600''' according to the [https://wiki.postgresql.org/wiki/FAQ#What_is_the_maximum_size_for_a_row.2C_a_table.2C_and_a_database.3F specifications] depending on column types.
	* '''Oracle-Backend''': The default maximum number of attribute columns is '''1000''' according to the [~~http~~://docs.oracle.com/cd/B19306_01/server.102/b14237/limits003.htm specifications].		* '''Oracle-Backend''': The default maximum number of attribute columns is '''1000''' according to the [https://docs.oracle.com/cd/B19306_01/server.102/b14237/limits003.htm specifications].

	==== Maximum file size of the attributes file ====		==== Maximum file size of the attributes file ====

⚠️El selvaje: add Float32 range-bound

2021-02-27T14:00:05Z

add Float32 range-bound

← Older revision		Revision as of 14:00, 27 February 2021
Line 95:		Line 95:
	==== Raster map precision types ====		==== Raster map precision types ====

	* '''CELL DATA TYPE''': a raster map from INTEGER type (4 bytes, whole numbers only)		* '''CELL DATA TYPE''': a raster map from INTEGER type (4 bytes, whole numbers only).
	** in GRASS GIS, CELL is a 32 bit integer with a range from -2,147,483,647 to +2,147,483,647. The value -2,147,483,648 is reserved for NODATA.		** In GRASS GIS, CELL is a 32 bit integer with a range from -2,147,483,647 to +2,147,483,647. The value -2,147,483,648 is reserved for NODATA.
	* '''FCELL DATA TYPE''': a raster map from FLOAT type (4 bytes, 7-9 digits precision)		* '''FCELL DATA TYPE''': a raster map from FLOAT type (4 bytes, 7-9 digits precision).
	** in GRASS GIS, FCELL is a 32 bit float (Float32) with a range from -3.4E38 to 3.4E38. However, the integer precision can be only ensured between -16777216 and 16777216. If your raster overpass this range we strongly suggest to use DCELL, as Float64 data type.		** In GRASS GIS, FCELL is a 32 bit float (Float32) with a range from -3.4E38 to 3.4E38. However, the integer precision can be only ensured between -16777216 and 16777216. If your raster overpass this range we strongly suggest to use DCELL, as Float64 data type.
	* '''DCELL DATA TYPE''': a raster map from DOUBLE type (8 bytes, 15-17 digits precision)		* '''DCELL DATA TYPE''': a raster map from DOUBLE type (8 bytes, 15-17 digits precision).
	** in GRASS GIS, DCELL is a 64 bit float (Float64) with a range from -3.~~4E38~~ to 3.~~4E38~~.		** In GRASS GIS, DCELL is a 64 bit float (Float64) with a range from -1.79E308 to 1.79E308.
	* '''NULL''': represents "no data" in raster maps, to be distinguished from 0 (zero) data value		* '''NULL''': represents "no data" in raster maps, to be distinguished from 0 (zero) data value.

	Aliases:		Aliases:

⚠️El selvaje: /* Raster map precision types */

2021-02-27T13:52:10Z

Raster map precision types

← Older revision		Revision as of 13:52, 27 February 2021
Line 98:		Line 98:
	** in GRASS GIS, CELL is a 32 bit integer with a range from -2,147,483,647 to +2,147,483,647. The value -2,147,483,648 is reserved for NODATA.		** in GRASS GIS, CELL is a 32 bit integer with a range from -2,147,483,647 to +2,147,483,647. The value -2,147,483,648 is reserved for NODATA.
	* '''FCELL DATA TYPE''': a raster map from FLOAT type (4 bytes, 7-9 digits precision)		* '''FCELL DATA TYPE''': a raster map from FLOAT type (4 bytes, 7-9 digits precision)
			** in GRASS GIS, FCELL is a 32 bit float (Float32) with a range from -3.4E38 to 3.4E38. However, the integer precision can be only ensured between -16777216 and 16777216. If your raster overpass this range we strongly suggest to use DCELL, as Float64 data type.
	* '''DCELL DATA TYPE''': a raster map from DOUBLE type (8 bytes, 15-17 digits precision)		* '''DCELL DATA TYPE''': a raster map from DOUBLE type (8 bytes, 15-17 digits precision)
			** in GRASS GIS, DCELL is a 64 bit float (Float64) with a range from -3.4E38 to 3.4E38.
	* '''NULL''': represents "no data" in raster maps, to be distinguished from 0 (zero) data value		* '''NULL''': represents "no data" in raster maps, to be distinguished from 0 (zero) data value

Neteler: added "Raster map precision types"

2021-02-25T19:30:46Z

added "Raster map precision types"

@@ Line 48: / Line 48: @@
 === Vector management ===
 ==== Vector geometry ====
@@ Line 82: / Line 86: @@
 * '''DBF-Backend''' (in GRASS 6 the default DB backend): ''to be added'' (2Gb? in case of LFS enabled?)
 * '''SQLite-Backend''' (in GRASS 7 the default DB backend): The maximum file size of a SQLite db is '''140 TB''', independent of the architecture, i.e. Large File Support (LFS) is always there. Usually SQLite will hit the maximum file size limit of the underlying filesystem or disk hardware size limit long before it hits its own [http://www.sqlite.org/fileformat2.html internal size limit].
 == Large file support ==

Neteler: openMP still experimental

2020-10-20T09:02:35Z

openMP still experimental

← Older revision		Revision as of 09:02, 20 October 2020
Line 117:		Line 117:
	== Parallelization ==		== Parallelization ==

	In GRASS 7, a few modules have been parallelized with OpenMP. However, if data can be processed in chunks, GRASS GIS can be used on clusters.		In GRASS 7, a few modules have been experimentally parallelized with OpenMP. However, if data can be processed in chunks, GRASS GIS can be used on clusters.

	* [[OpenMP/Benchmarks]]		* [[OpenMP/Benchmarks]]
Line 124:		Line 124:

	* v.surf.rst, r.sim.water, r.sun, ...		* v.surf.rst, r.sim.water, r.sun, ...

			Note: As of 2020, there are still issues with openMP (it may lead to weird results or perform slower).

	== Benchmarks ==		== Benchmarks ==

Neteler: /* See also */

2019-02-28T12:32:48Z

Neteler: remove hardcoded version specific urls, use latest version

2018-12-04T09:58:05Z

remove hardcoded version specific urls, use latest version

← Older revision		Revision as of 09:58, 4 December 2018
Line 23:		Line 23:
	See also		See also
	* {{cmd\|r.series}}		* {{cmd\|r.series}}
	* {{cmd\|r.series.accumulate~~\|version=70~~}}		* {{cmd\|r.series.accumulate}}

	=== Memory management ===		=== Memory management ===
Line 94:		Line 94:

	* Import of [http://www.ecad.eu/ ECAD 6.0] Tmean dataset: 22650 layers in single netCDF file: import takes 300 Seconds while reading file via NFS (i.e. 75 maps per second)		* Import of [http://www.ecad.eu/ ECAD 6.0] Tmean dataset: 22650 layers in single netCDF file: import takes 300 Seconds while reading file via NFS (i.e. 75 maps per second)
	* Calculation of watersheds, half basins, flow accumulation, drainage directions, and stream with {{cmd\|r.watershed~~\|version=70~~}} for an area of 90,000 rows x 100,000 cols (9,000,000,000 cells, metric) successfully done in 77.2 hours (Intel Xeon X5670, 2.93GHz)		* Calculation of watersheds, half basins, flow accumulation, drainage directions, and stream with {{cmd\|r.watershed}} for an area of 90,000 rows x 100,000 cols (9,000,000,000 cells, metric) successfully done in 77.2 hours (Intel Xeon X5670, 2.93GHz)
	* European DEM at 25m ([http://www.eea.europa.eu/data-and-maps/data/eu-dem#tab-original-data eudem_dem_3035_europe.tif], 24.1 GB GeoTIFF, 48 billion cells) processing:		* European DEM at 25m ([http://www.eea.europa.eu/data-and-maps/data/eu-dem#tab-original-data eudem_dem_3035_europe.tif], 24.1 GB GeoTIFF, 48 billion cells) processing:
	** Import of this GeoTIFF file with r.in.gdal on a blade via NFS: a) '''77h''' without memory option (hence 40MB = GDAL's default cache), b) ''''1.5h'''' with memory=300 (hence using 300MB GDAL cache), c) ''''1.5h'''' with memory=2000 (hence using 2GB GDAL cache)		** Import of this GeoTIFF file with r.in.gdal on a blade via NFS: a) '''77h''' without memory option (hence 40MB = GDAL's default cache), b) ''''1.5h'''' with memory=300 (hence using 300MB GDAL cache), c) ''''1.5h'''' with memory=2000 (hence using 2GB GDAL cache)

⚠️Wenzeslaus: /* Parallelization */

2018-08-12T03:21:02Z

Parallelization

@@ Line 117: / Line 117: @@
 == Parallelization ==
-In GRASS 7, a few modules have been parallelized with openMP. However, if data can be processed in chunks, GRASS GIS can be used on clusters.
+In GRASS 7, a few modules have been parallelized with OpenMP. However, if data can be processed in chunks, GRASS GIS can be used on clusters.
 * [[OpenMP/Benchmarks]]
 == User reports ==

Neteler: 1.15e+18 cells = 1.15 exa-cells

2017-11-28T23:30:54Z

1.15e+18 cells = 1.15 exa-cells

← Older revision		Revision as of 23:30, 28 November 2017
Line 99:		Line 99:
	* r.neighbors with [https://trac.osgeo.org/grass/ticket/2676#comment:2 3.694261e+12 pixels] (rows: 440046 cols: 830958 cells)		* r.neighbors with [https://trac.osgeo.org/grass/ticket/2676#comment:2 3.694261e+12 pixels] (rows: 440046 cols: 830958 cells)
	* Import of Global Forest Loss map with rows=560000 * cols=1440000 = 8.064e+11 pixels (see {{trac\|3365}}); map can be easily shown in GRASS GIS monitor		* Import of Global Forest Loss map with rows=560000 * cols=1440000 = 8.064e+11 pixels (see {{trac\|3365}}); map can be easily shown in GRASS GIS monitor
	* r.stream.extract: the upper limit matrix cell number that can handle is about 1.15e+18 cells. The number of detected stream segments must not be larger than 2,147,483,647 streams		* r.stream.extract: the upper limit matrix cell number that can handle is about 1.15e+18 raster cells (1.15 "[https://en.wikipedia.org/wiki/Unit_prefix exa]"-cells. The number of detected stream segments must not be larger than 2,147,483,647 streams.
	* ... add more		* ... add more

Neteler: +r.stream.extract notes from https://lists.osgeo.org/pipermail/grass-user/2017-November/077259.html

2017-11-10T16:19:20Z

+r.stream.extract notes from https://lists.osgeo.org/pipermail/grass-user/2017-November/077259.html

← Older revision		Revision as of 16:19, 10 November 2017
Line 99:		Line 99:
	* r.neighbors with [https://trac.osgeo.org/grass/ticket/2676#comment:2 3.694261e+12 pixels] (rows: 440046 cols: 830958 cells)		* r.neighbors with [https://trac.osgeo.org/grass/ticket/2676#comment:2 3.694261e+12 pixels] (rows: 440046 cols: 830958 cells)
	* Import of Global Forest Loss map with rows=560000 * cols=1440000 = 8.064e+11 pixels (see {{trac\|3365}}); map can be easily shown in GRASS GIS monitor		* Import of Global Forest Loss map with rows=560000 * cols=1440000 = 8.064e+11 pixels (see {{trac\|3365}}); map can be easily shown in GRASS GIS monitor
			* r.stream.extract: the upper limit matrix cell number that can handle is about 1.15e+18 cells. The number of detected stream segments must not be larger than 2,147,483,647 streams
	* ... add more		* ... add more

← Older revision		Revision as of 12:32, 28 February 2019
Line 152:		Line 152:
	[[Category: Hardware]]		[[Category: Hardware]]
	[[Category: massive data analysis]]		[[Category: massive data analysis]]
			[[Category: Raster]]
			[[Category: Vector]]