Large raster data processing - Revision history

⚠️Madi: Added how to category

2022-12-07T14:00:51Z

Added how to category

← Older revision		Revision as of 14:00, 7 December 2022
Line 160:		Line 160:
	[[Category: massive data analysis]]		[[Category: massive data analysis]]
	[[Category: raster]]		[[Category: raster]]
			[[Category: How To]]

Neteler: minor cleanup

2020-12-28T17:34:48Z

minor cleanup

← Older revision		Revision as of 17:34, 28 December 2020
Line 6:		Line 6:

	This page collects a number of workarounds for such cases.		This page collects a number of workarounds for such cases.

	~~== Large File Support (LFS) ==~~

	~~Affects 32bit systems which are normally limited to 2GB (2^31) per file. As workaround, LFS can be enabled in GRASS GIS at compile time.~~

	* GRASS GIS 6: much of the raster library and modules can be enabled for LFS during compilation
	* GRASS GIS 7: raster and vector libraries are LFS enabled.

	~~See also: [[Large File Support]]~~

	== Workaround: Select a smaller area within a very large dataset before importing in GRASS ==		== Workaround: Select a smaller area within a very large dataset before importing in GRASS ==
Line 53:		Line 44:

	(reference in the [https://github.com/OSGeo/grass/blob/master/include/gis.h#L588 GRASS GIS source code])		(reference in the [https://github.com/OSGeo/grass/blob/master/include/gis.h#L588 GRASS GIS source code])


	See also [[GRASS raster semantics]]		See also [[GRASS raster semantics]]

	== Create a mosaic ==		== Create a mosaic ==

	Now that we have a list of the files interesting our area, and since our files are all in the same datum (WGS84), we can use [http://www.gdal.org/gdalbuildvrt.html gdalbuildvrt] to create a		Now that we have a list of the files interesting our area, and since our files are all in the same datum (WGS84), we can use [http://www.gdal.org/gdalbuildvrt.html gdalbuildvrt] to create a
	[http://www.gdal.org/gdal_vrttut.html virtual] mosaic. This will take only a few seconds to run.		[http://www.gdal.org/gdal_vrttut.html virtual] mosaic. This will take only a few seconds to run.
Line 69:		Line 60:
	r.external input=/path/to/mosaic.vrt output=mosaic		r.external input=/path/to/mosaic.vrt output=mosaic

	== Split into tiles ==		GRASS GIS 7.8+ can internally generate VRT-style mosaics with {{cmd\|r.buildvrt}}

			== Split large maps into tiles ==

	Sometimes it is possible to work with tiles, i.e. splitting a big map into chunks. These tiles can be easily generated with {{cmd\|r.tile}}. It optionally allows for tiling with overlap.		Sometimes it is possible to work with tiles, i.e. splitting a big map into chunks. These tiles can be easily generated with {{cmd\|r.tile}}. It optionally allows for tiling with overlap.
Line 149:		Line 142:

	If you happen to use the "memory" option requesting more than 2GB (e.g. <tt>memory=3000</tt>) check if you are using 64bit executables since 32bit applications can use only up to 2GB memory. In this case it does not matter if the operating system is 64bit or if there is more than 2GB of memory available. You need to use 64bit binaries of GRASS GIS 7.		If you happen to use the "memory" option requesting more than 2GB (e.g. <tt>memory=3000</tt>) check if you are using 64bit executables since 32bit applications can use only up to 2GB memory. In this case it does not matter if the operating system is 64bit or if there is more than 2GB of memory available. You need to use 64bit binaries of GRASS GIS 7.

			== Large File Support (LFS) ==

			Affects 32bit systems which are normally limited to 2GB (2^31) per file. As workaround, LFS can be enabled in GRASS GIS at compile time.

			* GRASS GIS 6: much of the raster library and modules can be enabled for LFS during compilation
			* GRASS GIS 7: raster and vector libraries are LFS enabled.

			See also: [[Large File Support]]

	== See also ==		== See also ==

⚠️Smathermather at 16:06, 3 December 2020

2020-12-03T16:06:40Z

← Older revision		Revision as of 16:06, 3 December 2020
Line 121:		Line 121:
	* soft nofile 1500		* soft nofile 1500
	* hard nofile 1800		* hard nofile 1800

			On Ubuntu machines, it may be necessary to edit your /etc/pam.d/common-session file and add the following line:

			session required pam_limits.so

	Restart the user session (i.e., logout, login).		Restart the user session (i.e., logout, login).

Neteler: some intro wording added

2020-10-20T08:51:47Z

some intro wording added

← Older revision		Revision as of 08:51, 20 October 2020
Line 1:		Line 1:
	__TOC__		__TOC__

			'''Introduction'''

			While GRASS GIS can handle [[GRASS GIS Performance\|pretty large raster maps]], sometimes the user systems are hardware limited.

			This page collects a number of workarounds for such cases.

	== Large File Support (LFS) ==		== Large File Support (LFS) ==

Neteler: more details about CELL type added

2019-10-01T08:05:54Z

more details about CELL type added

← Older revision		Revision as of 08:05, 1 October 2019
Line 35:		Line 35:
	== Raster map precision types ==		== Raster map precision types ==

	* '''CELL DATA TYPE''': a raster map from INTEGER type (whole numbers only)		* '''CELL DATA TYPE''': a raster map from INTEGER type (4 bytes, whole numbers only)
			** in GRASS GIS, CELL is a 32 bit integer with a range from -2,147,483,647 to +2,147,483,647. The value -2,147,483,648 is reserved for NODATA.
	* '''FCELL DATA TYPE''': a raster map from FLOAT type (4 bytes, 7-9 digits precision)		* '''FCELL DATA TYPE''': a raster map from FLOAT type (4 bytes, 7-9 digits precision)
	* '''DCELL DATA TYPE''': a raster map from DOUBLE type (8 bytes, 15-17 digits precision)		* '''DCELL DATA TYPE''': a raster map from DOUBLE type (8 bytes, 15-17 digits precision)
Line 44:		Line 45:
	* '''FLOAT MAP''': see FCELL DATA TYPE		* '''FLOAT MAP''': see FCELL DATA TYPE
	* '''DOUBLE MAP''': see DCELL DATA TYPE		* '''DOUBLE MAP''': see DCELL DATA TYPE

			(reference in the [https://github.com/OSGeo/grass/blob/master/include/gis.h#L588 GRASS GIS source code])


	See also [[GRASS raster semantics]]		See also [[GRASS raster semantics]]

⚠️Madi: /* See also */

2017-05-08T14:54:52Z

Neteler: /* ERROR: G_malloc: unable to allocate xxxx bytes of memory ... */

2015-05-31T17:48:04Z

ERROR: G_malloc: unable to allocate xxxx bytes of memory ...

← Older revision		Revision as of 17:48, 31 May 2015
Line 130:		Line 130:
	However, sudo will probably reset certain environment variables, particularly LD_LIBRARY_PATH, so you may need to reset those in the new shell.		However, sudo will probably reset certain environment variables, particularly LD_LIBRARY_PATH, so you may need to reset those in the new shell.

	=== ERROR: G_malloc: unable to allocate xxxx bytes of memory ~~...~~ ===		=== ERROR: G_malloc: unable to allocate xxxx bytes of memory ===

	Be sure to use GRASS GIS 7 which has way better support for large files.		Be sure to use GRASS GIS 7 which has way better support for large files.

	If you happen to use the "memory" option requesting more than 2GB (e.g. <tt>memory=3000</tt>) check if you are using 64bit executables since 32bit applications can use only up to 2GB memory. In this case it does not matter if the operating system is 64bit or if there is more than 2GB of memory available. You need to use a 64bit ~~version~~ of GRASS GIS 7.		If you happen to use the "memory" option requesting more than 2GB (e.g. <tt>memory=3000</tt>) check if you are using 64bit executables since 32bit applications can use only up to 2GB memory. In this case it does not matter if the operating system is 64bit or if there is more than 2GB of memory available. You need to use 64bit binaries of GRASS GIS 7.

	== See also ==		== See also ==

Neteler: /* ERROR: G_malloc: unable to allocate xxxx bytes of memory ... = */

2015-05-31T17:45:58Z

ERROR: G_malloc: unable to allocate xxxx bytes of memory ... =

← Older revision		Revision as of 17:45, 31 May 2015
Line 130:		Line 130:
	However, sudo will probably reset certain environment variables, particularly LD_LIBRARY_PATH, so you may need to reset those in the new shell.		However, sudo will probably reset certain environment variables, particularly LD_LIBRARY_PATH, so you may need to reset those in the new shell.

	=== ERROR: G_malloc: unable to allocate xxxx bytes of memory ... ====		=== ERROR: G_malloc: unable to allocate xxxx bytes of memory ... ===

	Be sure to use GRASS GIS 7 which has way better support for large files.		Be sure to use GRASS GIS 7 which has way better support for large files.

	If you happen to use the "memory" option requesting more than 2GB (e.g. <tt>memory=3000</tt>) check if you are using 64bit executables since 32bit applications can use ~~max~~ 2GB memory. In this case it does not matter if the OS is 64bit or if there is more than 2GB of memory available. You need to use a 64bit version of GRASS GIS 7.		If you happen to use the "memory" option requesting more than 2GB (e.g. <tt>memory=3000</tt>) check if you are using 64bit executables since 32bit applications can use only up to 2GB memory. In this case it does not matter if the operating system is 64bit or if there is more than 2GB of memory available. You need to use a 64bit version of GRASS GIS 7.

	== See also ==		== See also ==

Neteler: +Solution for Windows: System-wise change

2015-05-31T12:23:25Z

+Solution for Windows: System-wise change

← Older revision		Revision as of 12:23, 31 May 2015
Line 113:		Line 113:

	Restart the user session (i.e., logout, login).		Restart the user session (i.e., logout, login).

			==== Solution for Windows: System-wise change ====

			From the Microsoft documentation it is not very clear how many files can be opened simulaneously (FIX IF POSSIBLE), here some hints:

			* https://support.microsoft.com/en-us/kb/87690
			* ...

	==== Solution for Linux: Session only change ====		==== Solution for Linux: Session only change ====

⚠️Madi at 09:14, 24 April 2015

2015-04-24T09:14:14Z

← Older revision		Revision as of 09:14, 24 April 2015
Line 135:		Line 135:
	[[Category:Tutorial]]		[[Category:Tutorial]]
	[[Category: massive data analysis]]		[[Category: massive data analysis]]
			[[Category: raster]]

← Older revision		Revision as of 14:54, 8 May 2017
Line 139:		Line 139:

	* [[GRASS GIS Performance]]		* [[GRASS GIS Performance]]
			* [[Large vector data processing]]

	[[Category:Tutorial]]		[[Category:Tutorial]]
	[[Category: massive data analysis]]		[[Category: massive data analysis]]
	[[Category: raster]]		[[Category: raster]]