Large File Support - Revision history

⚠️NikosA: Moved to Trac

2016-09-11T05:22:10Z

Moved to Trac

Neteler: added Glynn's comments from ML: http://lists.osgeo.org/pipermail/grass-dev/2014-June/069805.html

2014-08-17T22:47:40Z

added Glynn's comments from ML: http://lists.osgeo.org/pipermail/grass-dev/2014-June/069805.html

← Older revision		Revision as of 22:47, 17 August 2014
Line 159:		Line 159:

	* Already enabled for raster and vector libraries and modules, see http://trac.osgeo.org/grass/wiki/Grass7/NewFeatures		* Already enabled for raster and vector libraries and modules, see http://trac.osgeo.org/grass/wiki/Grass7/NewFeatures

			'''Q:''' Is xyz module (e.g. r.texture) supporting LFS?

			'''A:''' Modules typically don't need to do anything regarding LFS; the support is in the libraries. The main issue which affects modules is that they shouldn't assume that cell counts will fit into an "int" or even a "long". But even failing to do so won't have any effect upon I/O.

			'''Details:'''

			The most common way for the issue to arise is multiplying the number
			of rows by the number of columns to obtain the total number of cells.
			Most modules have no need to do this, but it is occasionally done e.g.
			when calculating statistics or storing the data in a temporary file.

			The number of rows and columns can reasonably be assumed to fit into a
			signed 32-bit integer, but their product cannot.

			Even on a system with a 64-bit "long"[1], multiplying 2 "int"s will
			produce an "int" result, and assigning the result to a "long" variable
			doesn't change that. E.g.

			int nrows = Rast_window_rows();
			int ncols = Rast_window_cols();
			long ncells = nrows * ncols;

			will truncate the result of the multiplication to an "int" (which is
			32 bits on all mainstream platforms) then expand the truncated value
			to 64 bits in the assignment. To perform the multplication using
			"long", one of the arguments must be converted, e.g.:

			long ncells = (long) nrows * ncols;

			[1] This doesn't include 64-bit versions of Windows, where "long" is only 32 bits for compatibility reasons.

			The issue isn't strictly related to LFS; due to compression, it's
			possible for a raster with more than 2^31 cells to take up less than
			2 GiB on disk. LFS just makes the issue more likely to arise in practice.

	== References ==		== References ==

Neteler: /* The need */

2014-06-25T10:17:41Z

The need

← Older revision		Revision as of 10:17, 25 June 2014
Line 4:		Line 4:
	== The need ==		== The need ==

	Standard C <stdio.h> file functions return file sizes as long integer. On 32-bit systems this overflows at 2 gigabytes. For support of files bigger than this, you need LFS. ~~Currently~~ only ~~implimented in GRASS~~ in libgis. (i.e. there is support for reading+writing raster maps, but not many import/export modules or vector functions have it)		Standard C <stdio.h> file functions return file sizes as long integer. On 32-bit systems this overflows at 2 gigabytes. For support of files bigger than this, you need LFS. In GRASS GIS 6, this is only implemented in libgis. (i.e. there is support for reading+writing raster maps, but not many import/export modules or vector functions have it). The situation is much better in GRASS GIS 7.

	In GRASS, configure.in/configure have been updated to support --enable-largefile based on code from "cdr-tools". See [http://trac.osgeo.org/grass/browser/grass/trunk/configure.in configure.in] (around line 1610).		In GRASS, configure.in/configure have been updated to support --enable-largefile (based on code from "cdr-tools"). See [http://trac.osgeo.org/grass/browser/grass/trunk/configure.in configure.in] (around line 1610).

	== The issues ==		== The issues ==

Neteler: /* Coding LFS in GRASS GIS 7 */

2013-04-20T09:18:05Z

Coding LFS in GRASS GIS 7

← Older revision		Revision as of 09:18, 20 April 2013
Line 158:		Line 158:
	== Coding LFS in GRASS GIS 7 ==		== Coding LFS in GRASS GIS 7 ==

	* enabled for raster and vector libraries and modules		* Already enabled for raster and vector libraries and modules, see http://trac.osgeo.org/grass/wiki/Grass7/NewFeatures

	== References ==		== References ==

Neteler: tiny update

2013-03-08T11:29:22Z

tiny update

← Older revision		Revision as of 11:29, 8 March 2013
Line 54:		Line 54:
	variable selection of additional switches (e.g. -D_LARGEFILE_SOURCE)		variable selection of additional switches (e.g. -D_LARGEFILE_SOURCE)
	with those switches being communicated via an additional variable?		with those switches being communicated via an additional variable?


	The test would need to be:		The test would need to be:
Line 95:		Line 94:
	fopen64() even without additional switches.		fopen64() even without additional switches.

	== Coding LFS in GRASS ==		== Coding LFS in GRASS GIS 6 ==

	Currently the <tt>--enable-largefile</tt> switch only enables LFS in libgis, not anywhere else.		Currently the <tt>--enable-largefile</tt> switch only enables LFS in libgis, not anywhere else.
Line 131:		Line 130:
	http://www.gdal.org/formats_list.html		http://www.gdal.org/formats_list.html

	== LFS-safe libs and module list ==		== GRASS GIS 6: LFS-safe libs and module list ==
	* libgis		* libgis

Line 141:		Line 140:
	* r.terraflow		* r.terraflow

	== LFS works in progress ==		== GRASS GIS 6: LFS works in progress ==

	* r.in.xyz		* r.in.xyz
	* r.terraflow (intregrate current LFS support into GRASS's --enable-largefile ./configure switch)<BR>(r.terraflow creates huge temporary files which can easily go over 2GB)		* r.terraflow (intregrate current LFS support into GRASS's --enable-largefile ./configure switch)<BR>(r.terraflow creates huge temporary files which can easily go over 2GB)

	== LFS wish list ==		== GRASS GIS 6: LFS wish list ==
	'''High priority modules to get LFS'''		'''High priority modules to get LFS'''
	* r.in.*		* r.in.*
Line 156:		Line 155:
	* v.in.ascii -bt (without topology)		* v.in.ascii -bt (without topology)
	* DB libs		* DB libs

			== Coding LFS in GRASS GIS 7 ==

			* enabled for raster and vector libraries and modules

	== References ==		== References ==

⚠️Madi: /* References */

2013-02-11T13:51:49Z

References

← Older revision		Revision as of 13:51, 11 February 2013
Line 164:		Line 164:

	[[Category: Development]]		[[Category: Development]]
			[[Category: massive data analysis]]

⚠️Landa: +template MoveToTrac

2008-04-12T23:46:09Z

+template MoveToTrac

← Older revision		Revision as of 23:46, 12 April 2008
Line 1:		Line 1:
			{{MoveToTrac}}
	(largely based on comments by Glynn Clements on the GRASS-dev mailing list)		(largely based on comments by Glynn Clements on the GRASS-dev mailing list)

Neteler: mention configure.in

2008-02-26T12:42:35Z

mention configure.in

← Older revision		Revision as of 12:42, 26 February 2008
Line 5:		Line 5:
	Standard C <stdio.h> file functions return file sizes as long integer. On 32-bit systems this overflows at 2 gigabytes. For support of files bigger than this, you need LFS. Currently only implimented in GRASS in libgis. (i.e. there is support for reading+writing raster maps, but not many import/export modules or vector functions have it)		Standard C <stdio.h> file functions return file sizes as long integer. On 32-bit systems this overflows at 2 gigabytes. For support of files bigger than this, you need LFS. Currently only implimented in GRASS in libgis. (i.e. there is support for reading+writing raster maps, but not many import/export modules or vector functions have it)

			In GRASS, configure.in/configure have been updated to support --enable-largefile based on code from "cdr-tools". See [http://trac.osgeo.org/grass/browser/grass/trunk/configure.in configure.in] (around line 1610).

	== The issues ==		== The issues ==

⚠️FrancescoLovergine at 15:03, 14 April 2007

2007-04-14T15:03:23Z

← Older revision		Revision as of 15:03, 14 April 2007
Line 155:		Line 155:
	* DB libs		* DB libs

			== References ==

			* [http://www.suse.de/~aj/linux_lfs.html LFS support in Linux]
			* [http://opengroup.org/platform/lfs.html Adding Large File Support to the Single UNIX® Specification]
			* [http://www.sun.com/software/whitepapers/wp-largefiles/largefiles.pdf Large Files in Solaris: A White Paper]

	[[Category: Development]]		[[Category: Development]]

Neteler: list of LFS safe modules

2007-04-14T05:25:30Z

list of LFS safe modules

← Older revision		Revision as of 05:25, 14 April 2007
Line 131:		Line 131:
	== LFS-safe libs and module list ==		== LFS-safe libs and module list ==
	* libgis		* libgis

			* r.in.arc
			* r.in.ascii
			* r.out.arc
			* r.out.ascii
			* r.proj.seg
			* r.terraflow

	== LFS works in progress ==		== LFS works in progress ==