Large File Support: Difference between revisions

From GRASS-Wiki
Jump to navigation Jump to search
(credit glynn's help)
m (Moved to Trac)
 
(15 intermediate revisions by 5 users not shown)
Line 1: Line 1:
(largely based on comments by Glynn Clements on the GRASS-dev mailing list)
{{MovedToTrac|LargeFileSupport}}
 
== The need ==
 
Standard C <stdio.h> file functions return file sizes as long integer. On 32-bit systems this overflows at 2 gigabytes. For support of files bigger than this, you need LFS. Currently only implimented in GRASS in libgis. (i.e. there is support for reading+writing raster maps, but not many import/export modules or vector functions have it)
 
 
== The issues ==
 
The problem is that ftell() returns the result as a (signed) long. If
the result won't fit into a long, it returns -1 (and sets errno to
EOVERFLOW).
 
This can only happen if you also set _FILE_OFFSET_BITS to 64 so that
fopen() is redirected to fopen64(), otherwise fopen() will simply
refuse to open files larger than 2GiB (apparently, this isn't true on
some versions of MacOSX, which open the file anyhow then fail on
fseek/ftell once you've passed the 2GiB mark).
 
If you want to obtain the current offset for a file whose size exceeds
the range of a signed long, you instead have to use the (non-ANSI)
ftello() function, which returns the offset as an off_t. '''''TODO:''''' But before we do that, we would need to add configure checks so that we don't try to use ftello() on systems which don't provide it.
 
 
== Coding LFS in GRASS ==
 
Currently the <tt>--enable-largefile</tt> switch only enables LFS in libgis, not anywhere else.
 
[Although config.h includes definitions to enable LFS automatically,
those definitions are currently inactive. This is probably a good
thing; a lot of GRASS' code isn't LFS-aware, and explicit failure is
preferable to silently corrupting data.]
 
To enable LFS elsewhere, you need to manually add
-D_FILE_OFFSET_BITS=64 to the compilation flags. The simplest approach
is to add to the module's Makefile:
 
ifneq ($(USE_LARGEFILES),)
EXTRA_CFLAGS = -D_FILE_OFFSET_BITS=64
endif
 
and add include <tt>config.h</tt> before '''all''' other header files in the code.
#include <grass/config.h>
#include <stdio.h>
#include <string.h>
#include <grass/gis.h>
...
 
=== int versus off_t ===
You may as well just use "off_t filesize" unconditionally. An
"off_t" will always be large enough to hold a "long".
 
== LFS-safe libs and module list ==
* libgis
 
 
== LFS works in progress ==
 
* r.in.xyz
* r.terraflow (intregrate current LFS support into GRASS's --enable-largefile ./configure switch)<BR>(r.terraflow creates huge temporary files which can easily go over 2GB)
 
== LFS wish list ==
'''High priority modules to get LFS'''
* r.in.*
* r.out.*
* GRASS GDAL plugin (??)
* v.surf.rst
* v.surf.idw(2)
* vector libs (limited by number of features)
* v.in.ascii -bt  (without topology)
* DB libs
 


[[Category: Development]]
[[Category: Development]]
[[Category: massive data analysis]]

Latest revision as of 05:22, 11 September 2016