|
|
| (22 intermediate revisions by 5 users not shown) |
| Line 1: |
Line 1: |
| == The need ==
| | {{MovedToTrac|LargeFileSupport}} |
|
| |
|
| Standard C <stdio.h> file functions return file sizes as long integer. On 32-bit systems this overflows at 2 gigabytes. For support of files bigger than this, you need LFS. Currently only implimented in GRASS in libgis. (i.e. there is support for reading+writing raster maps, but not many import/export modules or vector functions have it)
| | [[Category: Development]] |
| | | [[Category: massive data analysis]] |
| | |
| == The issues ==
| |
| | |
| The problem is that ftell() returns the result as a (signed) long. If
| |
| the result won't fit into a long, it returns -1 (and sets errno to
| |
| EOVERFLOW).
| |
| | |
| This can only happen if you also set _FILE_OFFSET_BITS to 64 so that
| |
| fopen() is redirected to fopen64(), otherwise fopen() will simply
| |
| refuse to open files larger than 2GiB (apparently, this isn't true on
| |
| some versions of MacOSX, which open the file anyhow then fail on
| |
| fseek/ftell once you've passed the 2GiB mark).
| |
| | |
| If you want to obtain the current offset for a file whose size exceeds
| |
| the range of a signed long, you instead have to use the (non-ANSI)
| |
| ftello() function, which returns the offset as an off_t. '''''TODO:''''' But before we do that, we would need to add configure checks so that we don't try to use ftello() on systems which don't provide it.
| |
| | |
| == Coding LFS in GRASS ==
| |
| | |
| Currently the <tt>--enable-largefile</tt> switch only enables LFS in libgis, not anywhere else.
| |
| | |
| [Although config.h includes definitions to enable LFS automatically, | |
| those definitions are currently inactive. This is probably a good
| |
| thing; a lot of GRASS' code isn't LFS-aware, and explicit failure is
| |
| preferable to silently corrupting data.]
| |
| | |
| To enable LFS elsewhere, you need to manually add
| |
| -D_FILE_OFFSET_BITS=64 to the compilation flags. The simplest approach
| |
| is to add to the module's Makefile:
| |
| | |
| ifneq ($(USE_LARGEFILES),)
| |
| EXTRA_CFLAGS = -D_FILE_OFFSET_BITS=64
| |
| endif
| |
| | |
| and add include '''config.h''' before any other header files in the code.
| |
| #include <grass/config.h>
| |
| | |
| '''''Question: does that mean <stdio.h> or <gis.h> ???'''''
| |
| | |
| | |
| == LFS safe libs and module list ==
| |
| * libgis
| |
| * r.in.xyz (work in progress)
| |
| | |
| == LFS wish list ==
| |
| '''High priority modules to get LFS'''
| |
| * r.in.*
| |
| * r.out.*
| |
| * GRASS GDAL plugin (??)
| |
| * r.terraflow (intregrate current LFS support into GRASS's --enable-largefile ./configure switch)
| |
| * v.surf.rst
| |
| * v.surf.idw(2)
| |
| * vector libs (limited by number of features)
| |
| * v.in.ascii -bt (without topology)
| |
| * DB libs
| |