|
|
| (4 intermediate revisions by one other user not shown) |
| Line 1: |
Line 1: |
| {{MoveToTrac}} | | {{MovedToTrac|LargeFileSupport}} |
| (largely based on comments by Glynn Clements on the GRASS-dev mailing list)
| |
| | |
| == The need ==
| |
| | |
| Standard C <stdio.h> file functions return file sizes as long integer. On 32-bit systems this overflows at 2 gigabytes. For support of files bigger than this, you need LFS. Currently only implimented in GRASS in libgis. (i.e. there is support for reading+writing raster maps, but not many import/export modules or vector functions have it)
| |
| | |
| In GRASS, configure.in/configure have been updated to support --enable-largefile based on code from "cdr-tools". See [http://trac.osgeo.org/grass/browser/grass/trunk/configure.in configure.in] (around line 1610).
| |
| | |
| == The issues ==
| |
| | |
| The problem is that ftell() returns the result as a (signed) long. If
| |
| the result won't fit into a long, it returns -1 (and sets errno to
| |
| EOVERFLOW).
| |
| | |
| This can only happen if you also set _FILE_OFFSET_BITS to 64 so that
| |
| fopen() is redirected to fopen64(), otherwise fopen() will simply
| |
| refuse to open files larger than 2GiB (apparently, this isn't true on
| |
| some versions of MacOSX, which open the file anyhow then fail on
| |
| fseek/ftell once you've passed the 2GiB mark).
| |
| | |
| If you want to obtain the current offset for a file whose size exceeds
| |
| the range of a signed long, you instead have to use the (non-ANSI)
| |
| ftello() function, which returns the offset as an off_t.
| |
| | |
| '''''TODO:''''' But before we do that, we would need to add configure checks so that we don't try to use ftello() on systems which don't provide it.
| |
| | |
| | |
| There isn't a truly portable solution. Some platforms might not even
| |
| have an integral type larger than 32 bits.
| |
| | |
| The most practicaly solution is to use ftello() if it's available.
| |
| | |
| This will require some configure checks. These are simple enough to
| |
| implement; it's the design which is problematic (as usual).
| |
| | |
| Unlike most HAVE_FOO checks, fseeko() isn't a simple have/don't-have
| |
| check. Rather, it's usually a case that the function is available only
| |
| when certain macros are defined (e.g. _LARGEFILE_SOURCE).
| |
| | |
| That gives rise to the question of what we check for, how we check for
| |
| it, how we pass that information to the code, and how we use it.
| |
| | |
| | |
| The trick is deciding what to test, and what to indicate. Do we want
| |
| to know:
| |
| | |
| 1. Whether ftello() exists with the default CFLAGS/CPPFLAGS?
| |
| | |
| 2. Whether ftello() exists with the default CFLAGS/CPPFLAGS plus a
| |
| fixed selection of additional switches (e.g. -D_LARGEFILE_SOURCE)?
| |
| | |
| 3. Whether ftello() exists with the default CFLAGS/CPPFLAGS plus a
| |
| variable selection of additional switches (e.g. -D_LARGEFILE_SOURCE)
| |
| with those switches being communicated via an additional variable?
| |
| | |
| | |
| The test would need to be:
| |
| #if defined(HAVE_FTELLO) && defined(HAVE_FSEEKO)
| |
| as you are using both of those.
| |
| | |
| Also, if the existence of FSEEKO/FTELLO is conditional upon certain
| |
| macros (e.g. _LARGEFILE_SOURCE), you have to ensure that those macros
| |
| are defined before the first inclusion of <stdio.h>, including any
| |
| "hidden" inclusion from other headers, which essentially means that
| |
| you have to ensure that the macros are defined before you include any
| |
| other headers (except for <grass/config.h>).
| |
| | |
| | |
| Rather than try to come up with some infrastructure which allows us to
| |
| use LFS in a piecemeal fashion, it would be preferable to clean up the
| |
| GRASS code so that we can enable LFS globally. Then, we can just add:
| |
| | |
| -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64
| |
| | |
| to CPPFLAGS, and not have to worry about adding the necessary macros
| |
| to individual files. Any HAVE_* checks then become simple
| |
| have/don't-have checks.
| |
| | |
| | |
| My inclination would be to add:
| |
| | |
| extern off_t G_ftell(FILE *fp);
| |
| extern int G_fseek(FILE *stream, off_t offset, int whence);
| |
| | |
| These would be implemented using fseeko/ftello where available, and
| |
| fseek/ftell otherwise. That eliminates the need to perform checks in
| |
| individual source files. However, we would need to take care not to
| |
| use them in code which can't handle large offsets (i.e. code which
| |
| will truncate off_t values to int/long).
| |
| | |
| | |
| If you use -D_FILE_OFFSET_BITS=64, fopen() should be redirected to
| |
| fopen64(). There are reports that the MacOSX fopen() is equivalent to
| |
| fopen64() even without additional switches.
| |
| | |
| == Coding LFS in GRASS ==
| |
| | |
| Currently the <tt>--enable-largefile</tt> switch only enables LFS in libgis, not anywhere else.
| |
| | |
| [Although config.h includes definitions to enable LFS automatically,
| |
| those definitions are currently inactive. This is probably a good
| |
| thing; a lot of GRASS' code isn't LFS-aware, and explicit failure is
| |
| preferable to silently corrupting data.]
| |
| | |
| To enable LFS elsewhere, you need to manually add
| |
| -D_FILE_OFFSET_BITS=64 to the compilation flags. The simplest approach
| |
| is to add to the module's Makefile:
| |
| | |
| ifneq ($(USE_LARGEFILES),)
| |
| EXTRA_CFLAGS = -D_FILE_OFFSET_BITS=64
| |
| endif
| |
| | |
| and add include <tt>config.h</tt> before '''all''' other header files in the code.
| |
| #include <grass/config.h>
| |
| #include <stdio.h>
| |
| #include <string.h>
| |
| #include <grass/gis.h>
| |
| ...
| |
| | |
| === int versus off_t ===
| |
| You may as well just use "<tt>off_t filesize</tt>" unconditionally. An
| |
| "off_t" will always be large enough to hold a "long".
| |
| | |
| If using "<tt>off_t</tt>", be sure to add:
| |
| #include <sys/types.h>
| |
| | |
| == Issues related to import/export of maps ==
| |
| | |
| Some exporting/inporting formats have their own intrinsic limitations, see for instance
| |
| http://www.gdal.org/formats_list.html
| |
| | |
| == LFS-safe libs and module list ==
| |
| * libgis
| |
| | |
| * r.in.arc
| |
| * r.in.ascii
| |
| * r.out.arc
| |
| * r.out.ascii
| |
| * r.proj.seg
| |
| * r.terraflow
| |
| | |
| == LFS works in progress ==
| |
| | |
| * r.in.xyz
| |
| * r.terraflow (intregrate current LFS support into GRASS's --enable-largefile ./configure switch)<BR>(r.terraflow creates huge temporary files which can easily go over 2GB)
| |
| | |
| == LFS wish list ==
| |
| '''High priority modules to get LFS'''
| |
| * r.in.*
| |
| * r.out.*
| |
| * GRASS GDAL plugin (??)
| |
| * v.surf.rst
| |
| * v.surf.idw(2)
| |
| * vector libs (limited by number of features)
| |
| * v.in.ascii -bt (without topology)
| |
| * DB libs
| |
| | |
| == References ==
| |
| | |
| * [http://www.suse.de/~aj/linux_lfs.html LFS support in Linux]
| |
| * [http://opengroup.org/platform/lfs.html Adding Large File Support to the Single UNIX® Specification]
| |
| * [http://www.sun.com/software/whitepapers/wp-largefiles/largefiles.pdf Large Files in Solaris: A White Paper]
| |
|
| |
|
| [[Category: Development]] | | [[Category: Development]] |
| [[Category: massive data analysis]] | | [[Category: massive data analysis]] |