Global datasets: Difference between revisions

From GRASS-Wiki
Jump to navigation Jump to search
(Fix coordinates of ESA Globcover 2009)
Line 578: Line 578:
* [http://mars.jrc.ec.europa.eu/mars/About-us/AGRI4CAST/Data-distribution MARS @ JRC] Temperature, vapour pressure, rainfall, relative humidity, cloud cover, solar radiation, wind speed.
* [http://mars.jrc.ec.europa.eu/mars/About-us/AGRI4CAST/Data-distribution MARS @ JRC] Temperature, vapour pressure, rainfall, relative humidity, cloud cover, solar radiation, wind speed.
* [http://www.efas.eu/ EFAS @ JRC] is a High resolution pan-European dataset for hydrologic modelling.
* [http://www.efas.eu/ EFAS @ JRC] is a High resolution pan-European dataset for hydrologic modelling.
* [http://data.jrc.it/ JRC Data Catalogue] This web site provides an inventory of data that is being created and held by the JRC. The content is being continuously updated.


=== National datasets ===
=== National datasets ===

Revision as of 11:14, 29 October 2015

Raster data

Elevation data

ASTER topography (GDEM V1)

Improved ASTER GDEM 1 from 2009:

GDEM global 30m elevation calculated from stereo-pair images collected by the Terra satellite. "This is the most complete, consistent global digital elevation data yet made available to the world." This is a very new dataset, at version 1 (treat as experimental). Accuracy will be improved in forthcoming versions (validation with SRTM, etc.; see assessment here and here).

Tutorial: ASTER topography

See also: ASTER GDEM 30m quality assessment

ASTER topography (GDEM V2)

Improved ASTER GDEM 2 from 2011:

The ASTER GDEM covers land surfaces between 83°N and 83°S and is comprised of 22,702 tiles. Tiles that contain at least 0.01% land area are included. The ASTER GDEM is distributed as Geographic Tagged Image File Format (GeoTIFF) files with geographic coordinates (latitude, longitude). The data are posted on a 1 arc-second (approximately 30–m at the equator) grid and referenced to the 1984 World Geodetic System (WGS84)/ 1996 Earth Gravitational Model (EGM96) geoid.

Notes: this DEM can be rather well filtered and smoothed with the Sun's denoising algorithm (using GDAL and free / open source program <mdenoise> or simply GRASS add-on r.denoise.

Experiments showed that the best smoothing of ASTER GDEM 2 is reached with such parameters of <mdenoise>:

  • threshold = 0.8
  • iterations = 10-20

Also filtering with r.neighbors by "average" method and window size >=5 is quite useful to remove some noise from DEM.

See also: Validation of the ASTER Global Digital Elevation Model Version 2 over the Conterminous United States

ACE2

The ACE2 Global Digital Elevation Model is available at 3", 30" and 5' spatial resolutions.

Import example: 

 r.in.bin -f input="00N105E_3S.ACE2" output="ACE2_00N105E" bytes=4 \
          order="native" north=15 south=0 east=120 west=105 \
          rows=18000 cols=18000

CleanTOPO2 (DEM)

Import in GRASS: 

 r.in.gdal CleanTOPO2.tif out=cleanTOPO2.tmp -l -o
 g.region rast=cleanTOPO2  -p -g
 # rescale from odd integer values to true world values
 r.rescale cleanTOPO2.tmp out=cleanTOPO2 to=-10701,8248
 r.colors cleanTOPO2_final col=terrain
Rescaled ClearTOPO2 map

EGM2008 Geoid Data (Earth Gravitational Model)

Global 2.5 Minute Geoid Undulations:

Geoid undulations in Trentino, Italy

Verifications of points can be done with the http://geographiclib.sourceforge.net/cgi-bin/GeoidEval

ETOPO (DEM)

The ETOPO datasets provide global topography and bathymetry at 1', 2', and 5' per-cell resolutions.

ETOPO1 (DEM)

  • The cell registered version can be loaded directly into a lat/lon location. GRASS raster data is cell registered (see the GRASS raster semantics page)


  • Special care must be taken with the grid registered version. It can not be loaded directly into a lat/lon location as the parameters found in the .hdr file exceed the limits of polar coordinate space: they have N,S rows which go 1/2 a cell beyond 90 latitude, when considered in the cell registered convention.
So the data needs to have those 90deg N,S rows cropped away, and while we're at it we crop away a redundant overlapping column at 180 longitude. To do this we have to first tell the GIS a little fib during import to squeeze the data into lat/lon space, then crop away the spurious rows and column, then finally reset the resulting map's bounds to its true extent.
 # Import grid registered binary float, fibbing about its true extent
 r.in.bin -f in=etopo1_bed_g.flt out=etopo1_bed_g.raw \
    n=90 s=-90 e=180 w=-180 rows=10801 cols=21601 anull=-9999
 
 # reduce the working region by 1 cell
 g.region rast=etopo1_bed_g.raw
 eval `g.region -g`
 g.region n=n-$nsres s=s+$nsres e=e-$ewres -p
 
 # save smaller raster and remove original
 r.mapcalc "etopo1_bed_g.crop = etopo1_bed_g.raw"
 g.remove etopo1_bed_g.raw

 # re-establish the correct bounds, now that they'll fit
 r.region etopo1_bed_g.crop n=89:59:30N s=89:59:30S w=179:59:30E e=179:59:30E
 g.region rast=etopo1_bed_g.crop

 # check that N,S,E,W and Res are all nice and clean:
 r.info etopo1_bed_g.crop

 # looks good, so accept the results by resetting the map name
 g.rename etopo1_bed_g.crop,etopo1_bed_g

 # set to use appropriate color rules
 r.colors etopo1_bed_g color=etopo2

 # set the 'units' metadata field (for elevation data contained within the map)
 r.support etopo1_bed_g units=meters
  • For the problematic grid registered version, the resulting r.info report should look like:
|   Rows:         10799                                                      |
|   Columns:      21600                                                      |
|   Total Cells:  233258400                                                  |
|        Projection: Latitude-Longitude                                      |
|            N:  89:59:30N    S:  89:59:30S   Res:  0:01                     |
|            E: 179:59:30E    W: 179:59:30E   Res:  0:01                     |
|   Range of data:    min = -10898  max = 8271                               |

(the east and west bounds of the map touch 1/2 a cell west of 180 longitude)

  • For the problematic grid registered version, since the data's grid is 1/2 a cell shifted from nicely rounded 1 arc-minutes (0:01), you'll need to ensure that the mapset's region preserves that alignment after zooming or panning:
g.region align=etopo1_bed_g -p
(or oversample and set the region resolution to 1/2 arc-minutes (0:00:30), which will be four times as slow)

ETOPO2 (DEM)

ETOPO2v2 data download (take for example the ETOPO2v2g_f4_LSB.flt file)

GTOPO30 (DEM)

Note: To avoid that the GTOPO30 data are read incorrectly, you can add a new line "PIXELTYPE SIGNEDINT" in the .HDR to force interpretation of the file as signed rather than unsigned integers. Then the .DEM file can be imported. Finally, e.g. the 'terrain' color table can be assigned to the imported map with r.colors.


Global Multi-resolution Terrain Elevation Data 2010 (GMTED2010)

Import of GMTED2010 tiles in GRASS GIS: 

 r.in.gdal 30N000E_20101117_gmted_mea075.tif out=gmted2010_30N000E_20101117
 r.colors gmted2010_30N000E_20101117 color=elevation
 g.region rast=gmted2010_30N000E_20101117
 r.shaded.relief gmted2010_30N000E_20101117 units=meters
 d.mon x0
 d.rast gmted2010_30N000E_20101117.shade
 d.grid 1 color=red textcolor=red
GMTED2010 example: Trento - Garda Lake - Verona area (Northern Italy)

GEBCO Bathymetric Chart

  • The General Bathymetric Chart of the Oceans (original 1' release 2003, new 1' and 30" releases 2008)
http://www.gebco.net/data_and_products/gridded_bathymetry_data/
http://www.bodc.ac.uk/data/online_delivery/gebco/

r.in.gdal can be used to import the GMT netCDF files directly, or if that doesn't work you can use GMT tools to convert to an old-style native GMT format and import that with r.in.bin.

example: (GEBCO 2003 1' data)
# convert to an old style GMT binary .grd using grdreformat
$ grdreformat 3n24s47w14w.grd 3n24s47w14w_Native.grd=bs

# then import into GRASS,
GRASS> r.in.bin -h -s bytes=2 in=3n24s47w14w_Native.grd out=3n24s47w14w

# and set some nice colors
GRASS> r.colors 3n24s47w14w rules=- << EOF
nv magenta
0% black
-7740 0:0:168
0 84:176:248
0 40:124:0
522 68:148:24
1407 148:228:108
1929 232:228:108
2028 232:228:92
2550 228:160:32
2724 216:116:8
2730 grey
2754 grey
2760 252:252:252
2874 252:252:252
2883 192:192:192
2913 192:192:192
100% 252:252:252
EOF

Global Multi-Resolution Topography (GMRT DEM)

From Columbia University's Lamont-Doherty Earth Observatory

(it is reported that this is what Google Maps uses for their global bathymetry) 

 export `g.region -g`
 wget "http://www.marine-geo.org/cgi-bin/getgridB?west=${w}&east=${e}&south=${s}&north=${n}&resolution=1" -O /tmp/test.grd
 r.in.gdal /tmp/test.grd output=GMRT -o
 rm /tmp/test.grd
  • Note: Downloaded file contains no projection information, but is EPSG:4326 (WGS84 Geographic). The file size is limited, but lower resolution (resolution=2,4,8) data can be downloaded for larger areas.

Smith and Sandwell DEM

SRTM DEM

Space Shuttle Radar Topography Mission

SRTM30plus data DEM

SRTM30plus data consists of 33 files of global topography in the same format as the SRTM30 products distributed by the USGS EROS data center. The grid resolution is 30 second which is roughly one kilometer.

Land data are based on the 1-km averages of topography derived from the USGS SRTM30 grided DEM data product created with data from the NASA Shuttle Radar Topography Mission. GTOPO30 data are used for high latitudes where SRTM data are not available.

Ocean data are based on the Smith and Sandwell global 2-minute grid between latitudes +/- 72 degrees. Higher resolution grids have been added from the LDEO Ridge Multibeam Synthesis Project and the NGDC Coastal Relief Model. Arctic bathymetry is from the International Bathymetric Chart of the Oceans (IBCAO).

All data are derived from public domain sources and these data are also in the public domain.

GRASS 6 script r.in.srtm described in GRASSNews vol. 3 won't work with this dataset (as it was made for the original SRTM HGT files). But you can import SRTM30plus tiles into GRASS this way: 

r.in.bin -sb input=e020n40.Bathmetry.srtm output=e020n40_topex bytes=2 north=40 south=-10 east=60 west=20 r=6000 c=4800
r.colors e020n40_topex rules=etopo2
Source
GRASS Users Mailing List http://lists.osgeo.org/pipermail/grass-user/2005-August/030063.html
Getting as SRTM30plus tiles
ftp://topex.ucsd.edu/pub/srtm30_plus/srtm30/data/
Getting as SRTM30plus huge file
ftp://topex.ucsd.edu/pub/srtm30_plus/topo30/
SRTMPLUS WCS server
http://svn.osgeo.org/gdal/trunk/autotest/gdrivers/data/srtmplus.wcs (read with r.external

Soil data

Harmonized World Soil Database (HWSD Database)

Download: http://webarchive.iiasa.ac.at/Research/LUC/External-World-soil-database/HTML/

Spatial reference system: EPSG:4326 (LatLong WGS84)

Import: 

grass70 -c EPSG:4326 ~/grassdata/hwsd
# -e: expand location to dataset; -o: override (missing) projection in input dataset:
r.in.gdal input=hwsd.bil output=hwSoil -e -o
g.region rast=hwSoil -p
r.category hwSoil

The data is distributed with an MSAccess .mdb which contains additional data for each of the categories in the raster file. Opening the file in access, the data is found in the query "HWSD_Q". Save this query in .csv format (with a name like "HWSD_Q.csv") so that it may then be imported into GRASS. Before you can import it, you also need a file "HWSD_Q.csvt", which contains a single line listing the type for each column in the database: 

"Integer","String","Integer","Integer","Integer","String","Integer","Integer","Real","Integer","String","Integer","String","Integer","String","Integer","Integer","Integer","Integer","Integer","Integer","Integer","Integer","Integer","Integer","Integer","Integer","Integer","Integer","Integer","Integer","Real","Real","Real","Real","Real","Real","Real","Real","Real","Real","Real","Real","Integer","Integer","Integer","Integer","Integer","Real","Real","Real","Real","Real","Real","Real","Real","Real","Real","Real","Real"


With both the .csv and the .csvt file in the same directory, you can then import them into GRASS: 

db.in.ogr dsn=~/grassdata/hwsd/HWSD_Q.csv output=hwsdData

The data cannot be connected directly to the raster, it must be converted to a vector first: 

g.region rast=hwSoil
r.to.vect -v in=hwSoil out=hwSoil feature=area
v.db.droptable hwSoil
db.droptable -f hwSoil # delete the table completely

Note that the table includes multiple rows for each polygon, corresponding to the dominant and various numbers of subdominant soils. To select only the dominant soil layer: 

db.select table=hwsdData sql='select * from hwsdData where SEQ = 1' \
         output=domSoil.csv fs=,

This saves a copy of the table that contains only the dominant soil type for each polygon as domSoil.csv. This needs to be reloaded into the GRASS database. Since it has the same columns as HWSD_Q.csv, we can use the labels for that file: 

cp HWSD_Q.csvt domSoil.csvt

Then we can load domSoil.csv: 

db.in.ogr \
   dsn=~/grassdata/downloads/harmonized_world_soil_database/domSoil.csv \
   output=domSoil

Now at last we can connect the database to the vector file: 

v.db.connect -o map=hwsd table=domSoil driver=sqlite key=MU_GLOBAL

To create a new raster map taking the values from the table: 

g.region rast=hwSoil  ## make sure we get the whole map
v.to.rast in=hwSoil out=T_SAND col=T_SAND

Landcover data

ESA Globcover dataset

Download: http://due.esrin.esa.int/page_globcover.php

Or via command line: 

wget http://due.esrin.esa.int/files/Globcover2009_V2.3_Global_.zip
unzip Globcover2009_V2.3_Global_.zip
# rm -f Globcover2009_V2.3_Global_.zip

Unfortunately the Globcover map exceeds the -180°..+180° range etc, indicating a shift of the map (see also this assessment by DWD): 

gdalinfo GLOBCOVER_L4_200901_200912_V2.3.tif
Driver: GTiff/GeoTIFF
Files: GLOBCOVER_L4_200901_200912_V2.3.tif
Size is 129600, 55800
Coordinate System is:
GEOGCS["WGS 84",
...
Origin = (-180.001388888888897,90.001388888888883)
...
Corner Coordinates:
Upper Left  (-180.0013889,  90.0013889) (180d 0' 5.00"W, 90d 0' 5.00"N)
Lower Left  (-180.0013889, -64.9986111) (180d 0' 5.00"W, 64d59'55.00"S)
Upper Right ( 179.9986111,  90.0013889) (179d59'55.00"E, 90d 0' 5.00"N)
Lower Right ( 179.9986111, -64.9986111) (179d59'55.00"E, 64d59'55.00"S)
Center      (  -0.0013889,  12.5013889) (  0d 0' 5.00"W, 12d30' 5.00"N)
...

How to fix this?

Option 1: You can use the -l flag of r.in.gdal to constrain the map coordinates to legal values (ref. But the resulting pixels will no longer have the original resolution. We will not do that.

Option 2: Shift the Globcover map slightly into the right position using gdal_translate: 

# coords are shifted, fix raster map
# -a_ullr Assign/override the georeferenced bounds of the output file
# use larger cache and compress result
gdal_translate --config GDAL_CACHEMAX 1200 -a_ullr -180 90 180 -65 \
     -co "COMPRESS=LZW" GLOBCOVER_L4_200901_200912_V2.3.tif GLOBCOVER_L4_200901_200912_V2.3_fixed.tif

# result:
gdalinfo GLOBCOVER_L4_200901_200912_V2.3_fixed.tif
...
Origin = (-180.000000000000000,90.000000000000000)
Pixel Size = (0.002777777777778,-0.002777777777778)
...
Corner Coordinates:
Upper Left  (-180.0000000,  90.0000000) (180d 0' 0.00"W, 90d 0' 0.00"N)
Lower Left  (-180.0000000, -65.0000000) (180d 0' 0.00"W, 65d 0' 0.00"S)
Upper Right ( 180.0000000,  90.0000000) (180d 0' 0.00"E, 90d 0' 0.00"N)
Lower Right ( 180.0000000, -65.0000000) (180d 0' 0.00"E, 65d 0' 0.00"S)
Center      (   0.0000000,  12.5000000) (  0d 0' 0.01"E, 12d30' 0.00"N)

Voilà! Now we can import the map into GRASS GIS: 

r.in.gdal input=GLOBCOVER_L4_200901_200912_V2.3_fixed.tif output=esa_globcover2009

Imagery

AVHRR

Blue Marble imagery

NASA's Blue Marble is a 500m-8 degree per-cell world wide visual image of the Earth from space, with the clouds removed.

EO-1 imagery

(Earth Observing-1)

  • "Advanced Land Imager (ALI) provides image data from ten spectral bands (band designations). The instrument operates in a pushbroom fashion, with a spatial resolution of 30 meters for the multispectral bands and 10 meters for the panchromatic band."
-- http://eros.usgs.gov/products/satellite/eo1.php
  • On-board Atmospheric Corrections

Global Land Cover Characteristics

USGS et al. generated dataset at 1km resolution. Provides global landcover characteristics.

LANDSAT imagery

Since October 1, 2008 all Landsat 7 ETM+ scenes held in the USGS EROS archive are available for download at no charge.

  • Download via the Glovis online search tool (req. Java)
  • Download via the USGS's EarthExplorer interface

Import Modules

  • r.in.gdal - Main import tool for complete multiband scenes
  • r.in.wms - Download data covering current map region via WMS server
  • r.in.onearth - WMS frontend for NASA's OnEarth Global Landsat Mosaic

Color balancing modules

See also

  • Processing tips can be found on the LANDSAT wiki page

Miscellaneous

Data sources

Import Modules

  • The r.in.gdal modules may be used to import data of many formats, including GMT netCDF
  • The r.in.bin module may be used to import raw binary files

MODIS imagery

Natural Earth imagery

Orthoimagery

Pathfinder AVHRR SST imagery

  • see the Pathfinder AVHRR SST wiki page

QuickBird imagery

SeaWiFS imagery

SPOT Vegetation imagery

SPOT Vegetation (1km) global: NDVI data sets

True Marble imagery

  • True Marble: 250m world wide visual image of the Earth from space, with the clouds removed. GeoTIFF (use the r.in.gdal module)

Climatic data

OGC WCS - Albedo example

TODO: update this example e.g. to http://demo.mapserver.org/cgi-bin/wcs?SERVICE=wcs&VERSION=1.0.0&REQUEST=GetCapabilities

GRASS imports OGC Web Coverage Service data. Example server (please suggest a better one!) 

 <WCS_GDAL>
 <ServiceURL>http://laits.gmu.edu/cgi-bin/NWGISS/NWGISS?</ServiceURL>
 <CoverageName>AUTUMN.hdf</CoverageName>
 <Timeout>90</Timeout>
 <Resample>nearest</Resample>
 </WCS_GDAL>

Save this as albedo.xml. Import into a LatLong WGS84 location: 

 r.in.gdal albedo.xml out=albedo

Unfortunately this server sends out the map shifted by 0.5 pixel. This requires a fix to the map boundary coordinates: 

 r.region albedo n=90 s=-90 w=-180 e=180

Now apply color table and look at the map: 

 r.colors albedo color=byr
 d.mon x0
 d.rast albedo

SNODAS maps

Snow Data Assimilation System data that support hydrological modeling and analysis. First download the data, and untar them (once for each month, and once for each day), and you should get pairs of “.dat” and “.Hdr” files. The data files are stored in flat 16-bit binary format, so assuming that “snowdas_in.dat” is the name of the input file, at the GRASS prompt: 

  r.in.bin -bs bytes=2 rows=3351 cols=6935 north=52.874583333332339 \
  south=24.949583333333454 east=-66.942083333334011 west=-124.733749999998366 \
  anull=-9999 input=snowdas_input.dat output=snowdas

WorldClim maps

WorldClim is a set of global climate layers (climate grids) with a spatial resolution of a square kilometer.

Population maps

Gridded Population of the World

Import with r.in.gdal, assign population color table with r.colors

Topographic maps

Soviet topographic maps

Vector data

CDC Geographic Boundary and Public Health Maps

Global Administrative Areas

  • GADM is a database of the location of the world's administrative areas (boundaries) available in shapefiles.
http://gadm.org (extracted by country here)
  • World Borders Dataset including ISO 3166-1 Country codes available in shapefiles.
http://thematicmapping.org/downloads/world_borders.php
  • Free GIS data from Mapping Hacks
http://mappinghacks.com/data/

GSHHS World Coastline

GSHHS is a high resolution shoreline dataset. It is derived from data in the public domain and licensed as GPL. The shorelines are constructed entirely from hierarchically arranged closed polygons. It is closely linked to the GMT project.

Availability

For GRASS 6 you can download 1:250,000 shoreline data from NOAA's site in Mapgen format, which can be imported with the v.in.mapgen module.

Import

OpenStreetMap

See the OpenStreetMap wiki page.

SALB

Second Administrative Level Boundaries: "The SALB dataset is a global digital dataset consisting of digital maps and codes that can be downloaded on a country by country basis."

VMap0

1:1 million vector data. Formerly known as Digital Chart of the World

Check the Wikipedia page on VMAP, see the links at the bottom of that article to shapefile versions of VMAP0 and VMAP1. Those look like the versions that were, several years ago, on a NIMA (predecessor to NGA, and successor to the Defense Mapping Agency that managed the Digital Chart of the World and VMAP project) Website. Many GRASS users may prefer the shapefiles to the original Vector Product Format data.


See also

Metadata Catalogues

Catalog Service for the Web (CSW) is an OGC standard for offering access to catalogues of geospatial information over the Internet (HTTP). CSW allow for discovering, browsing, and querying metadata about data, services, and similar resources. A list of Metadata Catalogues / CSW services from member states of the European Union can be found here:

And here: http://inspire-geoportal.ec.europa.eu/discovery/ one can search European Metadata Catalogues online.

European datasets

National datasets

Various datasets worldwide

WMS servers

River discharge data

Retrieved from "https://grasswiki.osgeo.org/w/index.php?title=Global_datasets&oldid=22081"