Time series development
This page is born out of discussions in Lausanne. A lot of people seem to be interested in having some standardized, documented format for dealing with time series in GRASS, so that we can deal with data (e.g. climate station, water gauges), link with models, etc., without having to come up with a custom solution each time. This could also lead to modules to help with interpolation to deal with missing values, etc.
There was also discussion of possibly setting up a mailing list. Do we want this?
- I would vote against just-another-mailing list Neteler 23:51, 18 September 2006 (CEST)
Ideas from MN
- each imported raster map get's automatically "registered" in an SQL table
- Table structure:
- map name
- map creator (optional)
- time stamp of import
- time stamp of map production (optional)
- time range of validity (optional)
- Table structure:
- g.list, g.rename etc. tools have a new "where" parameter do search maps in this table
- g.remove will also remove row from SQL table
- SQL commands can be used for search by time stamps
maxi's proposal
- what about a table with time series (sort of header of each time series) and another table for each time serie with a list of associated filename and related timestamp?
- time series head table structure:
- series name
- creator name
- timestamp of creation
- range of validity
- time step
- ....
- time serie associated map (table named like the related time serie):
- timestamp
- map name
- timestamp of import
- time series head table structure:
Soerens ideas
- the temporal gis extension of grass should manage:
- raster, raster3d, vecter maps and groups of raster maps
- the existing timestamps should be used for registration in a temporal database
- timestamps support for groups must be added
- the timestamp should be automatically created if a map is created
- g.remove, g.rename and g.copy will need to have access to the temporal database
- every map keeps the info in which temporal database it is registered (additional file or info in header)
- g.list should list all temporal databases, not the registered maps (t.info should do this)
Temporal region and time frames with timesteps should be supported, like the region and window functionality in grass.
temporal database
- the temporal database in which the maps are registered should provide temporal queries
- select all maps between 20 jan 2002 nad 13 feb 2004
- select all maps with a valid range of 3 months
- select all maps with datum < 13 sep 1999
- select all maps with a valid range of 10 years and a value range between [20:45]
- Postgresql is the only Open Source database which supports such queries and suitable temporal Datatypes [[1]] and [[2]]
- another way is to implement this functionality in a double linked list
timestamps
- the following timestamps should be created in the database
- a valid timestamp with startime or start- and endtime
- creation time
- modification time
- also statistical Informations about the maps (ranges) should be stored in the database
temporal management tools
- the temporal management should be done by specific commands starting with t.*
- tools for queries, register, unregister und extraction of maps from the temporal database have to be implemented, like t.info, t.register, t.unregister and t.extract ...
Example t.register:
GRASS 6.3.cvs > t.register help
Description:
Register groups, raster, raster3d and vector maps into the temporal database
Keywords:
temporal, time
Usage:
t.register [-s] [tgroup=string[,string,...]]
[trast=string[,string,...]] [trast3d=string[,string,...]]
[tvect=string[,string,...]] [group=string[,string,...]]
[rast=string[,string,...]] [rast3d=string[,string,...]]
[vect=string[,string,...]] [date=timestamp[,timestamp,...]]
[timestep=timestep[,timestep,...]]
Flags:
-s Use the timestep between new maps.
Parameters:
tgroup Temporal group database(s) in which the group(s) should be registered
trast Temporal raster database(s) in which the raster(s) should be registered
trast3d Temporal raster3d database(s) in which the raster3d(s) should be registered
tvect Temporal vector database(s) in which the vector(s) should be registered
group Group(s) to be registered in the temporal group database
rast Raster map(s) to be registered in the temporal raster database
rast3d Raster3d map(s) to be registered in the temporal raster3d database
vect Vector map(s) to be registered in the temporal vector database
date datetime, datetime1/datetime2 for map(s)
timestep timestep between the maps
example
register 3 raster maps beginning from 20 jan 2001 with a timestep of 3 month in database Landsat
- t.register -s trast=Landsat rast=LandsatJan,LandsatApr,LandsatJul date="20 jan 2001" timestep="3 mon"
Table:
1 LandsatJan 20 jan 2001 ... 2 LandsatApr 20 apr 2001 ... 3 LandsatJul 20 jul 2001 ...
OO Layer
- it should be possible to have data access with spatial-temporal functions:
- value = g4dDataObject->Get4DValue(x, y, z, timestamp) -- for volume maps
- value = g4dDataObject->Get4DValue(x, y, timestamp) -- for raster maps
- based on this functionality a 4d animation tool based on VTK should be implemented
Jachym's notes
On IRC, we discussed, that time series would be stored in database (PostgreSQL). If data format would be like YYYY-MM-DD-HH-MM-SS, the "database" could be some textfile, which would be sortable via standard 'sort' and g.* modules would not need to speek SQL - KISS, works even on a toaster.
g.list rast should print all raster maps. Raster map belonging to time serie does not stop to be raster map. New data type "times" has to be created, so that modules like g.list, g.remove ... can handle it.
Open Issues
- will 'g.list rast' show also rasters belonging to time series?
- how to deal with huge file number in a folder? (very long serie often deals with a huge number of maps)
- limitation: how many rasters can a folder contain ('fileno' in /etc/security/limits.conf).
- efficiency: huge number of files will tremendiously slow down each map listing procedures.
- associated color table: only one color table should serve one series (avoid multiple color table for each map)