https://grasswiki.osgeo.org/w/api.php?action=feedcontributions&feedformat=atom&user=%E2%9A%A0%EF%B8%8FErgetGRASS-Wiki - User contributions [en]2026-05-13T00:03:44ZUser contributionsMediaWiki 1.41.0https://grasswiki.osgeo.org/w/index.php?title=Parallelizing_Scripts&diff=16264Parallelizing Scripts2012-08-07T14:48:58Z<p>⚠️Erget: /* Python */ Added another (simple) implementation of a parallelization script</p>
<hr />
<div>=== Bourne shell script ===<br />
* '''Poor-man's multithreading''' using Bourne shell script & backgrounding. WARNING: not all GRASS modules and scripts are safe to have other things happening in the same mapset while they are running. Try at your own risk after performing a suitable safety audit. e.g. Make sure g.region is not run, externally changing the region settings.<br />
<br />
Example:<br />
<source lang="bash"><br />
### r.sun mode 1 loop ###<br />
SUNRISE=7.67<br />
SUNSET=16.33<br />
STEP=0.01<br />
# | wc -l 867<br />
<br />
if [ -z "$WORKERS" ] ; then<br />
WORKERS=4<br />
fi<br />
<br />
DAY=355<br />
for TIME in `seq $SUNRISE $STEP $SUNSET` ; do<br />
echo "time=$TIME"<br />
CMD="r.sun -s elevin=gauss day=$DAY time=$TIME \<br />
beam_rad=rad1_test.${DAY}_${TIME}_beam --quiet"<br />
<br />
# poor man's multi-threading for a multi-core CPU<br />
MODULUS=`echo "$TIME $STEP $WORKERS" | awk '{print $1 % ($2 * $3)}'`<br />
if [ "$MODULUS" = "$STEP" ] || [ "$TIME" = "$SUNSET" ] ; then<br />
# stall to let the background jobs finish<br />
$CMD<br />
sleep 2<br />
wait<br />
#while [ `pgrep -c r.sun` -ne 0 ] ; do<br />
# sleep 5<br />
#done<br />
else<br />
$CMD &<br />
fi<br />
done<br />
wait # wait for background jobs to finish to avoid race conditions<br />
</source><br />
<br />
* This approach has been used in the {{AddonCmd|r3.in.xyz}} addon script.<br />
* Another example using r.sun Mode 2 can be found on the [[r.sun]] wiki page.<br />
* See the {{cmd|i.landsat.rgb|version=65}} and {{cmd|i.oif|version=65}} examples in 6.5svn.<br />
<br />
<br />
Backgrounding code which sets environment variables with `eval` requires the use of grouping within ()s:<br />
<source lang="bash"><br />
eval `(<br />
r.univar -ge map="$RED" percentile=95 | grep '^percentile_' | sed -e 's/^/R_/' &<br />
r.univar -ge map="$GREEN" percentile=95 | grep '^percentile_' | sed -e 's/^/G_/' &<br />
r.univar -ge map="$BLUE" percentile=95 | grep '^percentile_' | sed -e 's/^/B_/'<br />
wait<br />
)`<br />
</source><br />
<br />
=== Python ===<br />
<br />
* Due to the "GIL" in Python 2.x-3.0, pure python will only run on a single core, even when multi-threaded. All multithreading schemes & modules for (pure) Python are therefore wrappers around multiple system processes, which are a lot more expensive than threads to create and destroy. Thus it is more efficient to create large high-level Python 'threads' (processes) than to bury them deep inside of a loop.<br />
<br />
Example of multiprocessing at the GRASS module level:<br />
<br />
Similar to the Bourne shell example above, but using the '''subprocess''' python module.<br />
The {{Cmd|i.oif|version=70}} script in GRASS7 is using this method.<br />
<br />
<source lang="python"><br />
bands = [1,2,3,4,5,7]<br />
<br />
# run all bands in parallel<br />
if "WORKERS" in os.environ:<br />
workers = int(os.environ["WORKERS"])<br />
else:<br />
workers = 6<br />
<br />
proc = {}<br />
pout = {}<br />
<br />
# spawn jobs in the background<br />
for band in bands:<br />
grass.debug("band %d, <%s> %% %d" % (band, image[band], band % workers))<br />
proc[band] = grass.pipe_command('r.univar', flags = 'g', map = image[band])<br />
if band % workers is 0:<br />
# wait for the ones launched so far to finish<br />
for bandp in bands[:band]:<br />
if not proc[bandp].stdout.closed:<br />
pout[bandp] = proc[bandp].communicate()[0]<br />
proc[bandp].wait()<br />
<br />
# wait for jobs to finish, collect the output<br />
for band in bands:<br />
if not proc[band].stdout.closed:<br />
pout[band] = proc[band].communicate()[0]<br />
proc[band].wait()<br />
<br />
# parse the results<br />
for band in bands:<br />
kv = grass.parse_key_val(pout[band])<br />
stddev[band] = float(kv['stddev'])<br />
</source><br />
<br />
<br />
This can also be accomplished fairly simply by using tracking the number of workers available and using grass.start_command as long as another job can be performed:<br />
<br />
<source lang="python"><br />
import os<br />
import multiprocessing as multi<br />
<br />
import grass.script as grass<br />
<br />
# Find number of workers that can be used on system. This variable could <br />
# also be set manually.<br />
workers = multi.cpu_count()<br />
# This is only a set of examples for r.slope.aspect jobs where the maps are<br />
# named serially.<br />
jobs = range(20)<br />
<br />
# Check if workers are already being used<br />
if workers is 1 and "WORKERS" in os.environ:<br />
workers = int(os.environ["WORKERS"])<br />
if workers < 1:<br />
workers = 1<br />
<br />
# Initialize process dictionary<br />
proc = {}<br />
<br />
# Loop over jobs<br />
for i in range(jobs):<br />
# Insert job into dictinoary to keep track of it<br />
proc[i] = grass.start_command('r.slope.aspect',<br />
elevation='elev_' + str(i),<br />
slope='slope_' + str(i))<br />
# If the workers are used up, wait for all of them from the last group to<br />
# finish.<br />
if i % workers is 0:<br />
for j in range(workers):<br />
proc.[i - j].wait()<br />
<br />
# Make sure all workers are finished.<br />
for i in range(jobs):<br />
if proc[i].wait() is not 0:<br />
grass.fatal(_('Problem running analysis on evel_' + str(i) + '.')<br />
</source><br />
<br />
=== GNU Parallel ===<br />
* [http://www.gnu.org/software/parallel/ GNU Parallel] is an advanced version of {{wikipedia|xargs}} which makes it easy to write parallel shell scripts.<br />
: See also the unrelated C "parallel" program which comes with the Linux "moreutils" package. It is tighter but less featureful than GNU Parallel.<br />
<br />
<source lang="bash"><br />
### r.sun mode 1 loop ###<br />
SUNRISE=7.67<br />
SUNSET=16.33<br />
STEP=0.01<br />
# | wc -l 867<br />
<br />
DAY=355<br />
<br />
seq $SUNRISE $STEP $SUNSET | parallel -j+0 r.sun -s elevin=gauss day=$DAY \<br />
time={} beam_rad=rad1_test.${DAY}_{}_beam --quiet<br />
</source><br />
<br />
GNU Parallel can also distribute work to other computers, see the video on how:<br />
: http://www.youtube.com/watch?v=LlXDtd_pRaY<br />
<br />
=== xargs ===<br />
* {{wikipedia|xargs}} can be told to limit itself to a certain number of processes at once. The r.sun example is almost exactly as with GNU Parallel, except for `-P $CORES -n 1` instead of `-j+0`.<br />
<br />
For example, convert a large number of Raster3D maps into 2D rasters:<br />
<source lang="bash"><br />
NUM_CORES=6<br />
g.mlist rast3d | xargs -P $NUM_CORES -n 1 -I{} \<br />
r3.to.rast -r in={} out={} --quiet<br />
</source><br />
<br />
For another example, here we spit apart a PDF and convert each page to a PNG image:<br />
<br />
<source lang="bash"><br />
pdftk pdfmovie.pdf burst<br />
NUM_CORES=6<br />
ls -1 pg_*.pdf | xargs -P $NUM_CORES -n 1 -I{} \<br />
sh -c "pdftoppm {} | pnmcut -width 1280 -height 1024 -left 0 -top 0 | \<br />
pnmtopng > \`basename {} .pdf\`.png"<br />
</source><br />
<br />
[[Category: Development]]<br />
[[Category: Parallelization]]</div>⚠️Ergethttps://grasswiki.osgeo.org/w/index.php?title=Vector_aggregate_values&diff=15352Vector aggregate values2012-04-11T16:18:28Z<p>⚠️Erget: /* Method 2 */</p>
<hr />
<div>== Introduction ==<br />
<br />
GIS users often wish to aggregate the values of one polygon layer into another. Although this is not integrated as a standard tool in GRASS, it is definitely possible. This article introduces two methods to do just that: A simple, quick operation will be described that can take place entirely in GRASS GIS and produces raster maps or, if desired, polygons with aggregated values from two maps and a second, more complex method that uses external editors and creates text files as outputs.<br />
<br />
=== Method 1 ===<br />
<br />
''written by Daniel Lee''<br />
<br />
In this example we assume that we have a vector map of catchment areas and a raster map of precipitation. We want to find out what the sum total of precipitation per catchment area is. This method would also work if the precipitation data were also stored as vector data, but the vector precipitation data would have to be turned into a raster map, just as the catchment areas are converted to rasters in the following steps. Remember to set the region, etc. in GRASS properly.<br />
<br />
''' Step 1: Convert catchment area polygons to rasters '''<br />
<br />
The polygons are converted to rasters using v.to.rast. The primary key for the polygons should be used as the values for the new rasters. If the polygons were imported from a shapefile, for example, the field FID could be used.<br />
<br />
v.to.rast input=CatchmentAreas output=CatchmentAreas_Raster column=FID<br />
<br />
''' Step 2: Convert precipitation raster to integer values '''<br />
<br />
The tool that we will be using later, r.statistics, requires integer values. Because the precipitation raster is composed of floating point values, we have to convert it to an integer map first. In order to avoid rounding areas in the final results, we also multiply the raster by 100. This way we can divide it by 100 later in order to the original float values back.<br />
If the raster that you want to aggregate into your vectors is already made of integer values or if rounding errors aren't a problem, this step can be changed or left out.<br />
<br />
r.mapcalc 'Precipitation_Integer = round (Precipitation * 100)<br />
<br />
''' Step 3: Aggregate the precipitation values by summing them in the catchment areas they fall in '''<br />
<br />
Now we have two rasters that can be combined. In order to find the sum of the pixels of the precipitation map we use r.statistics.<br />
<br />
r.statistics base=CatchmentAreas_Raster cover=Precipitation_Integer method=sum output=Precipitation_by_CatchmentArea<br />
<br />
''' Step 4: Extract sums from the new map '''<br />
<br />
Now the precipitation sums are aggregated by catchment area, but only as attributes and not as categories in the resulting raster. We now extract the aggregated sums, overwriting the previous raster, by using the "@" function in r.mapcalc. Additionally, the map is divided by 100 to scale the values back, reversing the change we made when we turned the original precipitation map into an integer map.<br />
Don't forget to not divide by 100 if you didn't multiply by 100 in step 2.<br />
<br />
r.mapcalc 'Precipitation_by_CatchmentArea = @Precipitation_by_CatchmentArea / 100'<br />
<br />
''' Step 5: Convert the aggregated precipitation raster to a vector map. '''<br />
<br />
This procedure should already be familiar for those who work with vector maps. If vectors are not needed, it can be skipped.<br />
<br />
r.to.vect input=Precipitation_by_CatchmentArea output=Precipitation_by_CatchmentArea_Vectors feature=area<br />
<br />
In an optional last step one can combine the new vectors, which will probably not be exactly the same as the old vectors, with the old vector attribute table, either with v.db.join or v.distance using e.g. the centroids as spatial join criterium.<br />
<br />
=== Method 2 ===<br />
<br />
''written by Michael O'Donovan''<br />
<br />
GIS users accustomed to proprietary vector-based products (MapInfo, MapMaker, Atlas and some other ESRI) are often disappointed when the try migrating to OpenSource software. Many of the key functions that make vector-based systems so useful are unavailable in OpenSource software like QGIS, Thuban and GRASS. One such function is the ability to aggregate values from one polygon layer to another.<br />
<br />
A typical example would be one of aggregating the population of census tracts (or municipalities) to derive the population of a suburb (or province). The aggregation is simply one of adding the tracts together only when the outer edge of some combination of polygons coincide. This happens when tracts are defined in such a way that they coincide with suburb perimeters. However when the boundaries do not coincide the situation usually calls for a vector-based system that will allocate populations according to the location of the tract centroid or the proportion of a tract falling into a particular suburb. The absence of this aggregation feature discourages potential users from adopting OpenSource software like GRASS. The section that follows shows how, with a little bit of preparation, aggregations can be easily performed by GRASS raster functions. It follows a hypothetical example inspired by the proposed redrawing of South Africa's nine provinces which currently dissect many of the 280+ municipalities. It shows how municipal populations can be re-combined to derive the provincial population by converting the vector features to rasters.<br />
<br />
For simplicity it is assumed that the user has two shape files "Municipalities" and "Province". The "Municipalities" file is linked to a file "Municipalities.dbf" which contains numerous variables including "pop" (its population) and "SHP_FID" (a unique number which identifies each polygon). Both files can be easily imported into GRASS (using v.in.ogr) and displayed along with their variables. Before the vector files can be converted to rasters an unusual preparatory step is required. This entails converting the variables of interest from a value per vector to a value per raster cell. To make this conversion it is necessary to place into the appropriate .dbf file a count of the number of raster cells that will represent that vector. In this example this will enable the user to represent the population not as a number per municipality but as a number per cell. All subsequent aggregations need to be based on these cell values and thus on the cell counts.<br />
<br />
The first step in preparation is to calculate the number of cells in each municipality. To do this convert the municipality vector into a raster file. Remember that, unlike vectors which can be linked to a large number of other variables, raster cells take on a single value only. For this step this value must be a number that uniquely identifies each municipality. If the municipality vector was obtained from an ESRI shapefile the field "SHP_FID" doesthe job. From the command line type:<br />
<br />
v.to.rast input=Municipalities output=municipalities use=attr col=SHP_FID<br />
<br />
This creates a raster file called "municipalities". As vectors and raster are of different types there is little prospect of their names being confused. (Nevertheless vectors are indicated by a starting capital while rasters are kept as lowercase words.) the new raster can be seen by typing... <br />
<br />
d.rast municipalities<br />
<br />
If is looks funny change the color scheme using something like...<br />
<br />
r.colors map=municipalities color=random<br />
and d.rast it again.<br />
<br />
The next step requires extracting a count of the number of raster cells in each municipality. Once again this is a simple procedure. To save do this and save the output to a file called "munic_count" type...<br />
<br />
r.stats input=municipalities output=munic_count -c<br />
<br />
This will produce a text file with the SHP_FID and a cell count seperated by a space. The final preparatory step is to place the count corresponding to each municipality into the Municipalities.dbf file and to calculate the variables of interest as an amount per cell.<br />
<br />
<hr><br />
This is how you would link the files in OpenOffice if you have not yet set up your data sources and do not know how to use VLOOKUP etc.<br />
<br />
# Open the file "munic_count" - it will default to a text page containing the SHP_FID, an open space and the cell count.<br />
# Convert the open spaces to tabs using "Edit", "Find and Replace". Place a space in the "search for" box and \t in the "replace with" boxes. Activate "Regular expressions" and click on "Replace all".<br />
# Select all of the data (Ctrl A) and copy it to a clipboard (Ctrl C)<br />
<br />
You now have all the data ordered by SHP_FID ready to be pasted into the Municpalities.dbf file. So open the Municipalities.dbf file - it will default to a spreadsheet.<br />
<br />
# Make sure that the spreadsheet data is in ascending ordered of SHP_FID. (Select all data other than the headers then go to "Data", "Sort" and select the appropriate column).<br />
# Click on to cell at the top of the first unused column and click on "Edit", "Paste special" and "unformatted text". The cells values and SHP_FID data should now be placed in the corresponding rows. Check that this has been done.<br />
<br />
=== Variables of interest ===<br />
<br />
If you are interested in, say, aggregating populations then calculate the population per cell for every municipality. (If your data starts in row 1, population is in column C and the cell count in column K type "=C1/K1". Copy and then paste the formula into all other rows.) Give the new column an appropraite name, say, "cell_value" and save the file under its original name (i.e. as a .dbf). You may well return at a latter stage and calculate new variables of interest using the cell counts. <br />
<br />
<hr><br />
<br />
You can now proceed with the aggregations.<br />
<br />
First convert the Municipalities vector into a raster (again). This time use the cell_value as the attribute of interest. At the command prompt type...<br />
<br />
v.to.rast input=Municipalities output=municipalities use=attr col=cell_value<br />
<br />
Note that I gave the output raster the same name as earlier. This will overwrite the original raster thereby minimizing clutter.<br />
<br />
Now also convert the Provinces vector to a raster file <br />
<br />
v.to.rast input=Provinces output=provinces use=attr col=SHP_FID<br />
<br />
The relationship between the two raster files can now be explored using a number of statistics commands most of which are found under r.statistics. Perhapsthe most important statistics are r.sum and r.average. <br />
<br />
The command<br />
r.sum municpalities <br />
<br />
will give the total of all the cell values in the municipalities raster. This equals the total population of the region. However what is actually required in the sum of the population for each province. To get this we use r.statistics. Unfortunately this script does not work for floating point rasters (which is what you are most likely to have). To overcome this problem create a new raster reflecting the integer value of each cell. To create a file useable for r.statistics use..<br />
<br />
r.mapcalc<br />
(then enter line-by-line)<br />
int_municpal=round(municipalities)<br />
end<br />
<br />
If rounding of the cell values to an integer results in a notable decline in accuracy then create a new raster file in which the values are multiplied by ten or a hundred and only then rounded. In latter analysis recall that the units have been dramatically (but precisely) inflated. To do this try "int_munic=round(100*municipalities)".<br />
<br />
To finally aggregate the municipal population to provinces type... <br />
<br />
r.statistics base=provinces cover=int_municipal method=sum output=prov_pop<br />
<br />
This creates a new raster "prov_pop" which can be explored using<br />
<br />
d.what.vect<br />
(click on each region of interest)<br />
<br />
You will see that the values for each province now equals the sum of all the cells that fall into that province. If you want to list the values so that you can put them into Provinces.dbf (i.e. the vector) use..<br />
<br />
r.stats input=RSA_STATS -l <br />
<br />
The output can then be placed into the corresponding data file using a method similar to that above. <br />
<br />
r.statistics comes with a range of other functions other than "sum". These substantially improve the functionality of the method. Statistics include distribution, average, mode, median, average deviation, standard deviation, variance, skewness, kurtosis, minimum and maximum. For assistance type "man r.statistics" from the command prompt. Additional functionality is gained by r.mapcalc's ability to perform functions on multiple raster files. This, in itself, makes the move to raster-based mapping most promising.<br />
<br />
<br />
Sometimes the above procedure results in a notable loss in accuracy. There are two main sources for this: <br />
<br />
The rounding of cell values (to use r.statistics) may be problematic for areas with very small values. If this is the case consider inflating cell values by one of several orders of magnitude. <br />
<br />
When the region was set up a resolution (E-W and N-S) was specified. If this resolution is not fine enough there may be some loss of accuracy through aggregation. Consider increasing the resolution.<br />
<br />
[[Category:FAQ]]<br />
[[Category:Vector]]</div>⚠️Ergethttps://grasswiki.osgeo.org/w/index.php?title=Vector_aggregate_values&diff=14798Vector aggregate values2012-01-25T12:06:25Z<p>⚠️Erget: /* Method 1 */</p>
<hr />
<div>== Introduction ==<br />
<br />
GIS users often wish to aggregate the values of one polygon layer into another. Although this is not integrated as a standard tool in GRASS, it is definitely possible. This article introduces two methods to do just that: A simple, quick operation will be described that can take place entirely in GRASS GIS and produces raster maps or, if desired, polygons with aggregated values from two maps and a second, more complex method that uses external editors and creates text files as outputs.<br />
<br />
=== Method 1 ===<br />
<br />
''written by Daniel Lee''<br />
<br />
In this example we assume that we have a vector map of catchment areas and a raster map of precipitation. We want to find out what the sum total of precipitation per catchment area is. This method would also work if the precipitation data were also stored as vector data, but the vector precipitation data would have to be turned into a raster map, just as the catchment areas are converted to rasters in the following steps. Remember to set the region, etc. in GRASS properly.<br />
<br />
''' Step 1: Convert catchment area polygons to rasters '''<br />
<br />
The polygons are converted to rasters using v.to.rast. The primary key for the polygons should be used as the values for the new rasters. If the polygons were imported from a shapefile, for example, the field FID could be used.<br />
<br />
v.to.rast input=CatchmentAreas output=CatchmentAreas_Raster column=FID<br />
<br />
''' Step 2: Convert precipitation raster to integer values '''<br />
<br />
The tool that we will be using later, r.statistics, requires integer values. Because the precipitation raster is composed of floating point values, we have to convert it to an integer map first. In order to avoid rounding areas in the final results, we also multiply the raster by 100. This way we can divide it by 100 later in order to the original float values back.<br />
If the raster that you want to aggregate into your vectors is already made of integer values or if rounding errors aren't a problem, this step can be changed or left out.<br />
<br />
r.mapcalc 'Precipitation_Integer = round (Precipitation * 100)<br />
<br />
''' Step 3: Aggregate the precipitation values by summing them in the catchment areas they fall in '''<br />
<br />
Now we have two rasters that can be combined. In order to find the sum of the pixels of the precipitation map we use r.statistics.<br />
<br />
r.statistics base=CatchmentAreas_Raster cover=Precipitation_Integer method=sum output=Precipitation_by_CatchmentArea<br />
<br />
''' Step 4: Extract sums from the new map '''<br />
<br />
Now the precipitation sums are aggregated by catchment area, but only as attributes and not as categories in the resulting raster. We now extract the aggregated sums, overwriting the previous raster, by using the "@" function in r.mapcalc. Additionally, the map is divided by 100 to scale the values back, reversing the change we made when we turned the original precipitation map into an integer map.<br />
Don't forget to not divide by 100 if you didn't multiply by 100 in step 2.<br />
<br />
r.mapcalc 'Precipitation_by_CatchmentArea = @Precipitation_by_CatchmentArea / 100'<br />
<br />
''' Step 5: Convert the aggregated precipitation raster to a vector map. '''<br />
<br />
This procedure should already be familiar for those who work with vector maps. If vectors are not needed, it can be skipped.<br />
<br />
r.to.vect input=Precipitation_by_CatchmentArea output=Precipitation_by_CatchmentArea_Vectors feature=area<br />
<br />
In an optional last step one can combine the new vectors, which will probably not be exactly the same as the old vectors, with the old vector attribute table, either with v.db.join or v.distance using e.g. the centroids as spatial join criterium.<br />
<br />
=== Method 2 ===<br />
<br />
''written by Michael O'Donovan''<br />
<br />
GIS users accustomed to proprietary vector-based products (MapInfo, MapMaker, Atlas and some other ESRI) are often disappointed when the try migrating to OpenSource software. Many of the key functions that make vector-based systems so useful are unavailable in OpenSource software like QGIS, Thuban and GRASS. One such function is the ability to aggregate values from one polygon layer to another.<br />
<br />
A typical example would be one of aggregating the population of census tracts (or municipalities) to derive the population of a suburb (or province). The aggregation is simply one of adding the tracts together only when the outer edge of some combination of polygons coincide. This happens when tracts are defined in such a way that they coincide with suburb perimeters. However when the boundaries do not coincide the situation usually calls for a vector-based system that will allocate populations according to the location of the tract centroid or the proportion of a tract falling into a particular suburb. The absence of this aggregation feature discourages potential users from adopting OpenSource software like GRASS. The section that follows shows how, with a little bit of preparation, aggregations can be easily performed by GRASS raster functions. It follows a hypothetical example inspired by the proposed redrawing of South Africa's nine provinces which currently dissect many of the 280+ municipalities. It shows how municipal populations can be re-combined to derive the provincial population by converting the vector features to rasters.<br />
<br />
For simplicity it is assumed that the user has two shape files "Municipalities" and "Province". The "Municipalities" file is linked to a file "Municipalities.dbf" which contains numerous variables including "pop" (its population) and "SHP_FID" (a unique number which identifies each polygon). Both files can be easily imported into GRASS (using v.in.ogr) and displayed along with their variables. Before the vector files can be converted to rasters an unusual preparatory step is required. This entails converting the variables of interest from a value per vector to a value per raster cell. To make this conversion it is necessary to place into the appropriate .dbf file a count of the number of raster cells that will represent that vector. In this example this will enable the user to represent the population not as a number per municipality but as a number per cell. All subsequent aggregations need to be based on these cell values and thus on the cell counts.<br />
<br />
The first step in preparation is to calculate the number of cells in each municipality. To do this convert the municipality vector into a raster file. Remember that, unlike vectors which can be linked to a large number of other variables, raster cells take on a single value only. For this step this value must be a number that uniquely identifies each municipality. If the municipality vector was obtained from an ESRI shapefile the field "SHP_FID" doesthe job. From the command line type:<br />
<br />
v.to.rast input=Municipalities output=municipalities use=attr col=SHP_FID<br />
<br />
This creates a raster file called "municipalities". As vectors and raster are of different types there is little prospect of their names being confused. (Nevertheless vectors are indicated by a starting capital while rasters are kept as lowercase words.) the new raster can be seen by typing... <br />
<br />
d.rast municipalities.<br />
<br />
If is looks funny change the color scheme using something like...<br />
<br />
r.colors map=municipalities color=random<br />
and d.rast it again.<br />
<br />
The next step requires extracting a count of the number of raster cells in each municipality. Once again this is a simple procedure. To save do this and save the output to a file called "munic_count" type...<br />
<br />
r.stats input=municipalities output=munic_count -c<br />
<br />
This will produce a text file with the SHP_FID and a cell count seperated by a space. The final preparatory step is to place the count corresponding to each municipality into the Municipalities.dbf file and to calculate the variables of interest as an amount per cell.<br />
<br />
<hr><br />
This is how you would link the files in OpenOffice if you have not yet set up your data sources and do not know how to use VLOOKUP etc.<br />
<br />
# Open the file "munic_count" - it will default to a text page containing the SHP_FID, an open space and the cell count.<br />
# Convert the open spaces to tabs using "Edit", "Find and Replace". Place a space in the "search for" box and \t in the "replace with" boxes. Activate "Regular expressions" and click on "Replace all".<br />
# Select all of the data (Ctrl A) and copy it to a clipboard (Ctrl C)<br />
<br />
You now have all the data ordered by SHP_FID ready to be pasted into the Municpalities.dbf file. So open the Municipalities.dbf file - it will default to a spreadsheet.<br />
<br />
# Make sure that the spreadsheet data is in ascending ordered of SHP_FID. (Select all data other than the headers then go to "Data", "Sort" and select the appropriate column).<br />
# Click on to cell at the top of the first unused column and click on "Edit", "Paste special" and "unformatted text". The cells values and SHP_FID data should now be placed in the corresponding rows. Check that this has been done.<br />
<br />
=== Variables of interest ===<br />
<br />
If you are interested in, say, aggregating populations then calculate the population per cell for every municipality. (If your data starts in row 1, population is in column C and the cell count in column K type "=C1/K1". Copy and then paste the formula into all other rows.) Give the new column an appropraite name, say, "cell_value" and save the file under its original name (i.e. as a .dbf). You may well return at a latter stage and calculate new variables of interest using the cell counts. <br />
<br />
<hr><br />
<br />
You can now proceed with the aggregations.<br />
<br />
First convert the Municipalities vector into a raster (again). This time use the cell_value as the attribute of interest. At the command prompt type...<br />
<br />
v.to.rast input=Municipalities output=municipalities use=attr col=cell_value<br />
<br />
Note that I gave the output raster the same name as earlier. This will overwrite the original raster thereby minimizing clutter.<br />
<br />
Now also convert the Provinces vector to a raster file <br />
<br />
v.to.rast input=Provinces output=provinces use=attr col=SHP_FID<br />
<br />
The relationship between the two raster files can now be explored using a number of statistics commands most of which are found under r.statistics. Perhapsthe most important statistics are r.sum and r.average. <br />
<br />
The command<br />
r.sum municpalities <br />
<br />
will give the total of all the cell values in the municipalities raster. This equals the total population of the region. However what is actually required in the sum of the population for each province. To get this we use r.statistics. Unfortunately this script does not work for floating point rasters (which is what you are most likely to have). To overcome this problem create a new raster reflecting the integer value of each cell. To create a file useable for r.statistics use..<br />
<br />
r.mapcalc<br />
(then enter line-by-line)<br />
int_municpal=round(municipalities)<br />
end<br />
<br />
If rounding of the cell values to an integer results in a notable decline in accuracy then create a new raster file in which the values are multiplied by ten or a hundred and only then rounded. In latter analysis recall that the units have been dramatically (but precisely) inflated. To do this try "int_munic=round(100*municipalities)".<br />
<br />
To finally aggregate the municipal population to provinces type... <br />
<br />
r.statistics base=provinces cover=int_municipal method=sum output=prov_pop<br />
<br />
This creates a new raster "prov_pop" which can be explored using<br />
<br />
d.what.vect<br />
(click on each region of interest)<br />
<br />
You will see that the values for each province now equals the sum of all the cells that fall into that province. If you want to list the values so that you can put them into Provinces.dbf (i.e. the vector) use..<br />
<br />
r.stats input=RSA_STATS -l <br />
<br />
The output can then be placed into the corresponding data file using a method similar to that above. <br />
<br />
r.statistics comes with a range of other functions other than "sum". These substantially improve the functionality of the method. Statistics include distribution, average, mode, median, average deviation, standard deviation, variance, skewness, kurtosis, minimum and maximum. For assistance type "man r.statistics" from the command prompt. Additional functionality is gained by r.mapcalc's ability to perform functions on multiple raster files. This, in itself, makes the move to raster-based mapping most promising.<br />
<br />
<br />
Sometimes the above procedure results in a notable loss in accuracy. There are two main sources for this: <br />
<br />
The rounding of cell values (to use r.statistics) may be problematic for areas with very small values. If this is the case consider inflating cell values by one of several orders of magnitude. <br />
<br />
When the region was set up a resolution (E-W and N-S) was specified. If this resolution is not fine enough there may be some loss of accuracy through aggregation. Consider increasing the resolution.<br />
<br />
[[Category:FAQ]]<br />
[[Category:Vector]]</div>⚠️Ergethttps://grasswiki.osgeo.org/w/index.php?title=Import_XYZ&diff=14456Import XYZ2011-12-02T17:12:30Z<p>⚠️Erget: </p>
<hr />
<div>== Import Gridded XYZ DEM data ==<br />
<br />
There are a couple of ways to import regular DEM (elevation) data in gridded XYZ ASCII format:<br />
* use {{cmd|v.in.ascii}}, then {{cmd|v.to.rast}}<br />
* use {{cmd|r.in.xyz}} along with {{cmd|g.region}}<br />
<br />
By ''gridded'' it is meant that the data covers exactly one x,y,z data point per raster cell. For ''sparse'' data points or grids rotated or of a different resolution compared to the current raster region you should use one of the vector interpolation (e.g. {{cmd|v.surf.rst}}) or raster resampling (e.g. {{cmd|r.resamp.interp}}) modules to fill in any gaps after importing; and adjusting the region by half a cell outwards is not needed.<br />
For rotated grids, if possible, it is better to import them in their native projection then use {{cmd|r.proj}} to pull them into the target location. For grids of different resolution to the current raster region settings, if possible, it is better to import them in their native resolution.<br />
<br />
=== Import with r.in.xyz ===<br />
<br />
'''Example:''' We want to import a XYZ ASCII file "VTL2733.XYZ". First we look at the structure of the file:<br />
<br />
head -n 4 VTL2733.XYZ<br />
617000.0 148000.0 157.92<br />
617025.0 148000.0 157.82<br />
617050.0 148000.0 157.70<br />
617075.0 148000.0 157.60<br />
...<br />
<br />
It indicates a 25m raster cell size. To import, we run a few commands:<br />
<br />
* First scan the map extent in human readable way (reduce white space on the fly, read from standard-input):<br />
cat VTL2733.XYZ| r.in.xyz -s in=- fs=space out=test<br />
Range: min max<br />
x: 617000.000000 619250.000000<br />
y: 148000.000000 151000.000000<br />
z: 149.160000 162.150000<br />
<br />
* set region, get from scanning the file again, now in machine-readable form<br />
# (eval and -g flag):<br />
eval `cat VTL2733.XYZ | r.in.xyz -s -g in=- fs=space out=test`<br />
g.region n=$n s=$s w=$w e=$e res=25 -p<br />
<br />
* enlarge region by half raster cell resolution to store values in cell-center:<br />
g.region n=n+12.5 s=s-12.5 w=w-12.5 e=e+12.5 -p<br />
<br />
* verify that the number of rows in the ASCII file corresponds to the number of cells in enlarged region<br />
wc -l VTL2733.XYZ<br />
<br />
* finally import as raster map:<br />
cat VTL2733.XYZ | r.in.xyz in=- fs=space out=vtl2733<br />
<br />
* and confirm that it all went cleanly, indicated by the absence of NULL cells:<br />
r.univar vtl2733<br />
<br />
Now you can analyse the imported elevation map or export into a reasonable raster format (e.g., GeoTIFF) with {{cmd|r.out.gdal}}.<br />
<br />
Inspecting the output of {{cmd|r.univar}} with r.in.xyz's <tt>method=n</tt> maps can be very useful for troubleshooting.<br />
<br />
=== Import multiple files in one step into a single DEM ===<br />
<br />
Just amend above procedure to use wildcards.<br />
Change in above example all occurencies of<br />
<br />
cat VTL2733.XYZ | ...<br />
to<br />
cat *.XYZ | ...<br />
<br />
and use a more reasonable output name of course. That's all to import even thousands of files (tiled DEM) easily.<br />
<br />
=== Hint if there are multiple white spaces as separator ===<br />
<br />
Sometimes one comes across ASCII tables formatted in unfortunate ways, e.g. like this:<br />
617000.0 148000.0 157.92<br />
617025.0 148000.0 157.82<br />
617050.0 148000.0 157.70<br />
617075.0 148000.0 157.60<br />
...<br />
<br />
Above mentioned GRASS commands refuse to import directly this file since they require single (and not replicated) field separators.<br />
<br />
'''Solution:''' On Unix-based systems, you can get rid of this on the fly and import the XYZ ASCII file to a raster map combining some text tools and GRASS commands. Just amend above procedure to use "tr".<br />
<br />
Change<br />
cat VTL2733.XYZ | r.in.xyz ...<br />
to<br />
cat VTL2733.XYZ | tr -s ' ' ' ' | r.in.xyz ...<br />
<br />
Then proceed as above.<br />
<br />
== See also ==<br />
<br />
* [http://tldp.org/LDP/abs/html/textproc.html Text Processing Commands]<br />
* [http://www.gnu.org/software/coreutils/manual/html_node/index.html GNU Coreutils]<br />
* The [[LIDAR]] wiki page (discusses the import of ''sparse'' XYZ datasets)<br />
* The [http://trac.osgeo.org/grass/browser/grass-addons/raster/r.in.xyz.auto r.in.xyz.auto] add-on script<br />
* [https://trac.osgeo.org/grass/query?status=new&status=assigned&status=reopened&keywords=%7Er.in.xyz Open support tickets concerning r.in.xyz]<br />
<br />
[[Category:Documentation]]<br />
[[Category:FAQ]]</div>⚠️Ergethttps://grasswiki.osgeo.org/w/index.php?title=Vector_aggregate_values&diff=14394Vector aggregate values2011-11-15T17:52:21Z<p>⚠️Erget: </p>
<hr />
<div>GIS users often wish toto aggregate the values of one polygon layer into another. Although this is not integrated as a standard tool in GRASS, it is definitely possible. This article introduces two methods to do just that: A simple, quick operation will be described that can take place entirely in GRASS GIS and produces rasters or, if desired, polygons with aggregated values from two maps and a second, more complex method that uses external editors and creates text files as outputs.<br />
<br />
'''Method 1:''' (written by Daniel Lee)<br />
<br />
In this example we assume that we have a vector map of catchment areas and a raster map of precipitation. We want to find out what the sum total of precipitation per catchment area is. This method would also work if the precipitation data were also stored as vector data, but the vector precipitation data would have to be turned into a raster map, just as the catchment areas are converted to rasters in the following steps. Remember to set the region, etc. in GRASS properly.<br />
<br />
''' Step 1: Convert catchment area polygons to rasters '''<br />
<br />
The polygons are converted to rasters using v.to.rast. The primary key for the polygons should be used as the values for the new rasters. If the polygons were imported from a shapefile, for example, the field FID could be used.<br />
<br />
v.to.rast input=CatchmentAreas output=CatchmentAreas_Raster column=FID<br />
<br />
''' Step 2: Convert precipitation raster to integer values '''<br />
<br />
The tool that we will be using later, r.statistics, requires integer values. Because the precipitation raster is composed of floating point values, we have to convert it to an integer map first. In order to avoid rounding areas in the final results, we also multiply the raster by 100. This way we can divide it by 100 later in order to the original float values back.<br />
If the raster that you want to aggregate into your vectors is already made of integer values or if rounding errors aren't a problem, this step can be changed or left out.<br />
<br />
r.mapcalc 'Precipitation_Integer = round (Precipitation * 100)<br />
<br />
''' Step 3: Aggregate the precipitation values by summing them in the catchment areas they fall in '''<br />
<br />
Now we have two rasters that can be combined. In order to find the sum of the pixels of the precipitation map we use r.statistics.<br />
<br />
r.statistics base=CatchmentAreas_Raster cover=Precipitation_Integer method=sum output=Precipitation_by_CatchmentArea<br />
<br />
''' Step 4: Extract sums from the new map '''<br />
<br />
Now the precipitation sums are aggregated by catchment area, but only as attributes and not as categories in the resulting raster. We now extract the aggregated sums, overwriting the previous raster, by using the "@" function in r.mapcalc. Additionally, the map is divided by 100 to scale the values back, reversing the change we made when we turned the original precipitation map into an integer map.<br />
Don't forget to not divide by 100 if you didn't multiply by 100 in step 2.<br />
<br />
r.mapcalc 'Precipitation_by_CatchmentArea = @Precipitation_by_CatchmentArea / 100'<br />
<br />
''' Step 5: Convert the aggregated precipitation raster to a vector map. '''<br />
<br />
This procedure should already be familiar for those who work with vector maps. If vectors are not needed, it can be skipped.<br />
<br />
r.to.vect input=Precipitation_by_CatchmentArea output=Precipitation_by_CatchmentArea_Vectors feature=area<br />
<br />
<br />
'''Method 2:''' (written by Michael O'Donovan)<br />
<br />
GIS users accustomed to proprietary vector-based products (MapInfo, MapMaker, Atlas and some other ESRI) are often disappointed when the try migrating to OpenSource software. Many of the key functions that make vector-based systems so useful are unavailable in OpenSource software like QGIS, Thuban and GRASS. One such function is the ability to aggregate values from one polygon layer to another.<br />
<br />
A typical example would be one of aggregating the population of census tracts (or municipalities) to derive the population of a suburb (or province). The aggregation is simply one of adding the tracts together only when the outer edge of some combination of polygons coincide. This happens when tracts are defined in such a way that they coincide with suburb perimeters. However when the boundaries do not coincide the situation usually calls for a vector-based system that will allocate populations according to the location of the tract centroid or the proportion of a tract falling into a particular suburb. The absence of this aggregation feature discourages potential users from adopting OpenSource software like GRASS. The section that follows shows how, with a little bit of preparation, aggregations can be easily performed by GRASS raster functions. It follows a hypothetical example inspired by the proposed redrawing of South Africa's nine provinces which currently dissect many of the 280+ municipalities. It shows how municipal populations can be re-combined to derive the provincial population by converting the vector features to rasters.<br />
<br />
For simplicity it is assumed that the user has two shape files "Municipalities" and "Province". The "Municipalities" file is linked to a file "Municipalities.dbf" which contains numerous variables including "pop" (its population) and "SHP_FID" (a unique number which identifies each polygon). Both files can be easily imported into GRASS (using v.in.ogr) and displayed along with their variables. Before the vector files can be converted to rasters an unusual preparatory step is required. This entails converting the variables of interest from a value per vector to a value per raster cell. To make this conversion it is necessary to place into the appropriate .dbf file a count of the number of raster cells that will represent that vector. In this example this will enable the user to represent the population not as a number per municipality but as a number per cell. All subsequent aggregations need to be based on these cell values and thus on the cell counts.<br />
<br />
The first step in preparation is to calculate the number of cells in each municipality. To do this convert the municipality vector into a raster file. Remember that, unlike vectors which can be linked to a large number of other variables, raster cells take on a single value only. For this step this value must be a number that uniquely identifies each municipality. If the municipality vector was obtained from an ESRI shapefile the field "SHP_FID" doesthe job. From the command line type:<br />
<br />
v.to.rast input=Municipalities output=municipalities use=attr col=SHP_FID<br />
<br />
This creates a raster file called "municipalities". As vectors and raster are of different types there is little prospect of their names being confused. (Nevertheless vectors are indicated by a starting capital while rasters are kept as lowercase words.) the new raster can be seen by typing... <br />
<br />
d.rast municipalities.<br />
<br />
If is looks funny change the color scheme using something like...<br />
<br />
r.colors map=municipalities color=random<br />
and d.rast it again.<br />
<br />
The next step requires extracting a count of the number of raster cells in each municipality. Once again this is a simple procedure. To save do this and save the output to a file called "munic_count" type...<br />
<br />
r.stats input=municipalities output=munic_count -c<br />
<br />
This will produce a text file with the SHP_FID and a cell count seperated by a space. The final preparatory step is to place the count corresponding to each municipality into the Municipalities.dbf file and to calculate the variables of interest as an amount per cell.<br />
<br />
<hr><br />
This is how you would link the files in OpenOffice if you have not yet set up your data sources and do not know how to use VLOOKUP etc.<br />
<br />
# Open the file "munic_count" - it will default to a text page containing the SHP_FID, an open space and the cell count.<br />
# Convert the open spaces to tabs using "Edit", "Find and Replace". Place a space in the "search for" box and \t in the "replace with" boxes. Activate "Regular expressions" and click on "Replace all".<br />
# Select all of the data (Ctrl A) and copy it to a clipboard (Ctrl C)<br />
<br />
You now have all the data ordered by SHP_FID ready to be pasted into the Municpalities.dbf file. So open the Municipalities.dbf file - it will default to a spreadsheet.<br />
<br />
# Make sure that the spreadsheet data is in ascending ordered of SHP_FID. (Select all data other than the headers then go to "Data", "Sort" and select the appropriate column).<br />
# Click on to cell at the top of the first unused column and click on "Edit", "Paste special" and "unformatted text". The cells values and SHP_FID data should now be placed in the corresponding rows. Check that this has been done.<br />
<br />
Variables of interest.<br />
<br />
If you are interested in, say, aggregating populations then calculate the population per cell for every municipality. (If your data starts in row 1, population is in column C and the cell count in column K type "=C1/K1". Copy and then paste the formula into all other rows.) Give the new column an appropraite name, say, "cell_value" and save the file under its original name (i.e. as a .dbf). You may well return at a latter stage and calculate new variables of interest using the cell counts. <br />
<br />
<hr><br />
<br />
You can now proceed with the aggregations.<br />
<br />
First convert the Municipalities vector into a raster (again). This time use the cell_value as the attribute of interest. At the command prompt type...<br />
<br />
v.to.rast input=Municipalities output=municipalities use=attr col=cell_value<br />
<br />
Note that I gave the output raster the same name as earlier. This will overwrite the original raster thereby minimizing clutter.<br />
<br />
Now also convert the Provinces vector to a raster file <br />
<br />
v.to.rast input=Provinces output=provinces use=attr col=SHP_FID<br />
<br />
The relationship between the two raster files can now be explored using a number of statistics commands most of which are found under r.statistics. Perhapsthe most important statistics are r.sum and r.average. <br />
<br />
The command<br />
r.sum municpalities <br />
<br />
will give the total of all the cell values in the municipalities raster. This equals the total population of the region. However what is actually required in the sum of the population for each province. To get this we use r.statistics. Unfortunately this script does not work for floating point rasters (which is what you are most likely to have). To overcome this problem create a new raster reflecting the integer value of each cell. To create a file useable for r.statistics use..<br />
<br />
r.mapcalc<br />
(then enter line-by-line)<br />
int_municpal=round(municipalities)<br />
end<br />
<br />
If rounding of the cell values to an integer results in a notable decline in accuracy then create a new raster file in which the values are multiplied by ten or a hundred and only then rounded. In latter analysis recall that the units have been dramatically (but precisely) inflated. To do this try "int_munic=round(100*municipalities)".<br />
<br />
To finally aggregate the municipal population to provinces type... <br />
<br />
r.statistics base=provinces cover=int_municipal method=sum output=prov_pop<br />
<br />
This creates a new raster "prov_pop" which can be explored using<br />
<br />
d.what.vect<br />
(click on each region of interest)<br />
<br />
You will see that the values for each province now equals the sum of all the cells that fall into that province. If you want to list the values so that you can put them into Provinces.dbf (i.e. the vector) use..<br />
<br />
r.stats input=RSA_STATS -l <br />
<br />
The output can then be placed into the corresponding data file using a method similar to that above. <br />
<br />
r.statistics comes with a range of other functions other than "sum". These substantially improve the functionality of the method. Statistics include distribution, average, mode, median, average deviation, standard deviation, variance, skewness, kurtosis, minimum and maximum. For assistance type "man r.statistics" from the command prompt. Additional functionality is gained by r.mapcalc's ability to perform functions on multiple raster files. This, in itself, makes the move to raster-based mapping most promising.<br />
<br />
<br />
Sometimes the above procedure results in a notable loss in accuracy. There are two main sources for this: <br />
<br />
The rounding of cell values (to use r.statistics) may be problematic for areas with very small values. If this is the case consider inflating cell values by one of several orders of magnitude. <br />
<br />
When the region was set up a resolution (E-W and N-S) was specified. If this resolution is not fine enough there may be some loss of accuracy through aggregation. Consider increasing the resolution.<br />
<br />
[[Category:FAQ]]</div>⚠️Ergethttps://grasswiki.osgeo.org/w/index.php?title=Vector_aggregate_values&diff=14365Vector aggregate values2011-11-07T21:54:44Z<p>⚠️Erget: </p>
<hr />
<div>GIS users often wish toto aggregate the values of one polygon layer into another. Although this is not integrated as a standard tool in GRASS, it is definitely possible. This article introduces two methods to do just that: A simple, quick operation will be described that can take place entirely in GRASS GIS and produces rasters or, if desired, polygons with aggregated values from two maps and a second, more complex method that uses external editors and creates text files as outputs.<br />
<br />
'''Method 1:''' (written by Daniel Lee)<br />
<br />
In this example we assume that we have a vector map of catchment areas and a raster map of precipitation. We want to find out what the sum total of precipitation per catchment area is. This method would also work if the precipitation data were also stored as vector data, but the vector precipitation data would have to be turned into a raster map, just as the catchment areas are converted to rasters in the following steps. Remember to set the region, etc. in GRASS properly.<br />
<br />
''' Step 1: Convert catchment area polygons to rasters '''<br />
<br />
The polygons are converted to rasters using v.to.rast. The primary key for the polygons should be used as the values for the new rasters. If the polygons were imported from a shapefile, for example, the field FID could be used.<br />
<br />
v.to.rast input=CatchmentAreas output=CatchmentAreas_Raster column=FID<br />
<br />
''' Step 2: Convert precipitation raster to integer values '''<br />
<br />
The tool that we will be using later, r.statistics, requires integer values. Because the precipitation raster is composed of floating point values, we have to convert it to an integer map first. In order to avoid rounding areas in the final results, we also multiply the raster by 100. This way we can divide it by 100 later in order to the original float values back.<br />
If the raster that you want to aggregate into your vectors is already made of integer values or if rounding errors aren't a problem, this step can be changed or left out.<br />
<br />
r.mapcalc 'Precipitation_Integer = round (Precipitation * 100)<br />
<br />
''' Step 3: Aggregate the precipitation values by summing them in the catchment areas they fall in '''<br />
<br />
Now we have two rasters that can be combined. In order to find the sum of the pixels of the precipitation map we use r.statistics.<br />
<br />
r.statistics base=CatchmentAreas_Raster cover=Precipitation_Integer method=sum output=Precipitation_by_CatchmentArea<br />
<br />
''' Step 4: Extract sums from the new map '''<br />
<br />
Now the precipitation sums are aggregated by catchment area, but only as attributes and not as categories in the resulting raster. We now extract the aggregated sums, overwriting the previous raster, by using the "@" function in r.mapcalc. Additionally, the map is divided by 100 to scale the values back, reversing the change we made when we turned the original precipitation map into an integer map.<br />
Don't forget to not divide by 100 if you didn't multiply by 100 in step 2.<br />
<br />
r.mapcalc 'Precipitation_by_CatchmentArea = @Precipitation_by_CatchmentArea / 100'<br />
<br />
''' Step 5: Convert the aggregated precipitation raster to a vector map. '''<br />
<br />
This procedure should already be familiar for those who work with vector maps. If vectors are not needed, it can be skipped.<br />
<br />
r.to.vect input=Precipitation_by_CatchmentArea output=Precipitation_by_CatchmentArea_Vectors<br />
<br />
<br />
'''Method 2:''' (written by Michael O'Donovan)<br />
<br />
GIS users accustomed to proprietary vector-based products (MapInfo, MapMaker, Atlas and some other ESRI) are often disappointed when the try migrating to OpenSource software. Many of the key functions that make vector-based systems so useful are unavailable in OpenSource software like QGIS, Thuban and GRASS. One such function is the ability to aggregate values from one polygon layer to another.<br />
<br />
A typical example would be one of aggregating the population of census tracts (or municipalities) to derive the population of a suburb (or province). The aggregation is simply one of adding the tracts together only when the outer edge of some combination of polygons coincide. This happens when tracts are defined in such a way that they coincide with suburb perimeters. However when the boundaries do not coincide the situation usually calls for a vector-based system that will allocate populations according to the location of the tract centroid or the proportion of a tract falling into a particular suburb. The absence of this aggregation feature discourages potential users from adopting OpenSource software like GRASS. The section that follows shows how, with a little bit of preparation, aggregations can be easily performed by GRASS raster functions. It follows a hypothetical example inspired by the proposed redrawing of South Africa's nine provinces which currently dissect many of the 280+ municipalities. It shows how municipal populations can be re-combined to derive the provincial population by converting the vector features to rasters.<br />
<br />
For simplicity it is assumed that the user has two shape files "Municipalities" and "Province". The "Municipalities" file is linked to a file "Municipalities.dbf" which contains numerous variables including "pop" (its population) and "SHP_FID" (a unique number which identifies each polygon). Both files can be easily imported into GRASS (using v.in.ogr) and displayed along with their variables. Before the vector files can be converted to rasters an unusual preparatory step is required. This entails converting the variables of interest from a value per vector to a value per raster cell. To make this conversion it is necessary to place into the appropriate .dbf file a count of the number of raster cells that will represent that vector. In this example this will enable the user to represent the population not as a number per municipality but as a number per cell. All subsequent aggregations need to be based on these cell values and thus on the cell counts.<br />
<br />
The first step in preparation is to calculate the number of cells in each municipality. To do this convert the municipality vector into a raster file. Remember that, unlike vectors which can be linked to a large number of other variables, raster cells take on a single value only. For this step this value must be a number that uniquely identifies each municipality. If the municipality vector was obtained from an ESRI shapefile the field "SHP_FID" doesthe job. From the command line type:<br />
<br />
v.to.rast input=Municipalities output=municipalities use=attr col=SHP_FID<br />
<br />
This creates a raster file called "municipalities". As vectors and raster are of different types there is little prospect of their names being confused. (Nevertheless vectors are indicated by a starting capital while rasters are kept as lowercase words.) the new raster can be seen by typing... <br />
<br />
d.rast municipalities.<br />
<br />
If is looks funny change the color scheme using something like...<br />
<br />
r.colors map=municipalities color=random<br />
and d.rast it again.<br />
<br />
The next step requires extracting a count of the number of raster cells in each municipality. Once again this is a simple procedure. To save do this and save the output to a file called "munic_count" type...<br />
<br />
r.stats input=municipalities output=munic_count -c<br />
<br />
This will produce a text file with the SHP_FID and a cell count seperated by a space. The final preparatory step is to place the count corresponding to each municipality into the Municipalities.dbf file and to calculate the variables of interest as an amount per cell.<br />
<br />
<hr><br />
This is how you would link the files in OpenOffice if you have not yet set up your data sources and do not know how to use VLOOKUP etc.<br />
<br />
# Open the file "munic_count" - it will default to a text page containing the SHP_FID, an open space and the cell count.<br />
# Convert the open spaces to tabs using "Edit", "Find and Replace". Place a space in the "search for" box and \t in the "replace with" boxes. Activate "Regular expressions" and click on "Replace all".<br />
# Select all of the data (Ctrl A) and copy it to a clipboard (Ctrl C)<br />
<br />
You now have all the data ordered by SHP_FID ready to be pasted into the Municpalities.dbf file. So open the Municipalities.dbf file - it will default to a spreadsheet.<br />
<br />
# Make sure that the spreadsheet data is in ascending ordered of SHP_FID. (Select all data other than the headers then go to "Data", "Sort" and select the appropriate column).<br />
# Click on to cell at the top of the first unused column and click on "Edit", "Paste special" and "unformatted text". The cells values and SHP_FID data should now be placed in the corresponding rows. Check that this has been done.<br />
<br />
Variables of interest.<br />
<br />
If you are interested in, say, aggregating populations then calculate the population per cell for every municipality. (If your data starts in row 1, population is in column C and the cell count in column K type "=C1/K1". Copy and then paste the formula into all other rows.) Give the new column an appropraite name, say, "cell_value" and save the file under its original name (i.e. as a .dbf). You may well return at a latter stage and calculate new variables of interest using the cell counts. <br />
<br />
<hr><br />
<br />
You can now proceed with the aggregations.<br />
<br />
First convert the Municipalities vector into a raster (again). This time use the cell_value as the attribute of interest. At the command prompt type...<br />
<br />
v.to.rast input=Municipalities output=municipalities use=attr col=cell_value<br />
<br />
Note that I gave the output raster the same name as earlier. This will overwrite the original raster thereby minimizing clutter.<br />
<br />
Now also convert the Provinces vector to a raster file <br />
<br />
v.to.rast input=Provinces output=provinces use=attr col=SHP_FID<br />
<br />
The relationship between the two raster files can now be explored using a number of statistics commands most of which are found under r.statistics. Perhapsthe most important statistics are r.sum and r.average. <br />
<br />
The command<br />
r.sum municpalities <br />
<br />
will give the total of all the cell values in the municipalities raster. This equals the total population of the region. However what is actually required in the sum of the population for each province. To get this we use r.statistics. Unfortunately this script does not work for floating point rasters (which is what you are most likely to have). To overcome this problem create a new raster reflecting the integer value of each cell. To create a file useable for r.statistics use..<br />
<br />
r.mapcalc<br />
(then enter line-by-line)<br />
int_municpal=round(municipalities)<br />
end<br />
<br />
If rounding of the cell values to an integer results in a notable decline in accuracy then create a new raster file in which the values are multiplied by ten or a hundred and only then rounded. In latter analysis recall that the units have been dramatically (but precisely) inflated. To do this try "int_munic=round(100*municipalities)".<br />
<br />
To finally aggregate the municipal population to provinces type... <br />
<br />
r.statistics base=provinces cover=int_municipal method=sum output=prov_pop<br />
<br />
This creates a new raster "prov_pop" which can be explored using<br />
<br />
d.what.vect<br />
(click on each region of interest)<br />
<br />
You will see that the values for each province now equals the sum of all the cells that fall into that province. If you want to list the values so that you can put them into Provinces.dbf (i.e. the vector) use..<br />
<br />
r.stats input=RSA_STATS -l <br />
<br />
The output can then be placed into the corresponding data file using a method similar to that above. <br />
<br />
r.statistics comes with a range of other functions other than "sum". These substantially improve the functionality of the method. Statistics include distribution, average, mode, median, average deviation, standard deviation, variance, skewness, kurtosis, minimum and maximum. For assistance type "man r.statistics" from the command prompt. Additional functionality is gained by r.mapcalc's ability to perform functions on multiple raster files. This, in itself, makes the move to raster-based mapping most promising.<br />
<br />
<br />
Sometimes the above procedure results in a notable loss in accuracy. There are two main sources for this: <br />
<br />
The rounding of cell values (to use r.statistics) may be problematic for areas with very small values. If this is the case consider inflating cell values by one of several orders of magnitude. <br />
<br />
When the region was set up a resolution (E-W and N-S) was specified. If this resolution is not fine enough there may be some loss of accuracy through aggregation. Consider increasing the resolution.<br />
<br />
[[Category:FAQ]]</div>⚠️Ergethttps://grasswiki.osgeo.org/w/index.php?title=Vector_aggregate_values&diff=14364Vector aggregate values2011-11-07T21:53:21Z<p>⚠️Erget: </p>
<hr />
<div>GIS users often wish toto aggregate the values of one polygon layer into another. Although this is not integrated as a standard tool in GRASS, it is definitely possible. This article introduces two methods to do just that: A simple, quick operation will be described that can take place entirely in GRASS GIS and produces rasters or, if desired, polygons with aggregated values from two maps and a second, more complex method that uses external editors and creates text files as outputs.<br />
<br />
'''Method 1:''' (written by Daniel Lee)<br />
<br />
In this example we assume that we have a vector map of catchment areas and a raster map of precipitation. We want to find out what the sum total of precipitation per catchment area is. This method would also work if the precipitation data were also stored as vector data, but the vector precipitation data would have to be turned into a raster map, just as the catchment areas are converted to rasters in the following steps. Remember to set the region, etc. in GRASS properly.<br />
<br />
''' Step 1: Convert catchment area polygons to rasters '''<br />
The polygons are converted to rasters using v.to.rast. The primary key for the polygons should be used as the values for the new rasters. If the polygons were imported from a shapefile, for example, the field FID could be used.<br />
<br />
v.to.rast input=CatchmentAreas output=CatchmentAreas_Raster column=FID<br />
<br />
''' Step 2: Convert precipitation raster to integer values '''<br />
The tool that we will be using later, r.statistics, requires integer values. Because the precipitation raster is composed of floating point values, we have to convert it to an integer map first. In order to avoid rounding areas in the final results, we also multiply the raster by 100. This way we can divide it by 100 later in order to the original float values back.<br />
If the raster that you want to aggregate into your vectors is already made of integer values or if rounding errors aren't a problem, this step can be changed or left out.<br />
<br />
r.mapcalc 'Precipitation_Integer = round (Precipitation * 100)<br />
<br />
''' Step 3: Aggregate the precipitation values by summing them in the catchment areas they fall in '''<br />
Now we have two rasters that can be combined. In order to find the sum of the pixels of the precipitation map we use r.statistics.<br />
<br />
r.statistics base=CatchmentAreas_Raster cover=Precipitation_Integer method=sum output=Precipitation_by_CatchmentArea<br />
<br />
''' Step 4: Extract sums from the new map '''<br />
Now the precipitation sums are aggregated by catchment area, but only as attributes and not as categories in the resulting raster. We now extract the aggregated sums, overwriting the previous raster, by using the "@" function in r.mapcalc. Additionally, the map is divided by 100 to scale the values back, reversing the change we made when we turned the original precipitation map into an integer map.<br />
Don't forget to not divide by 100 if you didn't multiply by 100 in step 2.<br />
<br />
r.mapcalc 'Precipitation_by_CatchmentArea = @Precipitation_by_CatchmentArea / 100'<br />
<br />
''' Step 5: Convert the aggregated precipitation raster to a vector map. '''<br />
This procedure should already be familiar for those who work with vector maps. If vectors are not needed, it can be skipped.<br />
<br />
r.to.vect input=Precipitation_by_CatchmentArea output=Precipitation_by_CatchmentArea_Vectors<br />
<br />
<br />
'''Method 2:''' (written by Michael O'Donovan)<br />
<br />
GIS users accustomed to proprietary vector-based products (MapInfo, MapMaker, Atlas and some other ESRI) are often disappointed when the try migrating to OpenSource software. Many of the key functions that make vector-based systems so useful are unavailable in OpenSource software like QGIS, Thuban and GRASS. One such function is the ability to aggregate values from one polygon layer to another.<br />
<br />
A typical example would be one of aggregating the population of census tracts (or municipalities) to derive the population of a suburb (or province). The aggregation is simply one of adding the tracts together only when the outer edge of some combination of polygons coincide. This happens when tracts are defined in such a way that they coincide with suburb perimeters. However when the boundaries do not coincide the situation usually calls for a vector-based system that will allocate populations according to the location of the tract centroid or the proportion of a tract falling into a particular suburb. The absence of this aggregation feature discourages potential users from adopting OpenSource software like GRASS. The section that follows shows how, with a little bit of preparation, aggregations can be easily performed by GRASS raster functions. It follows a hypothetical example inspired by the proposed redrawing of South Africa's nine provinces which currently dissect many of the 280+ municipalities. It shows how municipal populations can be re-combined to derive the provincial population by converting the vector features to rasters.<br />
<br />
For simplicity it is assumed that the user has two shape files "Municipalities" and "Province". The "Municipalities" file is linked to a file "Municipalities.dbf" which contains numerous variables including "pop" (its population) and "SHP_FID" (a unique number which identifies each polygon). Both files can be easily imported into GRASS (using v.in.ogr) and displayed along with their variables. Before the vector files can be converted to rasters an unusual preparatory step is required. This entails converting the variables of interest from a value per vector to a value per raster cell. To make this conversion it is necessary to place into the appropriate .dbf file a count of the number of raster cells that will represent that vector. In this example this will enable the user to represent the population not as a number per municipality but as a number per cell. All subsequent aggregations need to be based on these cell values and thus on the cell counts.<br />
<br />
The first step in preparation is to calculate the number of cells in each municipality. To do this convert the municipality vector into a raster file. Remember that, unlike vectors which can be linked to a large number of other variables, raster cells take on a single value only. For this step this value must be a number that uniquely identifies each municipality. If the municipality vector was obtained from an ESRI shapefile the field "SHP_FID" doesthe job. From the command line type:<br />
<br />
v.to.rast input=Municipalities output=municipalities use=attr col=SHP_FID<br />
<br />
This creates a raster file called "municipalities". As vectors and raster are of different types there is little prospect of their names being confused. (Nevertheless vectors are indicated by a starting capital while rasters are kept as lowercase words.) the new raster can be seen by typing... <br />
<br />
d.rast municipalities.<br />
<br />
If is looks funny change the color scheme using something like...<br />
<br />
r.colors map=municipalities color=random<br />
and d.rast it again.<br />
<br />
The next step requires extracting a count of the number of raster cells in each municipality. Once again this is a simple procedure. To save do this and save the output to a file called "munic_count" type...<br />
<br />
r.stats input=municipalities output=munic_count -c<br />
<br />
This will produce a text file with the SHP_FID and a cell count seperated by a space. The final preparatory step is to place the count corresponding to each municipality into the Municipalities.dbf file and to calculate the variables of interest as an amount per cell.<br />
<br />
<hr><br />
This is how you would link the files in OpenOffice if you have not yet set up your data sources and do not know how to use VLOOKUP etc.<br />
<br />
# Open the file "munic_count" - it will default to a text page containing the SHP_FID, an open space and the cell count.<br />
# Convert the open spaces to tabs using "Edit", "Find and Replace". Place a space in the "search for" box and \t in the "replace with" boxes. Activate "Regular expressions" and click on "Replace all".<br />
# Select all of the data (Ctrl A) and copy it to a clipboard (Ctrl C)<br />
<br />
You now have all the data ordered by SHP_FID ready to be pasted into the Municpalities.dbf file. So open the Municipalities.dbf file - it will default to a spreadsheet.<br />
<br />
# Make sure that the spreadsheet data is in ascending ordered of SHP_FID. (Select all data other than the headers then go to "Data", "Sort" and select the appropriate column).<br />
# Click on to cell at the top of the first unused column and click on "Edit", "Paste special" and "unformatted text". The cells values and SHP_FID data should now be placed in the corresponding rows. Check that this has been done.<br />
<br />
Variables of interest.<br />
<br />
If you are interested in, say, aggregating populations then calculate the population per cell for every municipality. (If your data starts in row 1, population is in column C and the cell count in column K type "=C1/K1". Copy and then paste the formula into all other rows.) Give the new column an appropraite name, say, "cell_value" and save the file under its original name (i.e. as a .dbf). You may well return at a latter stage and calculate new variables of interest using the cell counts. <br />
<br />
<hr><br />
<br />
You can now proceed with the aggregations.<br />
<br />
First convert the Municipalities vector into a raster (again). This time use the cell_value as the attribute of interest. At the command prompt type...<br />
<br />
v.to.rast input=Municipalities output=municipalities use=attr col=cell_value<br />
<br />
Note that I gave the output raster the same name as earlier. This will overwrite the original raster thereby minimizing clutter.<br />
<br />
Now also convert the Provinces vector to a raster file <br />
<br />
v.to.rast input=Provinces output=provinces use=attr col=SHP_FID<br />
<br />
The relationship between the two raster files can now be explored using a number of statistics commands most of which are found under r.statistics. Perhapsthe most important statistics are r.sum and r.average. <br />
<br />
The command<br />
r.sum municpalities <br />
<br />
will give the total of all the cell values in the municipalities raster. This equals the total population of the region. However what is actually required in the sum of the population for each province. To get this we use r.statistics. Unfortunately this script does not work for floating point rasters (which is what you are most likely to have). To overcome this problem create a new raster reflecting the integer value of each cell. To create a file useable for r.statistics use..<br />
<br />
r.mapcalc<br />
(then enter line-by-line)<br />
int_municpal=round(municipalities)<br />
end<br />
<br />
If rounding of the cell values to an integer results in a notable decline in accuracy then create a new raster file in which the values are multiplied by ten or a hundred and only then rounded. In latter analysis recall that the units have been dramatically (but precisely) inflated. To do this try "int_munic=round(100*municipalities)".<br />
<br />
To finally aggregate the municipal population to provinces type... <br />
<br />
r.statistics base=provinces cover=int_municipal method=sum output=prov_pop<br />
<br />
This creates a new raster "prov_pop" which can be explored using<br />
<br />
d.what.vect<br />
(click on each region of interest)<br />
<br />
You will see that the values for each province now equals the sum of all the cells that fall into that province. If you want to list the values so that you can put them into Provinces.dbf (i.e. the vector) use..<br />
<br />
r.stats input=RSA_STATS -l <br />
<br />
The output can then be placed into the corresponding data file using a method similar to that above. <br />
<br />
r.statistics comes with a range of other functions other than "sum". These substantially improve the functionality of the method. Statistics include distribution, average, mode, median, average deviation, standard deviation, variance, skewness, kurtosis, minimum and maximum. For assistance type "man r.statistics" from the command prompt. Additional functionality is gained by r.mapcalc's ability to perform functions on multiple raster files. This, in itself, makes the move to raster-based mapping most promising.<br />
<br />
<br />
Sometimes the above procedure results in a notable loss in accuracy. There are two main sources for this: <br />
<br />
The rounding of cell values (to use r.statistics) may be problematic for areas with very small values. If this is the case consider inflating cell values by one of several orders of magnitude. <br />
<br />
When the region was set up a resolution (E-W and N-S) was specified. If this resolution is not fine enough there may be some loss of accuracy through aggregation. Consider increasing the resolution.<br />
<br />
[[Category:FAQ]]</div>⚠️Ergethttps://grasswiki.osgeo.org/w/index.php?title=Vector_aggregate_values&diff=14363Vector aggregate values2011-11-07T21:49:59Z<p>⚠️Erget: </p>
<hr />
<div>GIS users often wish toto aggregate the values of one polygon layer into another. Although this is not integrated as a standard tool in GRASS, it is definitely possible. This article introduces two methods to do just that: A simple, quick operation will be described that can take place entirely in GRASS GIS and produces rasters or, if desired, polygons with aggregated values from two maps and a second, more complex method that uses external editors and creates text files as outputs.<br />
<br />
'''Method 1:''' (written by Daniel Lee)<br />
In this example we assume that we have a vector map of catchment areas and a raster map of precipitation. We want to find out what the sum total of precipitation per catchment area is. This method would also work if the precipitation data were also stored as vector data, but the vector precipitation data would have to be turned into a raster map, just as the catchment areas are converted to rasters in the following steps. Remember to set the region, etc. in GRASS properly.<br />
<br />
Step 1: Convert catchment area polygons to rasters<br />
The polygons are converted to rasters using v.to.rast. The primary key for the polygons should be used as the values for the new rasters. If the polygons were imported from a shapefile, for example, the field FID could be used.<br />
<br />
v.to.rast input=CatchmentAreas output=CatchmentAreas_Raster column=FID<br />
<br />
Step 2: Convert precipitation raster to integer values<br />
The tool that we will be using later, r.statistics, requires integer values. Because the precipitation raster is composed of floating point values, we have to convert it to an integer map first. In order to avoid rounding areas in the final results, we also multiply the raster by 100. This way we can divide it by 100 later in order to the original float values back.<br />
If the raster that you want to aggregate into your vectors is already made of integer values or if rounding errors aren't a problem, this step can be changed or left out.<br />
<br />
r.mapcalc 'Precipitation_Integer = round (Precipitation * 100)<br />
<br />
Step 3: Aggregate the precipitation values by summing them in the catchment areas they fall in<br />
Now we have two rasters that can be combined. In order to find the sum of the pixels of the precipitation map we use r.statistics.<br />
<br />
r.statistics base=CatchmentAreas_Raster cover=Precipitation_Integer method=sum output=Precipitation_by_CatchmentArea<br />
<br />
Step 4: Extract sums from the new map<br />
Now the precipitation sums are aggregated by catchment area, but only as attributes and not as categories in the resulting raster. We now extract the aggregated sums, overwriting the previous raster, by using the "@" function in r.mapcalc. Additionally, the map is divided by 100 to scale the values back, reversing the change we made when we turned the original precipitation map into an integer map.<br />
Don't forget to not divide by 100 if you didn't multiply by 100 in step 2.<br />
<br />
r.mapcalc 'Precipitation_by_CatchmentArea = @Precipitation_by_CatchmentArea / 100'<br />
<br />
Step 5: Convert the aggregated precipitation raster to a vector map.<br />
This procedure should already be familiar for those who work with vector maps. If vectors are not needed, it can be skipped.<br />
<br />
r.to.vect input=Precipitation_by_CatchmentArea output=Precipitation_by_CatchmentArea_Vectors<br />
<br />
<br />
'''Method 2:''' (written by Michael O'Donovan)<br />
<br />
GIS users accustomed to proprietary vector-based products (MapInfo, MapMaker, Atlas and some other ESRI) are often disappointed when the try migrating to OpenSource software. Many of the key functions that make vector-based systems so useful are unavailable in OpenSource software like QGIS, Thuban and GRASS. One such function is the ability to aggregate values from one polygon layer to another.<br />
<br />
A typical example would be one of aggregating the population of census tracts (or municipalities) to derive the population of a suburb (or province). The aggregation is simply one of adding the tracts together only when the outer edge of some combination of polygons coincide. This happens when tracts are defined in such a way that they coincide with suburb perimeters. However when the boundaries do not coincide the situation usually calls for a vector-based system that will allocate populations according to the location of the tract centroid or the proportion of a tract falling into a particular suburb. The absence of this aggregation feature discourages potential users from adopting OpenSource software like GRASS. The section that follows shows how, with a little bit of preparation, aggregations can be easily performed by GRASS raster functions. It follows a hypothetical example inspired by the proposed redrawing of South Africa's nine provinces which currently dissect many of the 280+ municipalities. It shows how municipal populations can be re-combined to derive the provincial population by converting the vector features to rasters.<br />
<br />
For simplicity it is assumed that the user has two shape files "Municipalities" and "Province". The "Municipalities" file is linked to a file "Municipalities.dbf" which contains numerous variables including "pop" (its population) and "SHP_FID" (a unique number which identifies each polygon). Both files can be easily imported into GRASS (using v.in.ogr) and displayed along with their variables. Before the vector files can be converted to rasters an unusual preparatory step is required. This entails converting the variables of interest from a value per vector to a value per raster cell. To make this conversion it is necessary to place into the appropriate .dbf file a count of the number of raster cells that will represent that vector. In this example this will enable the user to represent the population not as a number per municipality but as a number per cell. All subsequent aggregations need to be based on these cell values and thus on the cell counts.<br />
<br />
The first step in preparation is to calculate the number of cells in each municipality. To do this convert the municipality vector into a raster file. Remember that, unlike vectors which can be linked to a large number of other variables, raster cells take on a single value only. For this step this value must be a number that uniquely identifies each municipality. If the municipality vector was obtained from an ESRI shapefile the field "SHP_FID" doesthe job. From the command line type:<br />
<br />
v.to.rast input=Municipalities output=municipalities use=attr col=SHP_FID<br />
<br />
This creates a raster file called "municipalities". As vectors and raster are of different types there is little prospect of their names being confused. (Nevertheless vectors are indicated by a starting capital while rasters are kept as lowercase words.) the new raster can be seen by typing... <br />
<br />
d.rast municipalities.<br />
<br />
If is looks funny change the color scheme using something like...<br />
<br />
r.colors map=municipalities color=random<br />
and d.rast it again.<br />
<br />
The next step requires extracting a count of the number of raster cells in each municipality. Once again this is a simple procedure. To save do this and save the output to a file called "munic_count" type...<br />
<br />
r.stats input=municipalities output=munic_count -c<br />
<br />
This will produce a text file with the SHP_FID and a cell count seperated by a space. The final preparatory step is to place the count corresponding to each municipality into the Municipalities.dbf file and to calculate the variables of interest as an amount per cell.<br />
<br />
<hr><br />
This is how you would link the files in OpenOffice if you have not yet set up your data sources and do not know how to use VLOOKUP etc.<br />
<br />
# Open the file "munic_count" - it will default to a text page containing the SHP_FID, an open space and the cell count.<br />
# Convert the open spaces to tabs using "Edit", "Find and Replace". Place a space in the "search for" box and \t in the "replace with" boxes. Activate "Regular expressions" and click on "Replace all".<br />
# Select all of the data (Ctrl A) and copy it to a clipboard (Ctrl C)<br />
<br />
You now have all the data ordered by SHP_FID ready to be pasted into the Municpalities.dbf file. So open the Municipalities.dbf file - it will default to a spreadsheet.<br />
<br />
# Make sure that the spreadsheet data is in ascending ordered of SHP_FID. (Select all data other than the headers then go to "Data", "Sort" and select the appropriate column).<br />
# Click on to cell at the top of the first unused column and click on "Edit", "Paste special" and "unformatted text". The cells values and SHP_FID data should now be placed in the corresponding rows. Check that this has been done.<br />
<br />
Variables of interest.<br />
<br />
If you are interested in, say, aggregating populations then calculate the population per cell for every municipality. (If your data starts in row 1, population is in column C and the cell count in column K type "=C1/K1". Copy and then paste the formula into all other rows.) Give the new column an appropraite name, say, "cell_value" and save the file under its original name (i.e. as a .dbf). You may well return at a latter stage and calculate new variables of interest using the cell counts. <br />
<br />
<hr><br />
<br />
You can now proceed with the aggregations.<br />
<br />
First convert the Municipalities vector into a raster (again). This time use the cell_value as the attribute of interest. At the command prompt type...<br />
<br />
v.to.rast input=Municipalities output=municipalities use=attr col=cell_value<br />
<br />
Note that I gave the output raster the same name as earlier. This will overwrite the original raster thereby minimizing clutter.<br />
<br />
Now also convert the Provinces vector to a raster file <br />
<br />
v.to.rast input=Provinces output=provinces use=attr col=SHP_FID<br />
<br />
The relationship between the two raster files can now be explored using a number of statistics commands most of which are found under r.statistics. Perhapsthe most important statistics are r.sum and r.average. <br />
<br />
The command<br />
r.sum municpalities <br />
<br />
will give the total of all the cell values in the municipalities raster. This equals the total population of the region. However what is actually required in the sum of the population for each province. To get this we use r.statistics. Unfortunately this script does not work for floating point rasters (which is what you are most likely to have). To overcome this problem create a new raster reflecting the integer value of each cell. To create a file useable for r.statistics use..<br />
<br />
r.mapcalc<br />
(then enter line-by-line)<br />
int_municpal=round(municipalities)<br />
end<br />
<br />
If rounding of the cell values to an integer results in a notable decline in accuracy then create a new raster file in which the values are multiplied by ten or a hundred and only then rounded. In latter analysis recall that the units have been dramatically (but precisely) inflated. To do this try "int_munic=round(100*municipalities)".<br />
<br />
To finally aggregate the municipal population to provinces type... <br />
<br />
r.statistics base=provinces cover=int_municipal method=sum output=prov_pop<br />
<br />
This creates a new raster "prov_pop" which can be explored using<br />
<br />
d.what.vect<br />
(click on each region of interest)<br />
<br />
You will see that the values for each province now equals the sum of all the cells that fall into that province. If you want to list the values so that you can put them into Provinces.dbf (i.e. the vector) use..<br />
<br />
r.stats input=RSA_STATS -l <br />
<br />
The output can then be placed into the corresponding data file using a method similar to that above. <br />
<br />
r.statistics comes with a range of other functions other than "sum". These substantially improve the functionality of the method. Statistics include distribution, average, mode, median, average deviation, standard deviation, variance, skewness, kurtosis, minimum and maximum. For assistance type "man r.statistics" from the command prompt. Additional functionality is gained by r.mapcalc's ability to perform functions on multiple raster files. This, in itself, makes the move to raster-based mapping most promising.<br />
<br />
<br />
Sometimes the above procedure results in a notable loss in accuracy. There are two main sources for this: <br />
<br />
The rounding of cell values (to use r.statistics) may be problematic for areas with very small values. If this is the case consider inflating cell values by one of several orders of magnitude. <br />
<br />
When the region was set up a resolution (E-W and N-S) was specified. If this resolution is not fine enough there may be some loss of accuracy through aggregation. Consider increasing the resolution.<br />
<br />
[[Category:FAQ]]</div>⚠️Ergethttps://grasswiki.osgeo.org/w/index.php?title=Energy_calculations&diff=13547Energy calculations2011-05-29T18:26:03Z<p>⚠️Erget: </p>
<hr />
<div>== GRASS and (solar) energy calculations ==<br />
<br />
Documents in English:<br />
* GRASS4LEED: Building geospatial support for Leadership in Environmental and Energy Design. (http://www.foss4g2006.org/contributionDisplay.py?contribId=188&sessionId=40&confId=1)<br />
* Photovoltaic Geographical Information System (PVGIS, based on r.sun) (http://re.jrc.ec.europa.eu/pvgis/)<br />
* Solar Lab tutorial http://istgeo.ist.supsi.ch/site/?q=node/3<br />
* Hofierka, J & Šúri, M, 2002: The solar radiation model for Open source GIS: implementation and applications, Proceedings of the Open source GIS - GRASS users conference 2002 - Trento, Italy, 11-13 September 2002, [http://www.ing.unitn.it/~grass/conferences/GRASS2002/proceedings/proceedings/pdfs/Hofierka_Jaroslav.pdf PDF]<br />
* Huld, T. A., Šúri, M., Dunlop, E. D., 2003: GIS-based Estimation of Solar Radiation and PV Generation in Central and Eastern Europe on the Web. 9th EC-GI&GIS Workshop, A Coruña (Spain), 25-27 June, [http://re.jrc.ec.europa.eu/pvgis/doc/paper/2003_huld_coruna.pdf PDF]<br />
* Šúri, M., Huld, T. A., Dunlop, E. D., Hofierka, J. (2007): Solar resource modelling for energy applications. In: Peckham R.J., Jordan G. (eds.) Digital Terrain Modelling, Development and Applications in a Policy Support Environment, Series: Lecture Notes in Geoinformation and Cartography, Springer, pp. 259-273, ISBN: 978-3-540-36730-7<br />
* Hofierka, J. Solar radiation in the Slovak republic, [http://www.fhpv.unipo.sk/kagerr/pracovnici/hofierka/projekty/solar_rad_database.html PDF]<br />
* Hofierka, J., Kaňuk, J. (2009): Assessment of Photovoltaic Potential in Urban Areas Using Open-Source Solar Radiation Tools. Renewable Energy. In Press.<br />
* ISIS: International Solar Information Solutions. Solar analyses using GRASS, based largely on the work done for PVGIS (http://www.isi-solutions.org)<br />
* Search for [http://scholar.google.com/scholar?q=grass+gis+energy GRASS GIS energy] in Google Scholar<br />
<br />
Documents in Italian:<br />
* Mondogis N. 61 (luglio/agosto 2007): Il portale energetico sulle energie rinnovabili per la città di Laives [http://www.r3-gis.com/upload/pages/mondogis-61-2007_laives.pdf PDF] (GRASS, PostGIS)<br />
* Paolo Zatelli. Valutazione di radiazione solare diretta su falde di tetti inclinati con GRASS. IX Meeting degli Utenti Italiani di GRASS GIS-GFOSS. (http://www.grassmeeting2008.unipg.it/?q=node/9)<br />
<br />
Software:<br />
* GRASS modules {{cmd|r.horizon}} and [[r.sun]]<br />
<br />
<br />
[[Category:Applications]]<br />
[[Category: Documentation]]</div>⚠️Ergethttps://grasswiki.osgeo.org/w/index.php?title=Contact_Databases&diff=11978Contact Databases2010-10-23T21:44:19Z<p>⚠️Erget: /* German */</p>
<hr />
<div>=Contact Database=<br />
<br />
Here we can collect all contacts where we can send press releases. I will build a webbased formular later. <br />
The following is ordered by language.<br />
<br />
==English==<br />
*[http://www.slashgeo.org/submit.pl slashgeo] (online)<br />
*[http://freshmeat.net/projects/grass/?highlight=GRASS Freshmeat] (update: MN)<br />
*[http://linuxtoday.com/contribute.php3 linuxtoday]<br />
*[http://www.newsforge.com/submit.pl newsforge]<br />
*[http://www.macnn.com/contact/newstips/1] (email to "contact" -> news tips)<br />
*[http://www10.giscafe.com/submit_material/submit_options.php#Press giscafe]<br />
*[http://www.directionsmag.com/pressreleases.php directionsmag] (News -> Post Press Release)<br />
*[http://news.eoportal.org/cgi-bin/eoportal_header.pl?dw=psr&site=software&words=type eoportal] (Share your news with the EO community)<br />
*[http://gislounge.com/freisin/submits.shtml gislounge]<br />
*[http://cartographyonline.com] (admin at cartographyonline.com)<br />
*[http://www.earsc.org/contact EARSC EOMag Newsletter]<br />
*[http://e-geoinfo.net/index.html GeoinfoNEWS]<br />
*[http://mycoordinates.org/submit-papers/ Coordinates magazine] India<br />
*[http://www.osor.eu/ OSOR]<br />
*[http://www.geoconnexion.com/ GeoConnexion]<br />
*[http://www.gim-international.com GIM International]<br />
<br />
* Meta-Liste at: http://www.giswiki.org/wiki/News<br />
*[http://freshmeat.net/ freshmeat] (online)<br />
*[http://freegis.org/ freegis] (online)<br />
<br />
==German==<br />
*[http://www.heise.de heise] (print / online): presse_at_ct.heise.de + [http://www.heise.de/software/download/edit_7105 software]<br />
*[http://www.harzer.de www.harzer.de] (online): news_at_geobranchen.de<br />
* Universities:<br />
** [http://www.gis.uni-kiel.de/index.phtml LearnGIS]: Giszentrum der Universität Kiel<br />
** [http://www.gis-ma.org gis.ma]: GIS-Lab Marburg (Universität Marburg)<br />
<br />
== Italian ==<br />
<br />
* MondoGIS / GeoForus.it ([http://www.geoforus.it/index.php?option=com_contact&view=category&catid=2&Itemid=5 contatti])<br />
* [http://punto-informatico.it/ Punto informatico]<br />
<br />
==Other languages==<br />
...<br />
<br />
<br />
=Translators=<br />
Please add yourself here as volunteer to translate GRASS press releases (should be more than one for every language).<br />
<br />
==German==<br />
* Malte Halbey-Martin<br />
<br />
----<br />
<br />
[[Category:Promotion]]</div>⚠️Ergethttps://grasswiki.osgeo.org/w/index.php?title=GRASS_promotion_team&diff=11977GRASS promotion team2010-10-23T21:34:00Z<p>⚠️Erget: /* People (add yourself here) */</p>
<hr />
<div>=GRASS Promotion=<br />
Here can the GRASS promotion team coordinate its work.<br />
<br />
; Where can we get some resources (financial or support) to print the brochures<br />
: Ideally you get yourself a corporate sponsor who either does the printing (Autodesk has been very helpful in this respect and they have offices everywhere) or sponsors the printing costs. For example, in Germany you can also directly request for OSGeo funds through [http://www.fossgis.de/ FOSSGIS e.V.], in Italy through [http://www.gfoss.it/ GFOSS.it]. Describe what event you need the brochures for and add an estimate of the printing costs. For larger international events you should contact OSGeo's [http://www.osgeo.org/visibility Marketing Committee], they have a small budget but mostly they can help to coordinate.<br />
<br />
= Todo (add yourself) =<br />
== As soon as possible ==<br />
1. [[Grassbrochure]] <br />
*a multipage brochure, maybe something like: [http://www.esri.com/library/brochures/pdfs/avgisbro.pdf Example of the market leader]<br />
* Maybe same layout as OSGeo brochure, contact with OSGeo [http://www.osgeo.org/visibility Marketing Committee] <br />
<br />
2. <strike>[[Promotional material|GRASS Flyer]]</strike> (done)<br />
<br />
3. <strike>Technical Data sheet</strike><br />
*Done, 2nd page of flyer<br />
<br />
4. Newbie friendly tutorial<br />
* Update Lorzeno Moretti's "''[http://grass.bologna.enea.it/tutorial/01-tutorial/ Visual Tutorial for GRASS 6]''"?<br />
* [[GRASS_Help#First_Day_Documentation|First day documentation]]<br />
<br />
5. Live CD (maintenance to keep up to date,)<br />
* OSGeo LiveCD team provides a fancy promo CD: http://download.osgeo.org/livedvd/<br />
* [http://ldap.telascience.org/foss4g/ FOSS4G2006 Lausanne/Switzerland ''GRASS Workshop LiveCD 2006'']<br />
* Others: http://grass.osgeo.org/download/cdrom.php<br />
<br />
6. Find some funding for printing the flyer / brochure<br />
<!--* German GRASS user group ([http://www.grass-verein.de GAV e.V.]) offers some funding for flyer--><br />
* How much is needed?<br />
<br />
''Depends on the quality & number of flyers we want to order''<br />
''I found one shop where we can get 2500 flyer for about 170 - 270 € (aprox. 225 - 355 US $. depending on paper/colors...)''<br />
* Find a way to spread the flyer<br />
<br />
== Later or synchronous == <br />
* Translation of the GRASS-Flyer (Latex-file available on the Grass-Addons SVN [http://trac.osgeo.org/grass/browser/grass-promo/grassflyer GRASS-ADDONS-SVN)<br />
* <strike>Newbie Forum</strike> (done)<br />
* [[Contact Databases]]<br />
* Case Histories of GRASS adoption (?)<br />
** [[Publications|Publications made with GRASS]]: on this page publications made with GRASS should be listened<br />
** [[Use Cases|Use Cases successful applications with GRASS]]<br />
* Nice Posters with applications where GRASS hass been adopted<br />
* <strike>workaround of [http://www.osgeo.org/grass http://www.osgeo.org/grass]</strike> ''done''<br />
* Maybe offer some stuff on coffeepress / spreadshirt<br />
** for Germany / Europe in progress (thx to Martin Wegmann)<br />
* Stickers<br />
* Proofreading<br />
<br />
= People (add yourself here) =<br />
* Malte Halbey-Martin (maltehalbey on irc): email malte [at] geog.fu-berlin.de<br />
* Ominiverdi (doktoreas or ominoverde on irc) : email info [at] ominiverdi.org<br />
* Chip Mefford (cpm on irc): email cpm [at] daviswv.net or cpm [at] well.com<br />
* Carlos Grohmann: email carlos.grohmann [at] gmail.com<br />
* Milena Nowotarska (milenaN on irc): email do.milenki [at] gmail.com<br />
* Daniel Lee: email Lee.Daniel.1986 [at] gmail.com<br />
<br />
[[Category:Promotion]]</div>⚠️Ergethttps://grasswiki.osgeo.org/w/index.php?title=GRASS_promotion_team&diff=11976GRASS promotion team2010-10-23T21:33:05Z<p>⚠️Erget: /* Later or synchronous */</p>
<hr />
<div>=GRASS Promotion=<br />
Here can the GRASS promotion team coordinate its work.<br />
<br />
; Where can we get some resources (financial or support) to print the brochures<br />
: Ideally you get yourself a corporate sponsor who either does the printing (Autodesk has been very helpful in this respect and they have offices everywhere) or sponsors the printing costs. For example, in Germany you can also directly request for OSGeo funds through [http://www.fossgis.de/ FOSSGIS e.V.], in Italy through [http://www.gfoss.it/ GFOSS.it]. Describe what event you need the brochures for and add an estimate of the printing costs. For larger international events you should contact OSGeo's [http://www.osgeo.org/visibility Marketing Committee], they have a small budget but mostly they can help to coordinate.<br />
<br />
= Todo (add yourself) =<br />
== As soon as possible ==<br />
1. [[Grassbrochure]] <br />
*a multipage brochure, maybe something like: [http://www.esri.com/library/brochures/pdfs/avgisbro.pdf Example of the market leader]<br />
* Maybe same layout as OSGeo brochure, contact with OSGeo [http://www.osgeo.org/visibility Marketing Committee] <br />
<br />
2. <strike>[[Promotional material|GRASS Flyer]]</strike> (done)<br />
<br />
3. <strike>Technical Data sheet</strike><br />
*Done, 2nd page of flyer<br />
<br />
4. Newbie friendly tutorial<br />
* Update Lorzeno Moretti's "''[http://grass.bologna.enea.it/tutorial/01-tutorial/ Visual Tutorial for GRASS 6]''"?<br />
* [[GRASS_Help#First_Day_Documentation|First day documentation]]<br />
<br />
5. Live CD (maintenance to keep up to date,)<br />
* OSGeo LiveCD team provides a fancy promo CD: http://download.osgeo.org/livedvd/<br />
* [http://ldap.telascience.org/foss4g/ FOSS4G2006 Lausanne/Switzerland ''GRASS Workshop LiveCD 2006'']<br />
* Others: http://grass.osgeo.org/download/cdrom.php<br />
<br />
6. Find some funding for printing the flyer / brochure<br />
<!--* German GRASS user group ([http://www.grass-verein.de GAV e.V.]) offers some funding for flyer--><br />
* How much is needed?<br />
<br />
''Depends on the quality & number of flyers we want to order''<br />
''I found one shop where we can get 2500 flyer for about 170 - 270 € (aprox. 225 - 355 US $. depending on paper/colors...)''<br />
* Find a way to spread the flyer<br />
<br />
== Later or synchronous == <br />
* Translation of the GRASS-Flyer (Latex-file available on the Grass-Addons SVN [http://trac.osgeo.org/grass/browser/grass-promo/grassflyer GRASS-ADDONS-SVN)<br />
* <strike>Newbie Forum</strike> (done)<br />
* [[Contact Databases]]<br />
* Case Histories of GRASS adoption (?)<br />
** [[Publications|Publications made with GRASS]]: on this page publications made with GRASS should be listened<br />
** [[Use Cases|Use Cases successful applications with GRASS]]<br />
* Nice Posters with applications where GRASS hass been adopted<br />
* <strike>workaround of [http://www.osgeo.org/grass http://www.osgeo.org/grass]</strike> ''done''<br />
* Maybe offer some stuff on coffeepress / spreadshirt<br />
** for Germany / Europe in progress (thx to Martin Wegmann)<br />
* Stickers<br />
* Proofreading<br />
<br />
= People (add yourself here) =<br />
* Malte Halbey-Martin (maltehalbey on irc): email malte [at] geog.fu-berlin.de<br />
* Ominiverdi ( doktoreas or ominoverde on irc ) : email info [at] ominiverdi.org<br />
* Chip Mefford (cpm on irc) : email cpm [at] daviswv.net or cpm [at] well.com<br />
* Carlos Grohmann: email carlos.grohmann [at] gmail.com<br />
* Milena Nowotarska (milenaN on irc) : email do.milenki [at] gmail.com<br />
<br />
[[Category:Promotion]]</div>⚠️Erget