# Difference between revisions of "Limiting precision in vector nodes"

HamishBowman (talk | contribs) (howto) |
HamishBowman (talk | contribs) m (minor wording) |
||

Line 1: | Line 1: | ||

There are specifications (e.g. the Greek Cadastral System) which ask the digitizer to create their data with (for example) three digits behind the decimal place. As a general rule for the overwhelming majority of modern computers, exact representation of IEEE floating point numbers is not guaranteed in binary data storage (see links below), but if it's what the spec says it's what has to be attempted anyway. | There are specifications (e.g. the Greek Cadastral System) which ask the digitizer to create their data with (for example) three digits behind the decimal place. As a general rule for the overwhelming majority of modern computers, exact representation of IEEE floating point numbers is not guaranteed in binary data storage (see links below), but if it's what the spec says it's what has to be attempted anyway. | ||

− | Here is an example using the Spearfish sample dataset in GRASS 6.4 to reduce vector | + | Here is an example using the Spearfish sample dataset in GRASS 6.4 to reduce the node coordinates of vector polygons (''areas'') to three decimal digit precision: |

v.out.ascii in=quads format=standard dp=3 | v.in.ascii out=quads_dp3 format=standard | v.out.ascii in=quads format=standard dp=3 | v.in.ascii out=quads_dp3 format=standard |

## Revision as of 18:07, 11 July 2013

There are specifications (e.g. the Greek Cadastral System) which ask the digitizer to create their data with (for example) three digits behind the decimal place. As a general rule for the overwhelming majority of modern computers, exact representation of IEEE floating point numbers is not guaranteed in binary data storage (see links below), but if it's what the spec says it's what has to be attempted anyway.

Here is an example using the Spearfish sample dataset in GRASS 6.4 to reduce the node coordinates of vector polygons (*areas*) to three decimal digit precision:

v.out.ascii in=quads format=standard dp=3 | v.in.ascii out=quads_dp3 format=standard

Topology *should* be preserved, but it is worth paying close attention to the verification and topology building messages upon import. (in case of errors make sure that the original map did not have them too)

## See also

- Tricks and traps with floating point and decimal numbers by Brian Goetz
- What Every Computer Scientist Should Know About Floating-Point Arithmetic by David Goldberg