Neteler: + external links

2009-05-12T09:06:37Z

+ external links

← Older revision		Revision as of 09:06, 12 May 2009
Line 140:		Line 140:
	pages.		pages.

			== See also ==

			* [http://tldp.org/LDP/abs/html/textproc.html Text Processing Commands]
			* [http://www.gnu.org/software/coreutils/manual/html_node/index.html GNU Coreutils]

			[[Category:Documentation]]
	[[Category: FAQ]]		[[Category: FAQ]]

Neteler: Using GNU text tools for GIS data preparation

2007-07-21T12:10:32Z

Using GNU text tools for GIS data preparation

New page

'''Using GNU text tools for GIS data preparation'''

The GNU text tools cat, cut, join, head, more, paste, sed and tail and the
awk ("pattern scanning and processing language") provide a range of possibilities
to modify ASCII texts and tables. Often attribute tables are delivered
in ASCII formats such as CSV (Comma Separated Values format) or blank
delimited text. Especially in scripts the tools introduced here are quite helpful
to automate text formatting.
Here we show modifications of the Spearfish soils map
legend which is available at GRASS Web site and already included in the
Spearfish sample data set (see
/usr/local/share/grassdata/spearfish/soils_legend.txt). This legend is
an ASCII table, which contains further attributes for the soils map. We want
to show how to modify this table to a reclass rules file applicable to v.reclass
and r.reclass. First let's have a look into the file:

more soils_legend.txt

Within the more program continue to a next page with <SPACE>, quit with
<q>, search for a phrase with </>. Above file may look like this:

0:no data:
1:AaB:Alice fine sandy loam, 0 to 6
2:Ba:Barnum silt loam
3:Bb:Barnum silt loam, channeled
4:BcB:Boneek silt loam, 2 to 6
5:BcC:Boneek silt loam, 6 to 9
6:BeE:Butche stony loam, 6 to 50
[...]

The legend columns are separated by ":". In the first column the category
numbers (attribute IDs) are stored. In the second column the first letter always
capital is the initial letter of the soil name. The second letter is a capital if
the mapping unit is broadly defined; otherwise, it is a small letter. The third
letter, always a capital, A, B, C, D, E or F, indicates the slope. Symbols
without slope letter are those of mapping units that do not have slope as part
of the name. In the third column the full name of the soil is written along
with the typical slope.

First we want to reduce the legend to attribute ID, soil name initials and
text attribute without the slope information (this may go into another table
or derived from the Spearfish elevation.dem). Note that we proceed step-bystep
although you can also compose the commands to a few (or even a single)
lines. The cut tool cuts column-wise depending on the specified delimiter. We
specify delimiter : and select the first column (with field parameter f), then
pipe the result into a new file:

cut -d',' -f1 soils_legend.txt > soils_legend2.txt

Checking the new file with more shows us:

0:no data:
1:AaB:Alice fine sandy loam
2:Ba:Barnum silt loam
3:Bb:Barnum silt loam
4:BcB:Boneek silt loam
5:BcC:Boneek silt loam
6:BeE:Butche stony loam
[...]

Starting from the new file we will select only the text label:

cut -d':' -f3 soils_legend2.txt > soils_legendlabels.txt

Checking the new file with more shows us:

Alice fine sandy loam
Barnum silt loam
Barnum silt loam
Boneek silt loam
Boneek silt loam
Butche stony loam
[...]

Note that the first line is empty since the no-data field doesn't contain a text
label. Alternatively you can compose above steps to one command:

cut -d',' -f1 soils_legend.txt | cut -d':' -f3 > soils_legendlabels.txt

Now further hints: In case you want to cut a column at a specific position,
you can use the -b flag:

cut -b1,2 soils_legend.txt
0:n
1:A
2:B
3:B
4:B
5:B
6:B
[...]

If you want to see only the first lines of a file, use head. The number of lines
to be displayed has to be entered with a preceding minus character:

head -4 soils_legend.txt

The result looks as follows:

0:no data:
1:AaB:Alice fine sandy loam, 0 to 6
2:Ba:Barnum silt loam
3:Bb:Barnum silt loam, channeled

If you want to see only the last lines, use tail. It is used similar to head:

tail -3 soils_legend.txt

leading to the output:

53:WaA:Weber loam, 0 to 2
54:Wb:Winetti cobbly loam
55:water

Both may be combined to show a portion of the text file (here only lines 3 to 5):

head -5 soils_legend.txt | tail -3

Here the head command shows the first five lines, the tail command the last
three of these five lines:

2:Ba:Barnum silt loam
3:Bb:Barnum silt loam, channeled
4:BcB:Boneek silt loam, 2 to 6

In order to sequentially concatenate two files, use cat:

cat file1 file2 > file1and2

If you need to paste two files column-wise, use paste. You can optionally
change the column delimiter from the default tabulator to another character.
The command join is allowing to work similar to a simple database management
system - it joins together ASCII tables according to unique column
entries.

A powerful string editor is sed which allows to exchange, add or cut off
strings from text files by rule definitions.
With awk which we already used throughout the book, you can perform
calculations or formatted printing. For details please refer to the related manual
pages.

[[Category: FAQ]]

GNU text tools - Revision history

Neteler: + external links

Neteler: Using GNU text tools for GIS data preparation