MB-System: Difference between revisions

From GRASS-Wiki
Jump to navigation Jump to search
(export)
(fnv coverage example code)
Line 222: Line 222:
   
   
  d.what.vect datalist_tracks -f
  d.what.vect datalist_tracks -f
Comvert all-swath patch into a single polygon:
: (''fast & good enough method'')
This method generalizes the data by converting into a raster grid then back into a vector area.
<source lang="bash">
# display and set region & raster resolution
d.vect datalist_swaths type=area fcolor=none
d.zoom
g.region -a -p res=
# initialize base map
r.mapcalc "swath_coverage = 0"
# loop to systematically add a "1" every time a cell is
#  covered by a swath; highlights overlap areas.
for MAP in `g.mlist vect pattern="swath_*"` ; do
  echo "Adding <$MAP> ..."
  v.to.rast in="$MAP" use=val val=1 type=area out=tmp.$MAP.$$ --quiet
  r.series in=swath_coverage,tmp.$MAP.$$ method=sum out=tmp.series.$$ --quiet
  g.remove tmp.$MAP.$$,swath_coverage --quiet
  g.rename tmp.series.$$,swath_coverage --quiet
done
# flatten to a boolean coverage mask map
r.reclass in=swath_coverage out=swath_coverage.1 << EOF
  1 thru 999999 = 1
  0 = NULL
EOF
# convert back to a vector area polygon
r.to.vect -s -v in=swath_coverage.1 out=Swath_coverage feature=area
# remove tiny holes (single or a few cells)
v.clean in=Swath_coverage out=Swath_coverage_noHoles tool=rmarea thresh=10000
</source>
[[Image:Fnv_coverage.png|thumb|center|599px|Survey swaths imported with {{AddonCmd|v.in.mbsys_fnv}} and combined into a single coverage area. Track lines are overlain.]]
Clean all-swath patch into a single polygon:
: (''much slower but more exact method'')
<source lang="bash">
v.db.addcol map=datalist_swaths column='cat1 integer'
v.db.update map=datalist_swaths column=cat1 value=1
v.reclass in=datalist_swaths out=swaths1 column=cat1
v.db.dropcol map=datalist_swaths column=cat1
# this cleaning step may take a ''very'' long time
v.clean tool=break in=swaths1 out=swaths2
v.dissolve in=swaths2 out=datalist_swaths_coverage
g.remove vect=swaths1,swaths3
</source>





Revision as of 04:45, 11 January 2010

MB-System

  • MB-System is Free software for the processing and display of swath and sidescan sonar data. It can handle both multibeam bathymetry and sidescan sonar image data.

Import into GRASS

See the GRASS and GMT wiki help page for more information.
  • Ungridded data points may be piped directly from mblist to GRASS's v.in.ascii module.
d.vect's zcolor= option can be used to color by depth value.
See the v.colors addon script for colorizing point data in GRASS (v.colors may be unsuitable for massive datasets).
  • ".fnv" navigation files can be imported with the v.in.mbsys_fnv addon module in a number of different ways:
  1. track: ship's track
  2. port_trk: port-side outward track
  3. stbd_trk: starboard-side outward track
  4. scanlines: lines perpendicular to direction of travel
  5. swath: coverage area
  6. track_pts: ship's track as points
  7. all_pts: ship's track, port, and stbd track points
I think the swath area coverage is particularly neat.


Examples

Import x,y,z bathymetry into GRASS
Into a vector points map
  • Export Lat/Lon + depth data from XTF datafile into a GRASS Lat/Lon location
mblist -I 074.XTF -OXYz | v.in.ascii out=track074 x=1 y=2 fs=tab
  • Export Lat/Lon from the XTF datafile, reproject into the current GRASS location's projection, and import into GRASS with v.in.ascii
mblist -I 074.XTF -OXY | m.proj -i | cut -f1 -d' ' | \
  v.in.ascii out=track074 x=1 y=2 fs=tab

Idea: write a v.in.cdl script that will parse a NetCDF/CDL file and automatically set v.in.ascii's column= option with attribute table column names and types.

Into a gridded raster map
  • Read a series of .fbt pre-processed bathymetry files, reproject into the current projection, and grid into raster maps.

First create .fbt .fnv and .inf summary files

 find . | grep '.[xX][tT][fF]$' > tmplist
 mbdatalist -F-1 -I tmplist > datalist-1 
 mbdatalist -F-1 -I datalist-1 -N

Next create quick GMT plot of the nav lines to check coverage

 # scan multi-dir for bounds
 for lltype in Longitude Latitude ; do
   for tb in head tail ; do
     grep $lltype `find | grep '\.inf$'` | \
       awk '{print $3 "\n" $6}' | sort -n | $tb -n1
   done
 done
 EXTENT=167.68/169.09/40.48/40.82
 
 mbdatalist -F-1 -I datalist-1 -R$EXTENT > survey-datalist
 mbm_plot -F-1 -I survey-datalist -N
 ./survey-datalist.cmd

Based on the above PostScript file set GRASS region by eye:

#LL: n=40:48N s=40:37N w=168:40E e=169:06E

Convert to local projection with m.proj:

echo "168d40E 40d37N
169d06E 40d48N" | m.proj -i

Plug those numbers into g.region:

g.region s=4616845.02 n=4636836.33 w=520872.47 e=524862.14

Set the grid cell size to 5m, and align to whole numbers, and check that the rows x columns is reasonable (smaller than 40000x40000):

g.region res=5 -a -p

Adjust resolution to smaller grid size if needed (1000x1000 is fine for a summary image).

g.region res=10 -a -p

Scan, read, and reproject all .fbt data into a single x,y,z text file:

A datalist fed to mblist then piped through 'm.proj -i' and r.in.xyz to form a gridded DEM.
note that this file can get very large, perhaps too large for a 32bit OS/filesystem. In these cases you can pipe directly into r.in.xyz so no file is written to disk.
note that -Rw/e/s/n is used to skip over out-of-region data. MB-System and GMT will accept DDD:MM:SS.SSSh format as well as decimal degrees, just like GRASS.
The following assumes the output is projected, for import into WGS84 lat/lon skip the cs2cs command or at least change "%.3f" to "%.8f".
OUTPROJ="`g.proj -jf`"
( 
  for FILE in `find . | grep '\.fbt$'` ; do
    echo "Reading <$FILE> ..." 1>&2
    mblist -I$FILE -D3 -R168:40E/169:06E/40:37N/40:48N
  done 
) | cs2cs -f '%.3f' +init=epsg:4326 +to $OUTPROJ | \
   tr ' ' '\t' > fbt_UTM58_zoom.dat

Based on scan of the .inf files and personal knowledge set some reasonable bounds for the z-data:

# scan multi-dir for depth bounds
for tb in head tail ; do
   grep Depth `find | grep '\.inf$'` | \
      awk '{print $3 "\n" $6}' | sort -n | $tb -n1
done
#-2.7339
#200.6072

ZRANGE="-250,-2"

Run r.in.xyz to make aggregate raster maps.

r.in.xyz fs=tab x=1 x=2 z=3 zrange=$ZRANGE \
  in=zoom_fbt.dat out=zoom_fbt.n method=n

r.in.xyz fs=tab x=1 x=2 z=3 zrange=$ZRANGE \
  in=zoom_fbt.dat out=zoom_fbt.mean method=mean

r.in.xyz fs=tab x=1 x=2 z=3 zrange=$ZRANGE percent=25 \
  in=zoom_fbt.dat out=zoom_fbt.median method=median

r.in.xyz fs=tab x=1 x=2 z=3 zrange=$ZRANGE percent=25 \
  in=zoom_fbt.dat out=zoom_fbt.trim20 method=trimmean trim=20


Analyze "n" map for number of pings per square-meter.

no-data cells not randomly distributed so we set n=0 to NULL so we aren't biasing the data as much
r.null zoom_fbt.n setnull=0
r.univar zoom_fbt.n

"mean:" is number of pings per square-meter for cells which have data.

Now analyze bathymetry data:

r.univar zoom_fbt.median

and set some nice colors:

r.colors zoom_fbt.median color=bcyr

Finally display it:

d.mon x0
d.rast zoom_fbt.median
d.legend zoom_fbt.median
d.barscale at=5,5
Create vector maps from .fnv nav data
Add bounding box
MAP=swath_MBM04_007_0501_XTF

v.db.addcol $MAP column="north DOUBLE PRECISION, \
  south DOUBLE PRECISION, east DOUBLE PRECISION, west DOUBLE PRECISION"

eval `v.info -g $MAP`

echo "UPDATE $MAP SET north = $north, south = $south, \
  east = $east, west = $west" | db.execute
Import loop and patching

change DB to sqlite

db.connect driver=sqlite database='$GISDBASE/$LOCATION_NAME/$MAPSET/sqlite.db'


Import all files in the datalist using a loop:

for TYPE in track swath ; do
   for TRACK in `cut -f1 -d' ' datalist-1` ; do
      OUTNAME="${TYPE}_$( echo "`basename $TRACK .fnv`" | sed -e 's/[- .]/_/g' )"
      echo "Importing <$OUTNAME> ..."
 
      CAT=`echo "$OUTNAME" | cut -f3,4 -d'_' | tr -d '_'`
 
      v.in.mbsys_fnv in="${TRACK}.fnv" out="$OUTNAME" type=$TYPE cat=$CAT
   done
done


Patch together all maps with a loop:

for TYPE in track swath ; do

   # create an empty vector
   v.in.ascii -e out="datalist_${TYPE}s" --quiet

   # copy table structure
   BASE_MAP=`g.mlist vect pattern="${TYPE}_*" | head -n 1`
   exist_cols=`db.describe $BASE_MAP -c | grep '^Column' | cut -f2- -d: | \
     sed -e 's/:/ /' -e 's/CHARACTER:\(.*\)/VARCHAR(\1)/' | \
     cut -f1 -d: | tr '\n' ',' | sed -e 's/,$//' -e 's/^ //'`
   v.db.addtable "datalist_${TYPE}s" columns="$exist_cols"

   # run the patch loop
   for MAP in `g.mlist vect pattern="${TYPE}_*"` ; do
      echo "Patching <$MAP> ..."

      v.patch -a -e in="$MAP" out=datalist_${TYPE}s --overwrite --quiet

   done
done
  • trac #859: v.patch is failing to preserve our carefully selected category numbers.


View and interactive query:

d.mon start=x0
d.vect datalist_swaths type=area color=150:170:200 fcolor=200:220:250
d.vect datalist_tracks width=2

d.zoom

d.what.vect datalist_tracks -f


Comvert all-swath patch into a single polygon:

(fast & good enough method)

This method generalizes the data by converting into a raster grid then back into a vector area.

# display and set region & raster resolution
d.vect datalist_swaths type=area fcolor=none
d.zoom
g.region -a -p res=

# initialize base map
r.mapcalc "swath_coverage = 0"

# loop to systematically add a "1" every time a cell is
#   covered by a swath; highlights overlap areas.
for MAP in `g.mlist vect pattern="swath_*"` ; do
   echo "Adding <$MAP> ..."
   v.to.rast in="$MAP" use=val val=1 type=area out=tmp.$MAP.$$ --quiet
   r.series in=swath_coverage,tmp.$MAP.$$ method=sum out=tmp.series.$$ --quiet
   g.remove tmp.$MAP.$$,swath_coverage --quiet
   g.rename tmp.series.$$,swath_coverage --quiet
done

# flatten to a boolean coverage mask map
r.reclass in=swath_coverage out=swath_coverage.1 << EOF
  1 thru 999999 = 1
  0 = NULL
EOF

# convert back to a vector area polygon
r.to.vect -s -v in=swath_coverage.1 out=Swath_coverage feature=area

# remove tiny holes (single or a few cells)
v.clean in=Swath_coverage out=Swath_coverage_noHoles tool=rmarea thresh=10000


Survey swaths imported with v.in.mbsys_fnv and combined into a single coverage area. Track lines are overlain.


Clean all-swath patch into a single polygon:

(much slower but more exact method)
v.db.addcol map=datalist_swaths column='cat1 integer'
v.db.update map=datalist_swaths column=cat1 value=1
v.reclass in=datalist_swaths out=swaths1 column=cat1
v.db.dropcol map=datalist_swaths column=cat1 
# this cleaning step may take a ''very'' long time
v.clean tool=break in=swaths1 out=swaths2
v.dissolve in=swaths2 out=datalist_swaths_coverage
g.remove vect=swaths1,swaths3


Export to GMT

Processed raster grids and vector areas, lines, and points can be exported to back to GMT for plotting. See the GRASS and GMT wiki page for more information.