LIDAR

From GRASS-Wiki
Revision as of 12:00, 7 February 2012 by ⚠️HamishBowman (talk | contribs) (→‎Import LAS as raster DEM: las2txt changed its bloody delimiter)
Jump to navigation Jump to search

LIDAR and Multi-beam Swath bathymetry data

Modules

Import

  • r.in.xyz - Create a raster map from an assemblage of many coordinates using univariate statistics. (example)
  • v.in.ascii - Import data from an ASCII file to GRASS vector format.
Due to memory overhead vector point imports will be limited to a few million data points unless topology and database creation is skipped with the -bt flags. It may also be useful to clip the import file to only accept points falling within the current region by using the -r flag. See g.region for details on specifying the region bounds.

Analysis

  • v.outlier - Removes outliers from vector point data.
  • v.lidar.growing - Building contour determination and Region Growing algorithm for determining the building inside.
  • v.lidar.correction - Correction of the v.lidar.growing output. It is the last of the three algorithms for LIDAR filtering.

References:

  • Detailed description: here (FOSS4G 2006)
  • Summarised version: here (2010)
  • Calibration of the filtering parameters (around 20 parameters) by integrating the USGS UCODE and GRASS, see here

Surface generation

  • v.surf.rst - Spatial approximation and topographic analysis using regularized spline with tension.
  • v.surf.idw - Surface interpolation from vector point data by Inverse Distance Squared Weighting.
  • v.surf.bspline - Surface interpolation from vector point data by bicubic or bilineal interpolation with Tykhonov regularization.
  • r.fillnulls - Fills no-data areas in raster maps using v.surf.rst splines interpolation.

Swath Bathymetry Tools

see also the Marine_Science#Multibeam_sonar_processing wiki page

  • The v.swathwidth module by David Finlayson for planning surveys. (development page)

LIDAR Tools

  • r.terraflow - computation of flow direction, flow accumulation and other basic topographic terrain indices from massive raster digital elevation models (DEM). From the Duke University STREAM project.
  • LAStools are a set of simple command line tools for converting to/from ASCII, viewing, comparing, and compressing LIDAR data. While free to use source code is available for older verions, newer versions are not open source, only work on MS Windows, and are no longer free for commercial or government use.
  • libLAS ASPRS LiDAR data translation tools

Micro-tutorial for LAS data import

The following scripts are given for UNIX Bourne Shell; MS-Windows users should use the Msys terminal to use them.

Preparation

Conversion of text files to LAS

libLAS supports the following column types: 

  x - x coordinate
  y - y coordinate
  z - z coordinate
  a - scan angle
  i - intensity
  n - number of returns for given pulse (1..n)
  r - number of this return (1..r)
  c - classification
  u - user data (but only 1 byte)
  p - point source ID
  e - edge of flight line
  d - direction of scan flag
  t - GPS time
  s - skip column

Sample text data such as: 

  returntime,pulse,east,north,height,intensity,stripe
  549778.907200,1,673999.940,5099680.080,507.425,20.0,45105
  ...

can be converted to LAS format like this: 

  returntime - t
  pulse - r
  east - x
  north - y
  height - z
  intensity - i
  stripe - s

First parse (sanity check): 

  txt2las -parse trxyzis data.asc

Then convert: 

  txt2las -parse trxyzis -i data.asc -o data.las

While you may not need to do this conversion for GRASS import, the resulting files are way smaller than the uncompressed text files; additionally, they are in a defined format.

LAS Import

  • Use the libLAS utilities to import LAS data into GRASS.
  • Data stored in text files can generally be imported directly into GRASS.

Import LAS as raster DEM

Check bounds and SRS: 

$ lasinfo "Serpent_Mound_Model_LAS_Data.las"

 [...]
 Min X Y Z                  289020.900000 4320942.610000 166.780000
 Max X Y Z                  290106.020000 4323641.570000 215.480000
Spatial Reference           +proj=utm +zone=17 +ellps=WGS84 +units=m

After creating a suitable UTM zone 17 location (EPSG:32617) set the region according to the information from lasinfo at 1m resolution, rounding grid outwards to align to whole meters: 

GRASS> g.region n=4323641.57 s=4320942.61 w=289020.90 e=290106.02 res=1 -ap

Finally, import with r.in.xyz with data piped directly from the las2txt program and set a nice equalized color table: 

BASEMAP="Serpent_Mound_Model_LAS"

las2txt --stdout --delimiter ' ' "${BASEMAP}_Data.las" | \
   r.in.xyz in=- out=${BASEMAP}_Data fs=space method=mean

r.colors ${BASEMAP}_Data color=bcyr -e

The above example uses the default z-elevation level as the 3rd term, but by using the las2txt --parse command other fields (such as intensity) can be imported instead via "--parse xyi".

Import LAS as vector points

Region setting (establishing the grid) is not needed for vector features so we can go directly to the import step. To deal with millions of input points v.in.ascii should be run with the options to skip creation of an attribute database and building topology as these can consume large amounts of memory. Note that vector maps without topology built are somewhat limited in their ability to be processed. Most LIDAR specific modules have been adapted to not require topology. Even so, after initial cleaning steps it is often more efficient to work with huge datasets in GRASS as raster data. 

las2txt --stdout "${BASEMAP}_Data.las" | \
  v.in.ascii -tbz z=3 out="${BASEMAP}_pts" fs=space

If topology was built, you can use d.vect's -z flag to colorize by elevation value. Without topology you can still colorize, but you need to use color rules based on absolute elevations, not percentage of scale. 

# display colorized points for data with built topology
d.vect map=lidar_pts size=1 -z zcolor=elevation

Clean imported raster DEM

(fill holes) 

# convert to vector points
r.to.vect -z feature=point in=${BASEMAP}_Data out=${BASEMAP}_pt

# interpolate using a regularized spline fit
# this is very slow, but produces very high quality output
v.surf.rst layer=0 in=${BASEMAP}_pt elev=${BASEMAP}.rst

# create 5m buffer area around original data points
r.buffer in=${BASEMAP}_Data out=${BASEMAP}.5m_buff dist=5

# crop interpolated DEM to only include areas nearby actual data
r.mapcalc "${BASEMAP}.filled = \
   if( isnull(${BASEMAP}.5m_buff), null(), ${BASEMAP}.rst)"

# set colors to something nice
r.colors ${BASEMAP}.filled color=bcyr -e


Depending on your needs, r.fillnulls, v.surf.bspline, v.surf.idw, r.surf.idw, r.surf.idw2, or r.surf.nnbathy may be faster than the v.surf.rst method.

Visualize raster DEM in 3D

nviz ${BASEMAP}.filled
  • Set z-exag to 2.0
  • In Visualize → Raster Surfaces set the fine (final) resolution to 1, and coarse (preview) resolution to 5.
  • Set the height to 500.0, the perspective to 15.0, and drag the view-puck to the North-West and reasonably zoomed in.
You should now be able to see the serpent:
The Great Serpent Mound, Adams County, Ohio, USA


It is possible to show vector points in 3D, but millions of them may make the program run slow. Topology is required (v.build). Tick the "3D" box in the Visualize → Vector points dialog.

LAS Export

Export to LAS

This is the reverse of the import step, but using v.out.ascii or r.out.xyz with txt2las. Because the v.out.ascii module exports category number which we are not interested in, we cut it away with the UNIX cut utility. 

v.out.ascii ${BASEMAP}_pts fs=space | cut -f1-3 -d' ' \
    > ${BASEMAP}_export.txt

txt2las --parse xyz -i ${BASEMAP}_export.txt

Micro-tutorial for LIDAR data analysis

Simple analysis of single return data

Data source: North Carolina sample data set (Lidar data from 2001, near Raleigh, North Carolina). Download from 

  http://www.grassbook.org/ncexternal/
  -> File: BE3720079200WC20020829m.txt 3.6M (lidar bare earth points [spm])

Scan input file for spatial extent. The -g flag shows the result in a convenient copy-paste format for g.region: 

  r.in.xyz BE3720079200WC20020829m.txt out=dummy -s -g

We use this output to set region, predefine 2m raster cells, and polish the odd coordinates with -a: 

  g.region n=222504.439000 s=219456.442000 e=640081.274000 w=637033.274000 res=2 -a -p

QUESTION 1: Are these Lidar data sufficiently dense?

We compute a raster map representing number of points per cell 

  r.in.xyz BE3720079200WC20020829m.txt out=lid_792_binn2m method=n

Look at the resulting raster map: 

  d.mon x0
  d.font Vera
  d.rast.leg lid_792_binn2m

Report point distribution in percent: 

  r.report lid_792_binn2m unit=p
  r.univar lid_792_binn2m
  # (sometimes do `r.null setnull=0` first)

Reduce the resolution to 6m to get at least one point per cell: 

  g.region res=6 -a -p
  r.in.xyz BE3720079200WC20020829m.txt out=lid_792_binn6m method=n
  d.rast.leg lid_792_binn6m
  # ... a few holes remain but that's probably acceptable.

Compute raster maps representing mean elevation for each cell: 

  r.in.xyz BE3720079200WC20020829m.txt out=lid_792_binmean6m  \
           meth=mean
  d.rast.leg lid_792_binmean6m

Compute range and variation: 

  r.in.xyz BE3720079200WC20020829m.txt out=lid_792_binrange6m \
           meth=range
  d.rast.leg lid_792_binrange6m
  r.in.xyz BE3720079200WC20020829m.txt out=lid_792_binvar6m   \
           meth=coeff_var
  d.rast.leg lid_792_binvar6m

Overlay other GIS maps to map of mean elevation: 

  d.rast.leg lid_792_binmean6m
  d.vect streets_wake
  d.vect lakes type=boundary
  d.vect streams

Now we continue to work in a zoomed spatial subset of the area.

We import only points in the rural area without building topology and using z-coordinate for elevation: 

  g.region rural_1m -p
  v.in.ascii -ztbr BE3720079200WC20020829m.txt \
             out=elev_lidrural_bepts z=3

Clear monitor show black-white orthophoto: 

  d.erase -f
  d.rast ortho_2001_t792_1m

We now use already prepared Lidar vector point maps:

  • elev_lid792_bepts: Rural area (in tile 792), bare earth lidar point cloud
  • elev_lidrural_mrpts: Rural area multiple return lidar point cloud

Look at ground data: d.vect elev_lidrural_bepts siz=2 col=red

Look at "surface" data: d.vect elev_lidrural_mrpts siz=1 col=green

Visualize 3D Lidar multi-return points: nviz elev_lid792_1m point=elev_lidrural_mrpts 

# Visualize -> Vector Points -> 3D points
#                            -> Icon size 2.25
#  --> GRASS book p. 253, fig. 3.15 (see also screenshot here)

DEM/DSM separation the simple way by selection of Lidar returns

Find out where we have multiple returns: 

  d.rast ortho_2001_t792_1m
  d.vect elev_lidrural_mrpts where="return=1" col=red siz=2
  d.vect elev_lidrural_mrpts where="return=2" col=green siz=3
  d.vect elev_lidrural_mrpts where="return=3" col=blue
  d.vect elev_lidrural_mrpts where="return=4" col=yellow

DTM: extract last return(s): 

  v.extract elev_lidrural_mrpts out=elev_lidrural_mrpts_first where="Return < 2"
  nviz elev_lid792_1m point=elev_lidrural_mrpts_first

Interpolate to map and look at it: 

  v.surf.rst elev_lidrural_mrpts_first layer=0 elev=elev_lidrural_mrpts_DTM
  nviz elev_lidrural_mrpts_DTM col=ortho_2001_t792_1m

DSM: extract first return(s): 

  v.extract elev_lidrural_mrpts out=elev_lidrural_mrpts_last where="Return > 2"
  nviz elev_lid792_1m point=elev_lidrural_mrpts_last

Interpolate to map and look at it: 

  v.surf.rst elev_lidrural_mrpts_first layer=0 elev=elev_lidrural_mrpts_DSM
  nviz elev_lidrural_mrpts_DSM col=ortho_2001_t792_1m


DEM/DSM separation the more complex way

TODO: verify order of first and last returns in below text

General procedure:

  • Lidar point clouds (first and last return) are imported with v.in.ascii (-b flag to not build the topology).
  • Outlier detection is done with v.outlier on both first and last return data (NB currently a selection of </=4 for soe/son in v.outlier results in an error message).
  • Then, with v.lidar.edgedetection, edges are detected from last return data.
  • The building are generated by v.lidar.growing from detected edges.
  • The resulting data are post-processed with v.lidar.correction.
  • Finally, the DTM and DSM are generated with v.surf.bspline (DTM: uses the 'v.lidar.correction' output; DSM: uses last return output from outlier detection).
  • NB for v.outlier, v.lidar.edgedetection and v.surf.bspline, one spline steps equates to 1m. It is recommended as a starting point that the choice of spline step is roughly 3 or 4 times the planimetric resolution (potential grid resolution) of your data. Experiment from there to obtain better results.
 export GRASS_OVERWRITE=1
 v.extract elev_lidrural_mrpts out=elev_lidfirst_pts \
           where="return = 1"
 v.extract elev_lidrural_mrpts out=elev_lidlast_pts  \
           where="return >= 2"
 d.vect elev_lidfirst_pts col=red
 d.vect elev_lidlast_pts col=green

Outlier detection and separation into two maps 

 # 1st return
 v.outlier elev_lidfirst_pts output=elev_lidfirst_clean \
           outlier=elev_lidfirst_outl
 d.erase
 d.vect elev_lidfirst_clean siz=2
 d.vect elev_lidfirst_outl col=red

2nd return 

 v.outlier elev_lidlast_pts output=elev_lidlast_clean \
           outlier=elev_lidlast_outl
 d.erase
 d.vect elev_lidlast_clean siz=2
 d.vect elev_lidlast_outl col=red
 # -> no outliers visible

Run an edge detection on cleaned last return: 

 v.lidar.edgedetection elev_lidlast_clean \
                       out=elev_lidlast_edges

Buildings/vegetation are generated from detected edges (bug: you may need to specify the mapset): 

 v.lidar.growing elev_lidlast_edges@lidar out=elev_lidlast_grow \
                 first=elev_lidfirst_clean

Compare: 

 d.vect elev_lidlast_pts col=blue
 d.vect elev_lidlast_grow col=green

Correction (this is applied twice): 

 v.lidar.correction elev_lidlast_grow out=elev_lidlast_corr1 \
                    terrain=elev_lidlast_terr1
 v.lidar.correction elev_lidlast_corr1 out=elev_lid_dsm \
                    terrain=elev_lid_dtm

DEM and DSM are generated: 

 # Estimation of lambda_i parameter with cross validation (watch for RMS!)
 # and note use of bicubic for DSM and bilinear for DTM here and below
 v.surf.bspline -c elev_lid_dsm sie=100 sin=100 method=bicubic
 v.surf.bspline -c elev_lid_dtm sie=100 sin=100 method=bilinear
 
 # From the cross-validation, we select lambda with minimal RMS error:
 # generate raster surfaces at 1m resolution 
 v.surf.bspline elev_lid_dsm raster=lidar_dsm lambda=0.1 method=bicubic
 v.surf.bspline elev_lid_dtm raster=lidar_dtm lambda=0.0001 method=bilinear
 
 d.rast lidar_dsm
 d.rast lidar_dtm
 
 nviz lidar_dsm,lidar_dtm \
      col=ortho_2001_t792_1m,ortho_2001_t792_1m

...with the position slider you can visually separate DSM and DEM, increase z slider (Visualize -> Raster Surface -> Position).

Sample LIDAR data

Widely used in GRASS tutorials

Other

  • USGS Center for LIDAR Information Coordination and Knowledge (aka CLICK) - USGS LiDAR point cloud distribution site
    http://lidar.cr.usgs.gov
  • EarthScope Spatial Data Explorer - A java application for querying, browsing, and acquiring data from the EarthScope Spatial Data Repository. Currently includes a number of LiDAR datasets.
    http://www.earthscope.org/data/lidar.php
  • libLAS's sample file collection
http://liblas.org/samples/
  • NASA's Laser Vegetation Imaging Sensor (a.k.a. the Land, Vegetation, and Ice Sensor) or "LVIS"
https://lvis.gsfc.nasa.gov

Links

Breaklines
  • Not very well developed in GRASS so far, but we'd like to change that
  • v.surf.icw (only suitable for ~ 200 input points)
  • SurfIt (GPL, Tck/Tk)
  • SurGe (Trial/shareware)

Related publications

  • Brovelli, M.A., Lucca, S., 2011. Filtering LiDAR with GRASS: overview of the method and comparisons with Terrascan. Italian Journal of Remote Sensing 93-105. (PDF)
  • Brovelli, M.A., Cannata, M. & Longoni, U.M., 2004. LIDAR data filtering and DTM interpolation within GRASS. Transactions in GIS, 8(2), pp.155-174. PDF
  • Cebecauer, T., Hofierka, J. & Suri, M., 2002. Processing digital terrain models by regularized spline with tension: tuning interpolation parameters for different input datasets. In Proc. of the Open Source Free Software GIS -- GRASS users conference 2002, Trento, Italy, 11-13 September 2002. PDF
  • Mitasova, Helena et al., 2009. Raster-based analysis of coastal terrain dynamics from multitemporal Lidar data. Journal of Coastal Research, 25(2), pp.507-514.
  • Mitasova, H, Mitas, L. & Harmon, R., 2005. Simultaneous spline approximation and topographic analysis for lidar elevation data in open-source GIS. Geoscience and Remote Sensing Letters, IEEE, 2(4), pp.379, 375. PDF
  • Mitasova, H. et al., 2003. Spatio-temporal analysis of beach morphology using LIDAR, RTK-GPS and Open source GRASS GIS. In Proceedings Coastal Sediments. PDF

Search via Google Scholar

See also

Retrieved from "https://grasswiki.osgeo.org/w/index.php?title=LIDAR&oldid=14832"