Monday, August 3, 2015

Tools for Visualizing Lidar Data I


LidarViewer


A friend of mine recently sent me a link to an article on opensource.com titled   Manipulating data in 3D with LidarViewer.  His timing couldn't have been better, since LidarViewer is a free software tool for visualizing Lidar data, and is thus just the kind of thing I need for my Summer project.  Even better, the downloads page starts with an Ubuntu PPA, so installation should not be a nightmare.

I'll do my installation on a VirtualBox VM running Lubuntu 14.04.  I like to use VMs whenever I am trying new software that is not part of the standard Ubuntu software repositories, since this keeps my host machine stable, while letting me experiment without fear.

Here is what I did:
$ sudo apt-add-repository ppa:keckcaves/ppa
$ sudo aptitude update
The installation page doesn't list the packages included in the PPA, but the Launchpad page for the repo does.  There is a package named lidarviewer, so I'll install that:
$ sudo aptitude install lidarviewer
Running lidarviewer at the command prompt after installation completed gave me a "command not found", so I checked to see what the package had installed:
$ dpkg -L lidarviewer
/.
/etc
/etc/LidarViewer
/etc/LidarViewer/LidarViewer.cfg
/usr
/usr/bin
/usr/bin/LidarIlluminator
/usr/bin/PaulBunyan
/usr/bin/LidarPreprocessor
/usr/bin/LidarSubtractor
/usr/bin/LidarGridder
/usr/bin/CalcLasRange
/usr/bin/LidarExporter
/usr/bin/PointSetSimilarity
/usr/bin/PrintPrimitiveFile
/usr/bin/LidarViewer
/usr/bin/LidarColorMapper
/usr/bin/LidarSpotRemover
/usr/share
/usr/share/doc
/usr/share/doc/lidarviewer
/usr/share/doc/lidarviewer/changelog.Debian.gz
/usr/share/doc/lidarviewer/copyright
/usr/share/doc/lidarviewer/HISTORY.gz
/usr/share/doc/lidarviewer/README.gz

$
I tried:
$ LidarViewer
Caught exception LidarViewer::LidarViewer: No octree file name provided
A quick search on the exception led me to the Lidar Viewer Manual.  Since I already installed the application from the Ubuntu PPA (Ubuntu rocks!), I can skip most of the installation instruction section. In the MacOS instructions, however, I found sample data for testing the application.  Downloading and unzipping the sample data, I changed into the LidarViewerExamples directory and ran the following command and got the following error:
$ LidarViewer PtArena.lidar
Cache sizes: 4672 memory nodes, 1170 GPU nodes
libGL error: pci id for fd 12: 80ee:beef, driver (null)
OpenGL Warning: Failed to connect to host. Make sure 3D acceleration is enabled for this VM.
libGL error: core dri or dri2 extension not found
libGL error: failed to load driver: vboxvideo
I'm using a VirtualBox VM, and this message is telling me to enable 3D acceleration.  After enabling 3D acceleration on the VM, the application reported a long list of OpenGL errors.  I found this bug ticket showing I'm not the only one with the issue.

I added the same PPA to a laptop running Ubuntu 14.04 (thus loosing the safety of the virtual machine) and installed both the lidarviewer and  vrui-examples packages, after which I could run the examples on the laptop without error.

lidar2dems


Next I'm going to install another set of tools for visualizing and processing Lidar data, lidar2dems.  Installation instructions are found here, and contain a number of utilities such as LAStools, which I'll need to uncompress the LAZ files that are on the Virginia Lidar website.

Installation of the lidar2dems software is made easy by an installation script, easy-install.sh.  After downloading the script, run:
$ chmod +x easy-install.sh
$ sudo ./easy-install.sh
The script took almost two hours to complete on my VirtualBox VM, but it completed without error.

It did not, however, install many of the LAStools utilities, especially laszip, as I had hoped.

laszip


To get laszip, I went to http://www.laszip.org and downloaded laszip.zip.  Then:
$ unzip laszip.zip
 which created a LAStools directory with several subdirectories, including a bin subdirectory that contained windows .exe binaries and _README.txt text files for many LASzip utilities, including laszip.exe.  Next I:
$ cd LAStools
$ make
This created the following Linux binaries in the bin directory:
las2las  lasdiff   lasinfo   lasprecision  txt2las
las2txt  lasindex  lasmerge  laszip
To test if laszip works, I grabbed a LAZ file for downtown Leesburg:
$ wget https://703348910b61e15b5d68b83128b735c09c3849cb.googledrive.com/host/0B_5XlZJJ2R5tUjZFb1FQTnU1alk/18STJ7733.laz
This got me the file 18STJ7733.laz. Then I ran:
$ ls -l
total 16624
-rw-rw-r-- 1 user user 17019618 Aug 17  2012 18STJ7733.laz
$ laszip 18STJ7733.laz
$ ls -l
total 149256
-rw-rw-r-- 1 jelkner jelkner 135810762 Aug  3 12:05 18STJ7733.las
-rw-rw-r-- 1 jelkner jelkner  17019618 Aug 17  2012 18STJ7733.laz
So it appears to have uncompressed the LAZ file into a much larger (almost 8x larger) LAS file.

Given that it is free software with an LGPL license, I don't understand why someone in the Open Source GIS community hasn't made this installation much simpler on Ubuntu yet.  For now, I've made a modest contribution toward that goal by creating a page with the Ubuntu binaries on my Open Book Project site:
http://openbookproject.net/resources/laszip.php

Friday, July 31, 2015

Manual of Airborne Topographic Lidar - Chapter 2 Acronym Glossary

One of the most challenging things about reading this text is the incredible number of acronyms used.  I created the following glossary of acronyms to enable me to make sense of what I read in chapter two.

Glossary of Acronyms

  • ALS - Airborne Laser Scanning
  • AM - Amplitude modulation 
  • AOBD - Acousto-optic beam deflector
  • AOL - Airborne Oceanic Lidar
  • AOI - Angle of incidence
  • APD - Avalanche photodiode
  • ARD - Automatic rendezvous and docking 
  • ARV - Autonomous robotic vision
  • ASC - Advanced Scientific Concepts (company) 
  • ASPRS -  American Society for Photogrammetry and Remote Sensing
  • ATM - Airborne Topographic Mapper 
  • ATR - Automatic target recognition 
  • BATC - Ball Aerospace and Technologies Corporation
  • CCD - Charge-coupled device 
  • CIR - Color-infrared
  • CMOS - Complementary metal–oxide–semiconductor
  • COTS - Commercial off-the-shelf
  • CW - Continuous wave
  • DEM - Digital elevation model 
  • DESDynI - Deformation, Ecosystem Structure, and Dynamics of Ice
  • DHS SBIR - Department of Homeland Security Small Business Innovation Research program
  • DSM - Digital surface model
  • DSP - Digital signal processing
  • DTM - Digital terrain model
  • EDME - Electronic Distance Measuring Equipment
  • ESFL - Electronically Steerable Flash Lidar
  • FDC - Frequency-to-distance conversion
  • FFPA - Flash-focal-plane-array
  • FM - Frequency modulation
  • FOR - Field of regard
  • FOV - Field of view
  • FPGA - Field programmable gate array
  • FW - Full waveform
  • FWHM - Full width half maximum
  • GmAPD - Geiger Mode Avalanche Photodiode
  • GPS - Global positing system
  • HLDA - Autonomous hazard detection and avoidance 
  • ICP - Iterative closest point
  • IFOV - Instantaneous field of view 
  • IIP - Instrument Incubator Program
  • IMU - Inertial measurement unit
  • InGaAs - Indium-gallium-arsenide
  • INS - Inertial navigation system 
  • ISD - Integrated scanner, detector
  • ISDT - Integrated scanner, detector, telescope
  • ISR - Intelligence, Surveillance, and Reconnaissance
  • ISS - International space station
  • LAPF - Laser Altimeter Processing Facility 
  • LAS -  Laser (file format) developed by ASPRS
  • LA-WASM -  large-aperture wide-angle scanning mirror
  • LIF - Laser induced florescence 
  • LINS - Laser Inertial Measurement system
  • LRF - Laser range finder
  • LVIS - Laser Vegetation Imaging Sensor
  • MASER - Microwave amplification by stimulated emission of radiation
  • MIR - Mid range infrared
  • NASA - National Aeronautics and Space Administration
  • NAVOCEANO - Naval Oceanographic Office
  • NOAA - National Oceanic and Atmospheric Administration
  • OMCMFNC - Organic Mine Countermeasures Future Naval Capabilities
  • PRF - Pulse repetition frequency
  • PROXOPS - Proximity operations
  • PRR - Pulse repetition rate 
  • RAMS - Real-time Aerial Mapping System
  • RASCAL - RAster SCanning Airborne Laser Altimeter 
  • RF - Radio frequency
  • RGB - Red-green-blue
  • ROAR - Rapid Overt Airborne Reconnaissance
  • ROIC - Readout integrated circuit
  • RVS - Raytheon Vision Systems (company)
  • SLICER - Scanning Lidar Imager of Canopies by Echo Recovery
  • SNR - Signal-to-noise ratio 
  • SWaP - Size, weight, and power
  • SWIR - Short wave infrared
  • TMFL - Topographic mapping flash lidar
  • TOF - Time-of-flight
  • TRN - Terrain-relative navigation 
  • UAV - Unmanned aerial vehicles 
  • UPS - Uninterruptabke Power Supply
  • UV - Ultraviolet
  • VIS/IR - Visible / infrared
  • VTUAV - Vertical takeoff unmanned aerial vehicle

Additional Resources

Tuesday, July 21, 2015

NodeSchool

In a previous post I described how I setup a virtual machine and installed Node.js on it.  The goal was to help a group of Summer student interns get started with both Node.js and Firefox OS app development by working on a web application created by a previous student intern.

That plan didn't work out well, as the application in question did not provide the kind of on ramp to the technologies we wanted to learn that I had hoped it would. Since Node.js is new to both the student interns and me, we all decided to look at the tutorials on NodeSchool, particularly learnyounode.

With node, and thus npm already installed, I added the learnyounode materials to my virtual machine with:
$ sudo npm install -g learnyounode
Running
$ learnyounode
brings up a menu:
with links to instructions for each of the 13 exercises in the tutorial.

I spent the morning working through the first 6 exercises, and created a github repository with my solutions.  I'm setting myself the additional goal of learning to write good clean JavaScript while I'm at it, so I installed JSLint:
$ sudo npm install -g jslint
And made every effort to minimize the number of warnings jslint reports on each of my solutions.  Using JSLint is almost like having an automated Douglas Crockford to look over your coding style, so I feel I'm in pretty good hands as I learn.

Learnyounode is a fabulous tutorial.  It moves you quickly and efficiently toward learning key ideas of important Node.js programming concepts by having you complete a series of well designed exercises.

I'm out of time for today, but the main concept I think I have finally begun to wrap my head around from the first 6 exercises is the concept of a callback, which I had to use in my solution to exercise 6.

Thursday, July 16, 2015

Notes from Section 2.1 of Manual of Airborne Topographic Lidar


Chapter 2 of the Manual of Airborne Topographic Lidar, titled "An Overview of ALS Technology", provides a broad introduction to the operating principles and key elements of Lidar, and then discusses the specifics of several existing Lidar systems. Since it contains 90 dense pages of information, I'll break my notes on it into several posts.




Operating Principles

  • LASER - Light Amplified by Stimulated Emission of Radiation.
  • Lidar works with light in electromagnetic spectrum from ultraviolet through infrared.
  • Wavelengths are chosen with regard to range-performance, atmospheric or water absorption, eye-safety, and reflectivity of lased materials.
  • Lasers are nearly monochromatic (they emit light within an extremely narrow band of wavelengths).
  • The emitted wavelength is a function of the material in which the light is stimulated, so wavelength vs. material tables can be produced, like this one:
  • When laser light strikes a surface, parts of the light are:
    1. transmitted
    2. absorbed
    3. reflected
  • The distribution of these 3 parts is a function of the lased surface and the wavelength of the beam (see graph below for example distributions of common earth surfaces).
  •  All lidar systems are comprised of 2 parts:
    1. transmitter emitting laser pulses
    2. optical receiver detecting backscattered pulse
    1. The illustration below shows a simplified model of the laser ranging principle:

      which is represented by the equation:
      Δt = 2R/c
      Δt is the time between when the laser pulse is emitted and when its backscattered pulse reaches the receiver. c is the speed of light, which is approximately 299700 km/s in the earth's atmosphere. Solving for the range, R, we get:
      R = Δt·c/2
      so the range is half the product of the elapsed time and the speed of light.
    2. The square wave idealization is not a very realistic representation of an actual laser pulse, which is much more closely modeled by a Gaussian wave like this:
      The text discusses in some detail how this waveform is analyzed to determine the pulse. One common way this is done is referred to as full width half maximum (FWHM), which is the pulse duration at 50% or more of its peak intensity. Two other common measurements are 1/e and 1/e2 (shown in the above illustration) of peak intensity.

    Additional Resources

      Wednesday, July 15, 2015

      Learning Leaflet 1


      Leaflet is a free software JavaScript library for creating web mapping applications.  Since the Photovoltaic Viability Map project uses it, it's time I learned how to use it as well.

      Using the following three resources:
      I created a web page with a map displaying the area around where I live:

      Which rendered in a browser produces this:

      Lessons Learned

      From The Basics page on the SWITCH2OSM site, I learned that there are two components to a web based map application:
      1. A database of 'tiles' which are rendered (usually in 256x256 pixel sections) side by side to make the map.
      2. A JavaScript API for viewing the database tiles.
      While the OpenStreetMap (OSM) tiles are open data, and are thus free to use, the serving of them is not. Given the cost of maintaining and serving 46 gigabytes of data to the whole world, this is understandable, but I never thought about it before. OSM has a usage policy for their tile servers, which also suggests alternative tile servers for their tiles.

      So to setup a web map application that uses OSM tiles, one has two choices:
      1. Download the OSM database and generate the tiles yourself.
      2. Use a third-party supplier (some charge, some don't).
      The German company Geofabrik hosts the OSM database separated by regions, countries and states (within the US) in North America here. So instead of the 46 gigabytes of the entire OSM database, I could get just the 211 megabytes for the state of Virginia here.

      A content delivery network (CDN) is a distributed system of servers across the Internet for the purpose of rapid delivery of content.  The quick start guide on the leaflet site uses the Cloudfare CDN to serve leaflet. The pv-viability-map uses MapQuest's CDN as a tile server.  For my first leaflet map, I used both of these together.

      This is JavaScript that (using leaflet) does the work of rendering the map:
      var map = L.map('map').setView([38.8605579, -77.1166921], 15);

      L.tileLayer('http://otile{s}.mqcdn.com/tiles/1.0.0/map/{z}/{x}/{y}.jpg', {
          attribution: 'Portions Courtesy NASA/JPL-Caltech and U.S. Depart. of Agriculture, Farm Service Agency. Tiles Courtesy of <a href="http://www.mapquest.com/" target="_blank">MapQuest</a> <img src="http://developer.mapquest.com/content/osm/mq_logo.png">',
          maxZoom: 18,
          subdomains: '1234'
      }).addTo(map);
      The tileLayer method does the job retrieving and rendering the tiles selected by the setView method on the Map object.  The string substitution variables s, x, y, and z contain:
      s
      Sequence of available subdomains for the CDN. Set to '1234' in this case.
      x
      The longitude.
      y
      The latitude.
      z
      The zoom level.
      In this source listing I changed what had been 'sat' in the url to 'map' (colored green in the source listing for identification), which changed the rendered map to:
       I experimented with the zoom level and maxZoom settings.  Changing the zoom level to 18 and setting maxZoom to 19, the map looked like this:
      and allowed the + button to be clicked once, after which it became grayed out.  The highest usable zoom level seems to be 19 anyway, since when I tried 20 the map became solid gray. A zoom level of 1 shows the map of the entire planet.

      Saturday, July 11, 2015

      Photo Voltiac Viability Map II

      In a previous post I described installing the Photovoltaic Viability Map server software. I got the point where the 'make run' script was failing on connecting to the database.  In this post I'll describe how to get the database setup.

      I have installed PostGIS on the Ubuntu 14.04 server using:

      $ sudo aptitude install postgresql-9.3-postgis-2.1

      There is an init.sql file included with the web app to setup the database:



      When I run it with:
      $ sudo su postgres
      $ psql -d postgres -a -f init.sql
      I get an error:
      psql:init.sql:17: ERROR:  type "geometry" does not exist
      A post on stack overflow told me I needed to run:
      $ psql -d postgres -c "CREATE EXTENSION postgis;"
      After running this, init.sql ran without error, but 'make run' has a new problem:
      $ make run
      python3 httpserver.py
      Traceback (most recent call last):
        File "httpserver.py", line 17, in <module>
          host=app.config["DBHOST"]
        File "/usr/lib/python3/dist-packages/psycopg2/__init__.py", line 179, in connect
          connection_factory=connection_factory, async=async)
      psycopg2.OperationalError: FATAL:  database "test" does not exist

      make: *** [run] Error 1
      I ran:
      $ egrep test *
      and found:
      flaskconfig.py:DBNAME = 'test'
      After changing 'test' to 'postgres', I have a running server:
      $ make run
      python3 httpserver.py
      [15-07-11 11:57:54][NOTE] Starting server
       * Running on http://0.0.0.0:5000/
       * Restarting with reloader
      [15-07-11 11:57:54][NOTE] Starting server
      Connecting to it from the browser on my desktop machine, I see:
      My next task will be to populate the database with data.

      Thursday, July 9, 2015

      Adminning a PostGIS Server

      In a previous post I documented setting up a VirtualBox VM running Ubuntu 14.04 as a PostGIS server. In this post I'll investigate administering the server, including creating databases and database users. Much of the on-line documentation for PostgreSQL is geared toward running it on Windows, or else spends a lot of unnecessary time from my point of view covering all three platforms (Windows, Mac, and Linux).  I'm using an Ubuntu server, so I want documentation for that. Here are a few resources that look promising:
      Of these the first one looks most promising, so I'll start with that. From it, I learn that:
      • Configuration files are located at: /etc/postgresql/9.3/main
      • Main config file is: postgresql.conf
      • Inside this file:
        • data_directory directive specifies where the databases are stored (on my machine it is set to /var/lib/postgresql/9.3/main).
        • hba_file directive specifies the host-based authentication file (on my machnie set to /etc/postgresql/9.3/main/pg_hba.conf).
        • ident_file directive specifies the ident authentication file (on my machine set to /etc/postgresql/9.3/main/pg_ident.conf).
        • The default port is set to 5432, and changing this requires restarting the server.
        • listen_addresses defaults to localhost.  I changed it to:
          listen_addresses = '*'
          so the server would accept connections from other machines (my desktop machine in this case).
      • The following commands control the running server:
        • $ sudo service postgresql stop
        • $ sudo service postgresql start
        • $ sudo service postgresql restart
        • $ sudo service postgresql reload
      • After changing the listen_address value, I did:
        $ sudo service postgresql restart
        * Restarting PostgreSQL 9.3 database server   [ OK ]
        $
      • Note: In Ubuntu, the server is run as a service called postgresql (configured in /etc/init.d/postgresql).
      Now let me see if I can connect to the database server, first locally and then remotely. Logged into the server via ssh, I did:
      $ sudo -u postgres psql
      psql (9.3.9)
      Type "help" for help.

      postgres=# help
      You are using psql, the command-line interface to PostgreSQL.
      Type:  \copyright for distribution terms
             \h for help with SQL commands
             \? for help with psql commands
             \g or terminate with semicolon to execute query
             \q to quit
      postgres=# \q
      $
      The YouTube video I listed above suggests a slightly different approach:
      $ sudo su postgres
      $ psql -U postgres -d postgres
      psql (9.3.9)
      Type "help" for help.

      postgres=#
      The -U argument to psql is the user and the -d argument is the database (both postgres at this point).  To be able to connect remotely, the postgres user is going to need a password:
      postgres=# \password postgres
      Enter new password:
      Enter it again:
      postgres=#
      Now let me see if I can connect to the server remotely.  From a terminal on my client machine:
      $ psql -h 10.0.0.12 -p 5432 -U postgres -d postgres
      psql: FATAL:  no pg_hba.conf entry for host "10.0.0.5", user "postgres", database "postgres", SSL on
      FATAL:  no pg_hba.conf entry for host "10.0.0.5", user "postgres", database "postgres", SSL off
      $
      A bit of googling led me to a StackExchange post with a solution to this problem. Back on the server again, I did:
      $ sudo vi /etc/postgresql/9.3/main/pg_hba.conf
      and changed this line:
      host  all  all  127.0.0.1/32  md5
      to:
      host  all  all  0.0.0.0/0     md5
      and finally:
      $ sudo service postgresql restart
       * Restarting PostgreSQL 9.3 database server    [ OK ]

      $
      Time to try again from the client machine:
      $ psql -h 10.0.0.12 -p 5432 -U postgres -d postgres
      Password for user postgres:
      psql (9.4.4, server 9.3.9)
      SSL connection (protocol: TLSv1.2, cipher: DHE-RSA-AES256-GCM-SHA384, bits: 256, compression: off)
      Type "help" for help.

      postgres=#
      Success (and it appears to confirm that there is no problem mixing a 9.3 server with a 9.4 client)!  I can now connect to the database server remotely, and I'm ready to start doing things with the database.

      I'm also going to setup PostgreSQL's GUI management tool on my client machine, just in case that will make managing the server easier.

      Setting Up pgAdmin on the Client Machine

      There is a GUI administration application for PostgreSQL called pgAdmin. To install it on my client machine (my desktop), I:
      $ sudo aptitude install pgadmin3
      [sudo] password for [username]:
      The following NEW packages will be installed:
        libpq5{a} libwxbase3.0-0{a} libwxgtk3.0-0{a} pgadmin3 pgadmin3-data{a}
        pgagent{a} postgresql-client{a} postgresql-client-9.4{a}
        postgresql-client-common{a}
      0 packages upgraded, 9 newly installed, 0 to remove and 2 not upgraded.
      Need to get 11.3 MB of archives. After unpacking 47.1 MB will be used.
      Do you want to continue? [Y/n/?] Y
      Since my client machine is running Ubuntu 15.04, it installed the postgresql-client-9.4 package while the server is running version 9.3 of the server.  Hopefully there won't be a problem for the newer client to connect to the slightly older server.

      After which I could launch the application from the Ubuntu dash, and see this window:

      Figuring I could now connect to the server, I clicked the icon that looks like a power plug and:
      Clicking the OK button showed that it worked:
      Progress!