Getting Started with LIDAR Data

LIDAR, as I learned in my Geospatial Science Fundamentals course last Fall, is an active remote sensing technology that produces three dimensional representations of the earth's surface by gathering reflected laser light, usually from equipment mounted in an airplane passing over the modeled surface.  My Summer independent study at GMU will involve learning to process LIDAR data from the Northern Virginia region, with the aim of using the results to provide useful information to building owners in the region on the viability of putting solar panels on their roofs.

I will of course want to see if I can accomplish this task using free software, since citizen science is ill served by fees or licenses which restrict open collaboration on the widest possible scale.

I've been casually gathering information for the past few months, and here is what I've done so far:

• read about the US National Lidar Dataset project and found out Virginia's dataset status is listed as "partial".
• created an account on the USGS's EarthExplorer website.
• searched for Loudoun County Virginia and filtered on Lidar, which returned a single data set with LiDAR Entity ID: VA_LOUDOUNCO_2012_000298.
• downloaded the 76.7 MB "LAS Product" of this data.
• handed over the data to one of my students, Sam Phillips, who is helping me with this project.

Sam's blog is here. We plan to use our blogs to both document our work and have a digital dialog of our efforts. The impetus for this project comes from the GIS manager of the Northern Virginia Regional Commission, Shafi Khan, who came to our Code for NOVA meet up to propose the idea. I took notes from our first on the project, which included the following:

• The Minnesota Solar Suitability Analysis, a project of graduate students at the University of Minnesota, can serve as a model and resource for our project (see Minnesota LIDAR Solar Study for a YouTube video about the project).
• The Virginia Lidar website is a good resource for LIDAR data from Virginia.
• Harvard University Center for Geographic Analysis's WorldMap project could be a resource for making maps.

Finally, to provide structure to the theoretical study related to the project, I plan to make my way through Penn State University's GEOG 481: Topographic Mapping with Lidar, which wonderfully includes OpenCourseWare for the course.  I've ordered the required text book and will begin working on the lessons this week.

My next post on this topic will describe my first experiments using Python to process LIDAR data...

Learning and Teaching Firefox OS

With only about five weeks left in the school year, and with my Spring semester grad school class completed, I'll now shift focus to three things which are all part of student projects this year, Summer internships, and technologies I hope to integrate into the IT program of Arlington Tech next year:

1. Firefox OS app development
2. Mapping LIDAR data with Python
3. Robotics with MicroPython and the pyboard

I will document each of these efforts in separate blog posts.

First on the list is Firefox OS app development.  I've written about my early enthusiasm for FXOS in a previous post.  The challenge now will be to gather and organize a collection of accessible and high quality resources to support incorporation of FXOS into a high school / community college curriculum.

The principal obstacle in meeting this challenge is that Firefox OS is a rapidly developing technology, so in attempting to write educational materials for it you are aiming at a moving target.  I've already found this to be a problem with existing materials.  One of the best written tutorials I've found thus far for FXOS is on the blog iRomin.  Written as a series of 10 episodes, Episode 1 : Setting up your Development Environment, is already woefully out of date.  It describes setup procedures that have already been replaced by better and easier to use technologies.

It seems important to me to get folks who want to help nurture the fledgling OS to release their books, blogs, and tutorials as open educational resources. I certainly will, but my life as a free software activist would be made so much easier if I could build on the work of others as I set about creating a curriculum for FXOS.  I'll be contacting the authors of the usable materials I find and asking them if I can update and adapt their work.

Here is a list of resources I plan to start with:

• iRomin's Firefox OS Episodes
• Firefox OS in Action by Jan Jongboom
• Firefox OS app beginners tutorial
• Creating Your First Firefox OS App
• Simulator Walkthrough

One of the first things I want to do is create a Firefox OS app "zoo", along the lines of the jQuery Zoo. The idea is to populate the site (and accompanying github repo) with a collection of small, easy to understand apps that are designed primarily to meet pedagogical goals.  I should start with a "hello world" app and build from there.

I've got a student coming in tomorrow morning for help getting started with FXO, so I best get working on the app zoo now...

Geospatial Representation of Roads

It turns out that the polygons in the Roads shapefiles do not represent roads the way I thought they would.  To explore the shapefile, I wrote a script using pyshp: explore_shapefile.py.  The pyshp module is definitely more rewarding to use than GDAL, since it is pure Python and it is thus possible to use Python's type introspection to explore.  I've found that all of the scripts I've written in Python 3 will run also in Python 2, which is convenient.  I also came across one of the big improvements Python 3 has over Python 2 in terms of type introspection.

The following line of code prints out the type of the first element of a list from a shapefile: 

print('type(sr[0]): {0}'.format(type(sr[0])))

Running the script in Python 2 results in: 

type(sr[0].shape): <type 'instance'>

which is not very informative.  Python 3 does much better: 

type(sr[0]): <class 'shapefile._ShapeRecord'>

Anyway, since my final project is due tomorrow, I had to cut the investigation into using pyshp short, but I'll be sure to continue looking into it soon.

My goal was to represent Columbia Pike in bright yellow or orange on my map. Columbia Pike is the road on which the proposed streetcar that had such a profound impact on the Vihstadt / Howze election ran.  I thought adding it to the map would provide visual interest.  It also gave me a chance to explore adding more than one layer to a map.

I wrote several scripts to make maps as part of investigating this.

draw_roads.py

Renders all 883 shapes from the Roads shapefile in a map.

a_road_shape.py

Renders a single shape from the Roads shapefile as a new layer on the precincts map.

roads_near_columbia_pike.py

I used trial and error (and a bit of divide and conquer) to find shapes among the 883 that represented roads on and near Columbia Pike.

election_map_with_roads_near_cp.py

Finally, I added this roads layer to the election results map I made yesterday.

The first script, draw_roads.py, generates a map of the roads in Arlington County:

I have yet to begin studying QGIS, but I googled how to load a shapefile as a vector layer to see how it would compare:

Since I generally feel more comfortable expressing my thoughts in code rather than using tools, I think I'm going to generally prefer writing python scripts to using qgis to make maps, but I realize I'll have to do both.

I assumed that each of the 883 shapes in the Roads shapefile represented a single road.  I was disappointed that the metadata didn't include names associated with each of the shapes, but I figured I could use a kind of binary search to print out half of them, render the map, and see which half Columbia Pike was in, cut that in half, and so forth, until I found Columbia Pike.  Soon I'll be able to read geospatial coordinates from the shapes and write a script that selects the shapes I want, but for now I figured this approach would be the quickest way to get to my goal.

It turns out that the reason there are no road names associated with the shapes is that each shape represents a cluster of connected road segments rather then a single road.  a_road_shape.py illustrates this:

The orange line segments are from a single shape from the Roads shapefile.  Once I realized that, I spent some time gathering shapes that had a piece of the Pike on them, and made roads_near_columbia_pike.py to render them:

Finally, I merged this with the color coded election results map I made yesterday in election_map_with_roads_near_cp.py to produce:

The color contrast is bad here, but I'm running out of time to complete the project, and since this fails to render just Columbia Pike the way I wanted anyway, I decided not to spend any more time on it.

Where I'll Go From Here

OpenStreetMap is an incredible resource for open source mapping projects, and yesterday I did a quick search to see if it might provide me with a solution to my rendering Columbia Pike project.  I found out that OSM stores streets as a collections of ordered nodes called ways. This way holds a piece of Columbia Pike near where I live:

https://www.openstreetmap.org/way/334620323#map=18/38.85606/-77.10834

Here are the things I think I should investigate next:

• OSM data
• pyproj

My immediate project goal will be to render Columbia Pike, this time from OSM, onto the election map as I had originally intended.  I anticipate that the pyproj module will be the tool to use to convert among different geospatial data formats.

A Color Coded Election Results Precinct Map

Learning to use Python to make maps is becoming more fun the more time I spend with it.  This morning I was able to successfully generate a color coded precinct map showing which precincts were won by Alan Howze and John Vihstadt in last November's Arlington County Board election.

Here is what I did:

1. Found a data set containing the election data.  The Virginia Department of Elections website has a CSV file with the data I needed here.
2. Learned to use Python to extract the data I needed from this source.  This turned out to be surprisingly easy (maybe not so surprising, that's why I love Python!). I just needed the cvs module which comes with the standard library.  I then wrote a small script, precinct_winners.py, which produced a list containing the winner for each precinct.
3. Wrote a script, vihstadt_howze_election_map.py, using mapnik and my precinct_winners.py script to generate a color coded map showing precincts which Howze won in green and those Vihstadt won in red.

Here is the resulting map:

I plan to spend the rest of my available time today trying to overlay Columbia Pike, the road along which the proposed streetcar that caused such a stir in this election had been planned to run.

I found a shapefile with all the roads in Arlington County here.  I'm planning to explore this file with the pyshp rather than the GDAL module, since pyshp is pure python instead of a wrapper around a C++ library and thus will be easier to work with.

My next post will report on my results.

Drawing My First Map with Python

I was able to use python to read data from a shapefile and generate a map of voter precincts in Arlington County, Virginia.

Arlington County, Virginia makes numerous maps available on-line here.  The specific shapefile I started with holds boundaries for voter precincts in the county, and is available here.

I used the Voter_Precinct shapefile and adapted what I learned about gdal and mapnik from Python Geospatial Development. Specifically, the following chapters contained the examples I needed for this first map:

• Chapter 3 - Python Libraries for Geospatial Development
• Chapter 5 - Working with Geospatial Data in Python
• Chapter 8 - Using Python and Mapnik to Produce Maps

I also read through the ESRI Shapefile Technical Description.

The Scripts

I wrote the following two scripts, which are now located in my gis_front_and_back repo on github:

• analyze_voter_precinct_shapefile.py
• make_precinct_map.py

The README.md file, also in the github repo, contains a link to where to get the shapefile.

Since I use Python 3 whenever I can, the first script uses Python 3, while the 2nd, which requires mapnik, is written in Python 2.  Running the first script and redirecting the output to a file produces this.

Running the 2nd script generates this map:

Making this map by adapting the example scripts from the text was fairly trivial.  I've been unable to get filters to work thus far, however, which will enable me to vary the colors of the individual precincts based on rules.

I've been wanting to make a map ever since last November's election which will color code the results by precinct in the race between John Vihstadt and Alan Howze (see John Vihstadt beats Democrat Alan Howze in race for Arlington County Board seat).  The key issue in this race was the proposed Columbia Pike Streetcar, which was canceled soon after the election (see Arlington Cancels Streetcar Program).

I grabbed precinct election results from the elections.virginia.gov website and saved them in a PDF file which I still have here. I was struck by how the support for the streetcar was clearly visible near the communities most effected by it (along the Columbia Pike corridor), while opposition (assuming a direct correlation between opposition to the streetcar and support for Vihstadt) was strongest in North Arlington, furthest away from where the streetcar would have run. This kind of data cries out for a good map, so I've been wanting to make one ever since.

I just downloaded the complete election result data in CVS format from here.  My most immediate problem now is to figure out how to color individual precincts based on filters.  I'll also look further into how to manipulate shapefiles, perhaps using the pyshp library, which can be installed on Ubuntu through the python-pyshp and python3-pyshp debian packages for versions 2 and 3 of Python respectively.  Then I'm thinking I'll either figure out how to read the election result data from the CVS file, or use the PDF to read the election results for each precinct and to color code them on the map accordingly.

The pyboards Have Arrived!

Arlington Tech, the name we have informally given to the new school being planed for the current Arlington Career Center, will have a focus on STEM ("green STEM" in particular).  Our carpentry instructor has both experience with and a big interest in high school robotics, and since robotics is an inherently interdisciplinary endeavor, I'll be working with him next year to help students learn to program the robots.

When I heard about the programmable microprocessor that runs Python 3 called the pyboard at Pycon a last week, I was naturally interested in investigating it as a possible tool for our robotics effort.

I ordered two pyboards, which arrived today.  Here is a picture of one of them:

I searched up a quick tutorial on using it, and found Micro Python and pyboard - a Python running on a microcontroller scripts your electronics.

Here is a screen shot of one of the example scripts from this tutorial, which prints out value from the accelerometer on the board:

I have to stay focused on GIS programming and Firefox OS, but I have a friend interested in microprocessors who plans to explore this further.

Installing Python GIS Libraries on Ubuntu 14.04

I'm working my way through Python Geospatial Development, which as good fortune would have it, is the book we are using in my Introduction to GIS Algorithms and Programming course at George Mason University.  At the beginning of the course we were given a VirtualBox image of Ubuntu 12.04 with all the software and data files we need for the course preloaded.  Since I'm already running Ubuntu 14.04 as my regular desktop OS, I preferred to just install the packages (all of which are in the standard debian repositories) onto my desktop machines.

Here are the libraries we are using:

• GDAL - The Geospatial Data Abstraction Library (see Python GDAL/OGR Cookbook for Python specific "recipes") for reading and writing geospatial data (GDAL for raster, OGR for vector).
• pyproj - Performs cartographic transformations and geodetic computations.
• Shapely - Manipulation and analysis of geometric objects in the Cartesian plane.
• Mapnik - Toolkit for building mapping applications.

To install them for Python 2 on Ubuntu 14.04, run the following in a unix shell:

• $ sudo apt-get install python-gdal
• $ sudo apt-get install python-pyproj
• $ sudo apt-get install python-shapely
• $ sudo apt-get install python-mapnik2

Even on Ubuntu 15.04 Mapnik is only available for Python 2.  The other three libraries (gdal, pyproj, and shapely) are all available for Python 3.  This means that I will unfortunately have to use Python 2 for my mapping project.

I've now read the first eight chapters of Python Geospatial Development, which contain all the information I'll need for my project:

• Chapter 1 - Geospatial Development Using Python
• Chapter 2 - GIS
• Chapter 3 - Python Libraries for Geospatial Development
• Chapter 4 - Sources of Geospatial Data
• Chapter 5 - Working with Geospatial Data in Python
• Chapter 6 - GIS in the Database
• Chapter 7 - Working with Spatial Data
• Chapter 8 - Using Python and Mapnik to Produce Maps

It is often quite difficult to find good books about rapidly developing technologies like this, but this book does a good job.  The material is relevant, the presentation clear, and the examples both engaging and illuminating.

With background reading done and infrastructure setup, it is time to dig into the project.