Mobile Happiness with Firefox OS!

As I described in a previous post back in mid June at the start of the LibriFox "Summer of Code", I am not as a rule an early adopter of new technologies, and am now making regular use of a mobile phone for the first time in my life.

I just purchased a Nexus 5 and so far I am delighted with it.

Here is why:

• I was able to easily root the device, install MultiROM Manager, and then make it dual-boot (triple boot, actually, since I installed Ubuntu as well) with Firefox OS.  I will be using Firefox OS as my main OS, and I'm happy to say that the version v2.2 pre-release image I found here is working great.  The only app that doesn't work is the FM radio, since the Nexus 5 unfortunately doesn't have an FM chip. What it does have that I really wanted was 4G connectivity. Browsing the web or loading a location map through the mobile data connection are quite pleasant now.
• Alex Hirschberg completed a wonderfully successful "Summer of Code", delivering the LibriFox app to the Firefox OS Marketplace.  For me this is a "killer app" that alone makes having the phone worthwhile.  That's why I setup the "Summer of Code" in the first place, and I am completely satisfied with the results that Alex delivered.  I would not hesitate to fund "Summer of Code" again next year, provided I can find another graduate at least half as good as Alex. Here is a screen shot of LibriFox running on my phone:
  
• As a regular Capital Bikeshare user, I thought I was going to have to boot into Android to use BikeShare! Not so. DC Bike Finder will serve my needs nicely, and keeps me from having to reboot. Here is a screenshot of DC Bike Finder running on my phone:
  

What drew me to Firefox OS in the first place was a developer environment that lowered the barrier to entry (see my April 16 post, Firefox OS and Lowering the Barrier to Entry into ICT) and would thus be a platform friendly to student learning. With a mobile device that I like to use and feel closely connected with thanks to contributing to it through LibriFox, I am excited about the prospect of further developing the Firefox OS curriculum inroads we began last year and seeing both how far we can progress in the new year and what surprises our journey will bring.

Setting Up GeoDjango II

In my previous post on this topic, I ended up stuck with a database not existing error, which my good friend Kevin Cole very politely (only implying that I'm an idiot, while refraining from directly saying so ;-) pointed out that the documentation I'm using contains the instructions I need to create the database:

(env)$ createdb -T template_postgis geodjango

Unfortunately, running that gave me the following error:

createdb: database creation failed: ERROR:  template database "template_postgis" does not exist

Let me try a modified version (modified because my user doesn't have the privileges needed to create extensions) of the steps laid out here:

(env)$ createdb geodjango
(env)$ sudo su - postgres
$ psql geodjango
psql (9.3.9)
Type "help" for help.

geodjango=# CREATE EXTENSION postgis;
CREATE EXTENSION
geodjango=# \q

$ exit
logout
(env)$

Now let me resume where I left off before the error:

(env)$ python manage.py sqlmigrate world 0001
BEGIN;
CREATE TABLE "world_worldborder" ("id" serial NOT NULL PRIMARY KEY, "name" varchar(50) NOT NULL, "area" integer NOT NULL, "pop2005" integer NOT NULL, "fips" varchar(2) NOT NULL, "iso2" varchar(2) NOT NULL, "iso3" varchar(3) NOT NULL, "un" integer NOT NULL, "region" integer NOT NULL, "subregion" integer NOT NULL, "lon" double precision NOT NULL, "lat" double precision NOT NULL, "mpoly" geometry(MULTIPOLYGON,4326) NOT NULL);
CREATE INDEX "world_worldborder_mpoly_id" ON "world_worldborder" USING GIST ("mpoly" );

COMMIT;
(env)$ python manage.py migrate
Operations to perform:
  Synchronize unmigrated apps: gis, messages, staticfiles
  Apply all migrations: world, admin, contenttypes, auth, sessions
Synchronizing apps without migrations:
  Creating tables...
    Running deferred SQL...
  Installing custom SQL...
Running migrations:
  Rendering model states... DONE
  Applying contenttypes.0001_initial... OK
  Applying auth.0001_initial... OK
  Applying admin.0001_initial... OK
  Applying contenttypes.0002_remove_content_type_name... OK
  Applying auth.0002_alter_permission_name_max_length... OK
  Applying auth.0003_alter_user_email_max_length... OK
  Applying auth.0004_alter_user_username_opts... OK
  Applying auth.0005_alter_user_last_login_null... OK
  Applying auth.0006_require_contenttypes_0002... OK
  Applying sessions.0001_initial... OK
  Applying world.0001_initial... OK(env)$

Progress!  Let me keep going and see if my good fortune holds:

(env)$ python manage.py shell

Python 3.4.0 (default, Jun 19 2015, 14:20:21)
[GCC 4.8.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> import os
>>> import world
>>> world_shp = os.path.abspath(os.path.join(os.path.dirname(world.__file__),
...                             'data/TM_WORLD_BORDERS-0.3.shp'))
>>> from django.contrib.gis.gdal import DataSource
>>> ds = DataSource(world_shp)
>>> print(ds)
/home/[user/geodjango/geodjango/world/data/TM_WORLD_BORDERS-0.3.shp (ESRI Shapefile)
>>> print(len(ds))
1
>>> lyr = ds[0]
>>> print(lyr)
TM_WORLD_BORDERS-0.3
>>> print(lyr.geom_type)
Polygon
>>> print(len(lyr))
246
>>>

The tutorial continues with several other interactive examples showing how to use GeoDjango's pythonic interface to the GDAL library.  I began experimenting with python's GDAL wrapper back in April as part of the Introduction to GIS Programming and Algorithms course I took at George Mason University.  I documented the installation of these tools in a post at that time.  The ability to "play" with data at run time is one of the many things I love about Python, and this tutorial, like most python tutorials, is making good use of that powerful pedagogical feature of the language.  There is no need for me to recount the other examples here, however, so I'll skip over them.

The next step in the tutorial is to create a file in the world app named load.py that contains the following:

import os
from django.contrib.gis.utils import LayerMapping
from .models import WorldBorder

world_mapping = {
    'fips' : 'FIPS',
    'iso2' : 'ISO2',
    'iso3' : 'ISO3',
    'un' : 'UN',
    'name' : 'NAME',
    'area' : 'AREA',
    'pop2005' : 'POP2005',
    'region' : 'REGION',
    'subregion' : 'SUBREGION',
    'lon' : 'LON',
    'lat' : 'LAT',
    'mpoly' : 'MULTIPOLYGON',
}

world_shp = os.path.abspath(os.path.join(os.path.dirname(__file__), 'data/TM_WORLD_BORDERS-0.3.shp'))

def run(verbose=True):
    lm = LayerMapping(WorldBorder, world_shp, world_mapping,
                      transform=False, encoding='iso-8859-1')

    lm.save(strict=True, verbose=verbose)

Note: The tutorial lists: from models import WorldBorder, which will cause an import error.  models needs to be .models for this to work.

After making that change, I was able to:

(env) python manage.py shell
>>> from world import load
>>> load.run()

and watch as the countries of the world were loaded into the database.

Creating the github repo

Now would a good time to create a github repo.  First, I'll create a .gitignore file inside the top level geodjango directory (where manage.py is located) that lists the things I don't want in the repository:

env/
__pycache__/
*.py[cod]
*.sw?

This will tell git not to include the virtual environment, the python byte code files, and any vim swap files.  Next install git and initialize the repository:

(env)$ sudo aptititude install git
(env)$ git init

Now I'll check to see what a git add . would add:

(env)$ git add -n .

Since it looked good, I'll add do it, after configuring my git email and user:

(env)$ git config --global user.email [github email address]
(env)$ git config --global user.name [github name]
(env)$ git add .
(env)$ git commit -a
Initial commit.

Now push it to github (after adding an ssh key and creating a learn_geodjango project on github):

git remote add origin git@github.com:[user]/learn_geodjango.git
git push -u origin master

With the github repo now created, I'll continue the tutorial in a future post.

Setting Up GeoDjango I

According to the documentation website,

"GeoDjango intends to be a world-class geographic Web framework. Its goal is to make it as easy as possible to build GIS Web applications and harness the power of spatially enabled data."

Since my goal is to become a free software GIS application developer, and since Python and Django are two of the core technologies I hope to utilize, GeoDjango seems like a no-brainer as something I should learn.

In this post I'm going to document the beginning process of setting up a GeoDjango server on a local VirtualBox VM.  In a later post, I'll look into installing it on the WebFaction application hosting service. Throughout this process, I'm going to modify the steps described in the tutorial to facility creating a github repo and migrating the app to WebFaction.

Since GeoDjango comes with Django, I'm going to start by creating a virtualenv on the same virtual machine I used to setup a django CMS virtualenv, the process for which I described in a previous post, Installing Django CMS on Ubuntu 14.04.  I'll install this new virtualenv right alongside the other one.  I'll be following the installation instructions from the GeoDjango Installation page, together with blog posts I've made previously documenting specific steps to install in my Ubuntu 14.04 VM. Installing GeoDjango requires installation of:

1. Django
2. Spatial database

I'll tackle each in turn.

Installing Django

$ mkdir geodjango
$ cd geodjango
$ virtualenv  env
$ source env/bin/activate
(env)$ pip install django
(env)$ django-admin --version
1.8.3
(env)$ deactivate

Installing a Spatial Database

According to the documentation, "PostGIS is recommended, because it is the most mature and feature-rich open source spatial database." It also recommends that Ubuntu installation use packages.  Since I'm new to this, I'll follow the recommendation.

$ sudo aptitude install postgresql-9.3 postgresql-9.3-postgis-2.1 postgresql-server-dev-9.3 python3-psycopg2

Next setup a database user who can create databases:

$ sudo su - postgres
$ createuser --createdb [user]
$ exit

Now follow the tutorial to see if it works (note: the command in red has been modified from tutorial with the aim of keeping the virtual environment directory (env) inside the project directory):

$ cd geodjango
$ source env/bin/activate
(env)$ django-admin startproject geodjango .
(env)$ python manage.py startapp world
(env)$ vi geodjango/settings.py

Change the DATABASES section to match the following:

DATABASES = {
    'default': {
        'ENGINE': 'django.contrib.gis.db.backends.postgis',
        'NAME': 'geodjango',
        'USER': '[user]',
    }
}

Also add the last two items to INSTALLED_APPS so that it looks like this:

INSTALLED_APPS = (
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'django.contrib.gis',
    'world',
)

For the next step, we will need some gdal packages (and I'll grab unzip while I'm at it:

(env)$ sudo aptitude install gdal-bin python3-gdal unzip
(env)$ mkdir world/data
(env)$ cd world/data
(env)$ wget http://thematicmapping.org/downloads/TM_WORLD_BORDERS-0.3.zip
(env)$ unzip TM_WORLD_BORDERS-0.3.zip
(env)$ cd ../..
(env)$ ogrinfo world/data/TM_WORLD_BORDERS-0.3.shp
INFO: Open of `world/data/TM_WORLD_BORDERS-0.3.shp'
      using driver `ESRI Shapefile' successful.
1: TM_WORLD_BORDERS-0.3 (Polygon)
(env)$

The tutorial includes another ogrinfo example command that provides more detail about the world borders shape file, which I'll skip in the interest of space.  I will include the excellent short description of each of the component files in the shapefile:

.shp:

Holds the vector data for the world borders geometries.

.shx:

Spatial index file for geometries stored in the .shp.

.dbf:

Database file for holding non-geometric attribute data (e.g., integer and character fields).

.prj:

Contains the spatial reference information for the geographic data stored in the shapefile.

Next edit the models.py file so that it looks like this:

from django.contrib.gis.db import models

class WorldBorder(models.Model):
    name = models.CharField(max_length=50)
    area = models.IntegerField()
    pop2005 = models.IntegerField('Population 2005')
    fips = models.CharField('FIPS Code', max_length=2)
    iso2 = models.CharField('2 Digit ISO', max_length=2)
    iso3 = models.CharField('3 Digit ISO', max_length=3)
    un = models.IntegerField('United Nations Code')
    region = models.IntegerField('Region Code')
    subregion = models.IntegerField('Sub-Region Code')
    lon = models.FloatField()
    lat = models.FloatField()
    mpoly = models.MultiPolygonField()
    objects = models.GeoManager()

    def __str__(self):              # __unicode__ on Python 2
        return self.name

The default spatial reference system is WGS84.  New to me from the documentation is the existence of Open Geospatial Consortium (OGC) Spatial Reference System Identifier (SRID), which in the case of WGS84 is 4326.

The next step in the instructions were to run:

(env)$ python manage.py makemigrations

This didn't work for me, giving me the following error:

ImportError: No module named 'psycopg2'

I fixed this with:

(env)$ pip install psycopg2
(env)$ python manage.py makemigrations
Migrations for 'world':
  0001_initial.py:
    - Create model WorldBorder
(env)$

But when I ran the next command:

(env)$ python manage.py sqlmigrate world 0001

I got a database error:

psycopg2.OperationalError: FATAL:  database "geodjango" does not exist

Anticipating that I might need knowledge of PostgreSQL databased administration, I began looking into that in an earlier post. I better look into that further before returning to the present task.

Tools for Visualizing Lidar Data I

LidarViewer

A friend of mine recently sent me a link to an article on opensource.com titled   Manipulating data in 3D with LidarViewer.  His timing couldn't have been better, since LidarViewer is a free software tool for visualizing Lidar data, and is thus just the kind of thing I need for my Summer project.  Even better, the downloads page starts with an Ubuntu PPA, so installation should not be a nightmare.

I'll do my installation on a VirtualBox VM running Lubuntu 14.04.  I like to use VMs whenever I am trying new software that is not part of the standard Ubuntu software repositories, since this keeps my host machine stable, while letting me experiment without fear.

Here is what I did:

$ sudo apt-add-repository ppa:keckcaves/ppa
$ sudo aptitude update

The installation page doesn't list the packages included in the PPA, but the Launchpad page for the repo does.  There is a package named lidarviewer, so I'll install that:

$ sudo aptitude install lidarviewer

Running lidarviewer at the command prompt after installation completed gave me a "command not found", so I checked to see what the package had installed:

$ dpkg -L lidarviewer
/.
/etc
/etc/LidarViewer
/etc/LidarViewer/LidarViewer.cfg
/usr
/usr/bin
/usr/bin/LidarIlluminator
/usr/bin/PaulBunyan
/usr/bin/LidarPreprocessor
/usr/bin/LidarSubtractor
/usr/bin/LidarGridder
/usr/bin/CalcLasRange
/usr/bin/LidarExporter
/usr/bin/PointSetSimilarity
/usr/bin/PrintPrimitiveFile
/usr/bin/LidarViewer
/usr/bin/LidarColorMapper
/usr/bin/LidarSpotRemover
/usr/share
/usr/share/doc
/usr/share/doc/lidarviewer
/usr/share/doc/lidarviewer/changelog.Debian.gz
/usr/share/doc/lidarviewer/copyright
/usr/share/doc/lidarviewer/HISTORY.gz
/usr/share/doc/lidarviewer/README.gz
$

I tried:

$ LidarViewer
Caught exception LidarViewer::LidarViewer: No octree file name provided

A quick search on the exception led me to the Lidar Viewer Manual.  Since I already installed the application from the Ubuntu PPA (Ubuntu rocks!), I can skip most of the installation instruction section. In the MacOS instructions, however, I found sample data for testing the application.  Downloading and unzipping the sample data, I changed into the LidarViewerExamples directory and ran the following command and got the following error:

$ LidarViewer PtArena.lidar
Cache sizes: 4672 memory nodes, 1170 GPU nodes
libGL error: pci id for fd 12: 80ee:beef, driver (null)
OpenGL Warning: Failed to connect to host. Make sure 3D acceleration is enabled for this VM.
libGL error: core dri or dri2 extension not found
libGL error: failed to load driver: vboxvideo

I'm using a VirtualBox VM, and this message is telling me to enable 3D acceleration.  After enabling 3D acceleration on the VM, the application reported a long list of OpenGL errors.  I found this bug ticket showing I'm not the only one with the issue.

I added the same PPA to a laptop running Ubuntu 14.04 (thus loosing the safety of the virtual machine) and installed both the lidarviewer and  vrui-examples packages, after which I could run the examples on the laptop without error.

lidar2dems

Next I'm going to install another set of tools for visualizing and processing Lidar data, lidar2dems.  Installation instructions are found here, and contain a number of utilities such as LAStools, which I'll need to uncompress the LAZ files that are on the Virginia Lidar website.

Installation of the lidar2dems software is made easy by an installation script, easy-install.sh.  After downloading the script, run:

$ chmod +x easy-install.sh
$ sudo ./easy-install.sh

The script took almost two hours to complete on my VirtualBox VM, but it completed without error.

It did not, however, install many of the LAStools utilities, especially laszip, as I had hoped.

laszip

To get laszip, I went to http://www.laszip.org and downloaded laszip.zip.  Then:

$ unzip laszip.zip

 which created a LAStools directory with several subdirectories, including a bin subdirectory that contained windows .exe binaries and _README.txt text files for many LASzip utilities, including laszip.exe.  Next I:

$ cd LAStools
$ make

This created the following Linux binaries in the bin directory:

las2las  lasdiff   lasinfo   lasprecision  txt2las
las2txt  lasindex  lasmerge  laszip

To test if laszip works, I grabbed a LAZ file for downtown Leesburg:

$ wget https://703348910b61e15b5d68b83128b735c09c3849cb.googledrive.com/host/0B_5XlZJJ2R5tUjZFb1FQTnU1alk/18STJ7733.laz

This got me the file 18STJ7733.laz. Then I ran:

$ ls -l
total 16624
-rw-rw-r-- 1 user user 17019618 Aug 17  2012 18STJ7733.laz
$ laszip 18STJ7733.laz
$ ls -l
total 149256
-rw-rw-r-- 1 jelkner jelkner 135810762 Aug  3 12:05 18STJ7733.las
-rw-rw-r-- 1 jelkner jelkner  17019618 Aug 17  2012 18STJ7733.laz

So it appears to have uncompressed the LAZ file into a much larger (almost 8x larger) LAS file.

Given that it is free software with an LGPL license, I don't understand why someone in the Open Source GIS community hasn't made this installation much simpler on Ubuntu yet.  For now, I've made a modest contribution toward that goal by creating a page with the Ubuntu binaries on my Open Book Project site:

http://openbookproject.net/resources/laszip.php