Sunday, November 1, 2015

Setting-up OpenGeo and TileCache Servers


Installing OpenGeo Suite

In my previous post I talked about the impact that Boundless is having on my GGS 692: Web-based GIS course.  It turns out the impact is even more than I reported, since our final project for the course will be based on the Building a GeoNames Heat Map tutorial found on the Boundless website.

We were given a VirtualBox hard drive image at the beginning of the course, with Ubuntu 12.04 and all the software we would need for class, including the OpenGeo Suite. While I've been using this image, I want to be able to set up everything myself, so I'm going to try to install the OpenGeo software on an Ubuntu 14.04 virtual machine.  I'll be following the instructions for installing OpenGeo Suite 4.7 on Ubuntu 14.04 here.  Following those instructions, I did:
$ sudo su -
# wget -qO- | apt-key add -
# echo "deb trusty main" > /etc/apt/sources.list.d/opengeo.list
# apt-get update
# apt-get install opengeo-server
This installed 244 new packages, including each of the components listed here. The OpenGeo Suite is certainly free software, but it uses both Java and Mono, and I'm a Python fan, so while I'll learn to use GeoServer for class, I'm going to pursue another, lighter map server stack at the same time.


Installing TileCache

TileCache is an implementation of the Tile Map Service (TMS) specification written in Python.  I setup an Ubuntu 15.10 server for this project and installed tilecache with:
$ sudo aptitude install tilecache
It installed 17 dependencies, including python 2.7, along with the tilecache package.  I am beginning to wrap my head around how this map server works.  It is built on three layers:
The geospatial data objects are stored in a PostGIS database.  Mapnik turns the data into images (tiles), which TileCache serves through the web. TileCache implements the Tile Map Service (TMS), a specification developed by OSGeo.

On the client side, a JavaScript library like Leaflet assembles the tiles for viewing.

Continuing with the server installation:
$ sudo aptitude install python-mapnik
$ sudo aptitude install postgresql-9.4-postgis-2.1
 With the software for the two servers installed, my next task will be to load the data for my GIS 692 class onto the OpenGeo server and the Arlington County map data onto the TileCache server.

Saturday, October 31, 2015

Boundless Geospatial Opportunities

A few weeks ago in GGS 692: Web-based GIS we were assigned a set of exercises writing spatial SQL queries on a spatially enabled database. The reading material for the lesson consists mainly sections from an Introduction to PostGIS tutorial produced by a company named Boundless.

It wasn't until I began reading the tutorial that I realized just how much of the curriculum of our course comes directly from the Boundless materials.  By providing high quality, classroom ready learning materials, Boundless has changed the curriculum in the GIS program GMU from one using proprietary software to one built on free software tools.

I only wish other projects in the free software movement could take this valuable lesson from Boundless and find a way to get their tools into classrooms. If we want to change the world, this is an important part of how we have to do it!

Sunday, October 4, 2015

A Geospatial Werewolf

I've been contemplating a post with the tile "Seduced by the Dark Side - Or Why I Love My MacBook Air".  I decided not to devote a whole post to such evil things, so I'll restrict myself to a few paragraphs and then move on to better things.  Let me just admit it.  I really like using the MacBook Air that I was given at work.  I didn't buy it.  I know it's evil (really, really evil, in fact, since it is sooo seductive), and I will certainly do all I can to resist its evil temptations.  I've already removed all the Apple Store paid for applications that were on it, and I've installed Homebrew and a host of free software (things like QGIS and pgAdmin3).  I also installed VirtualBox and set up a Lubuntu 14.04 VM on it, which I use most of the time. In spite of all that, I have to admit that I really like using it.  I like the way it looks.  I like the way it feels in my hands and under my fingers as I type on it.  I like the way it responds so smoothly and quickly. There, I've come clean and got that out of my system.  The last thing to mention is that when I told my friend Kevin Cole that I had been seduced by the dark side, he knew without being told what I was going to say, and sent me this link, which pretty much hits the nail on the head...

Lubuntu 15.10 - A Geospatial Werewolf

Moving back into the realm of freedom, the upcoming release of Lubuntu 15.04, which I am running now on a few different machines but which will have its official release on October 22nd, offers new Python 3 versions of the geospatial libraries I described in a post back on April 19th. It also comes with QGIS version 2.8, which is a good thing indeed, since the version 2.4 that came with previous versions did not work properly (I couldn't get it to create layers using the DB Manager).  Here are the Python 3 GIS packages I installed on this new version, code named Wily Werewolf:
  • python3-gdal
  • python3-pyproj
  • python3-shapely
  • python3-mapnik
The last one in particular was not available in previous releases. And as if a sign of its strong GIS support, the default background on the Lubuntu version features a TIN pattern.  Here is a screenshot with QGIS running:
I've been tasked this week with looking into how to use mapnik.  Thanks to Wily, I'll be able to do this in Python 3.

Saturday, October 3, 2015

Getting an Arlington Bounding Box of PBF Data

The best source for a current, relatively small data set for OSM PBF data I could find is the Geofabrik website, specifically the Virginia page, which has a PBF file for the state that weighs in at less than 230 MB. After downloading the virginia-latest.osm.pbf file, my next task will be to use osmosis to extract data from within a bounding box containing Arlington County.  The web application BoundingBox provides a nice visual way to do this.

Here is a screenshot of the Arlington County bounding box:
An interesting little curiosity I observed from looking at this bounding box is that little Andrew Ellicot Park lies within Arlington County, Fairfax County, and the city of Falls Church:
After Googling it, it is listed as the originial Western cornerstone of the District of Columbia:
Anyway, enough distraction, back to work.  With osmosis installed and the Virginia data on hand, I ran this command to get just the Arlington data:
$ osmosis --read-pbf file=virginia-latest.osm.pbf --bounding-box top=38.9342803955 left=-77.1723251343 bottom=38.8272895813 right=-77.032081604 --write-pbf file=arlington.osm.pbf
Only a few more steps to create a database and load this data into it:
$ createdb map_arlington $ psql -U postgres -d map_arlington -c 'CREATE EXTENSION postgis' $ imposm --read arlington.osm.pbf $ imposm --write -d map_arlington -U postgres
A database user other than postgres should be used, but this at least documents the process.

Thursday, September 24, 2015

Importing OSM Data into PostGIS - Part 1

The PV Viability Map project seems to really be picking up some steam!  Thanks to our good fortune in having David Winslow join us, we now have someone with the skills and experience to move us forward -- thanks, David!  In our meet up last Thursday we had a first discussion of the requirements for the project, which I'll do my best to summarize here.

PV Viability Map Requirements

  • Display map in web page.
  • Search an address and display the building there.
  • Standard map mouse navigation (drag to pan, mouse wheel to zoom, etc.)
  • Click on building to identify.
I have been tasked with seeing how far I can get before next Thursday's meet up with the first task - displaying a map in a web page.  Actually, I've been tasked with the first part of this process - getting OpenStreetMap (OSM) data, trimming for Arlington, and loading it into PostGIS.

Process Overview

Before diving into looking for a solution, a pause to consider the process is in order.  From the Wikipedia page for OpenStreetMap we learn:
  1. The main copy of the OSM data is stored in OSM's main database. The main database is a PostgreSQL database, which has one table for each data primitive, with individual objects stored as rows. All edits happen in this database, and all other formats are created from it.
  2. For data transfer, several database dumps are created, which are available for download. The complete dump is called planet.osm. These dumps exist in two formats, one using XML and one using the Protocol Buffer Binary Format (PBF).
From what I can gather (and the notes I took at our last Code for NoVA meet up), I will be using a combination of two tools to grab the data from OSM's server and then push it into my local database:
  1. osmosis
  2. imposm
I was able to install both of those with:
$ sudo aptitude install osmosis imposm
I'm not clear when one would use imposm and when one would use osm2psql, which seems to be for the same purpose. I installed the latter as well with:
$ sudo aptitude install osm2psql
To be honest, I find the OSM wiki documentation to be very challenging to read. It appears thorough and is well written, but it assumes a wealth of background I lack.  Hopefully with time I'll be able to make better use of it.  For now I'll wait for our meet up this evening to see if David can help me figure out how to use osmosis to get data for Arlington County.

Tuesday, September 22, 2015

Creating a Lubuntu Custom Install Disk for the School Lab

I'm a high school teacher in a lab full of Windows 7 machines, and it's just not
working for me.  I'm teaching Web Page Design I, which is a course in HTML and CSS, and by using WinSCP and Notepad++, the Windows workstations are adequate for the task.  But for my Computer Science class, where we will be exploring mathematics with Python and will need to install lots of Python libraries and tools, I need Ubuntu.  Actually, I'll be using Lubuntu, since it is light weight, supports the same software, and works much better with the old NVIDIA GeForce 200 graphics cards that are in the machines (the Unity desktop wants to make use of the 3D graphics features of the card, which looks awful and crashes with both the free nvidia driver and the proprietary ones from nvidia.  Lubuntu is perfectly happy in 2D, and the free driver works very well with it.

The Plan

  1. Create a VirtualBox VM with Lubuntu and all the software I plan to use in the lab installed on it.
  2. Install  Remastersys on this VM and use it to create a custom installation disk.
  3. Use the custom installation disk thus created to install Lubuntu along side Windows on the lab workstations.

Step 1: The VM

I created a new VirtualBox VM and installed Lubuntu 14.04 (64 bit) on it.  After running all the updates, I did the following:
  • Installed virtualbox-guest-utils. This package will not be needed on the lab workstations, but without it screen resolution on the VM is limited to 640x480, making it too difficult to work with.
  • Added this PPA and then installed remastersys-gtk. I figured better to test this early, since if it doesn't work there is not point in doing the rest of the setup.
  • Installed gnome-screenshot and gimp. I added these two now to be able to take screenshots of the remastersys screens and edit them.
  • From Synaptic package manager I enabled the "Canonical Partners" repository and installed adobe-flashplugin.
  • Added the ubuntugis-unstable PPA.
After this I tried using remastersys, since if that doesn't work, there is no point in continuing.  I'm glad I did this early in the process, since I encountered a problem after launching System Tools -> Remastersys and selecting dist:
I found an Ubuntu forum post here, which suggested creating an empty lightdm.conf file in /etc/lightdm would fix the problem. It did! The process completed and I found a custom-dist.iso file in the /home/remastersys/remastersys directory.  I installed successfully with the custom-dist.iso, with one caveat -- when I tried to select "Encrypt my home folder" during the install, it crashed.  Encrypting the home directory does not appear to work with the Remastersys created iso, so I'll just make sure not to select that option, and to use the process described here to encrypt home directories later.

The last thing I needed by way of infrastructure on this VM is Grub Customizer, which will make it easy to edit the bootloader menu to have it boot to Windows by default (it pains me to do that, but I'm sharing the lab this year so I have to play nice ;-)


Step 2: Adding Software

I know I'll continue to add other software as the year goes on and I find other things I need, but for the first go round, here is a list of packages I know I'll want, that I have just installed on the VM:
  • python3-pip
  • idle3
  • python3-matplotlib
  • python3-sphinx
  • python3-pep8
  • python3-bs4
  • python3-w3lib
  • python3-scipy
  • python3-pyqt5
  • ipython3
  • ipython3-notebook
  • python3-termcolor
  • python3-cairo
  • python3-paste
  • python3-cherrypy3
  • python3-flask
  • python3-bottle
  • spyder3
  • inkscape
  • gftp
  • vim
  • vim-gtk
  • most
  • openjdk-7-jdk
  • sqlite3
  • spatialite-bin
  • spatialite-gui
  • qgis
  • grass
  • pgadmin3
  • postgresql-client
I also installed SymPy, using the command: 'sudo pip3 install sympy', since sympy is not in the package repository. Finally, I installed Google Chrome using the installer from here and Opera using the installer here.

With all this software installed, I ran Remastersys again to make an installation iso disk image.  Tomorrow I'll try it out in the lab...

Sunday, September 20, 2015

Web-based GIS Assignment 2

With pgAdmin running on my Ubuntu 14.04 desktop, and PostGIS setup on an Internet virtual machine (running Ubuntu 14.04 server), I connected pgAdmin to the server by double clicking on the server name (NYC) in the list.  It connected fine, but not without giving me a warning:
A quick Google search brought me to this Ask Ubuntu link:
which says I need to install postgresql-contrib.  Here goes:
$ sudo aptitude install postgresql-contrib
It installed a single package without incident.  After clicking the "Fix It!" button, I can now connect to the remote server without warning.

To install pgAdmin on my desktop at home, I ran:
$ sudo aptitude install pgadmin3
it returned that:
The following NEW packages will be installed:
  pgadmin3 pgadmin3-data{a} pgagent{a} postgresql-client{a}
  postgresql-client-9.3{a} postgresql-client-common{a}
While it is no surprise that pgAdmin would depend on postgresql-client (which has the command line psql program), it is convenient. For one thing, it makes it easier to test out the connection to the remote database server, since the psql command:
$ psql -h [server name] [database name]
Is easy to remember.  I still don't know off the bat how to fill in the fields of the pgAdmin connection screen without looking things up.  Here is a screen shot of the New Server Registration screen:
I know what to put in the Host, Username, and Password fields, but I'm not sure about Name and Maintenance DB.  Is Name the name of the database, or is that Maintenance DB?

This documentation page proved most helpful:
I put "nyc" (the name of the database we are using for class) in both the Name and Maintenance DB fields, and it connected to the server without incident.

Setting Up the Local Virtual Database Server for Testing

My next task, assigned to me at our last Code for NoVA meet up, is to look into loading OpenStreetMap (OSM) data into a PostGIS database. As cool as it is having a remote server out on the web running a database server 24/7 that I can connect to whenever I want, I'm not going run new "experiments" on that machine.  That's where local VirtualBox VMs come in handy.

I described my first attempt at setting up a PostGIS server in a previous post. As I've learned since, starting out with the command:
$ sudo aptitude install postgresql-9.3-postgis-2.1
is the best way to get going, since it installs postgresql itself and most everything else you need to get started. I still have the VM I made back then, so now I'm going to copy over the nyc database to it and configure it for remote access.

I'll repeat the process I used to move the nyc database from the VirtualBox VM we were given in our Web-based GIS class, described in my previous post, only this time I'll export it from my database server, so that I can skip the steps where I had to rename the owner of the database.

Here is what I did:
username@local_machine:~$ ssh [database server name]
username@dbserver:~$ pg_dump -c nyc > nyc.sql
username@dbserver:~$ exit
username@local_machine:~$ scp [database server name]:nyc.sql .
I could export the database as me, since I've been setup as a database user. I don't remember whether I did that on the VirtualBox VM, so now is a good time to learn some more PostgreSQL administration.  Here is a documentation page which has what I need:
Trying to start psql on the VirtualBox server shows that I didn't add my user as a database user:
$ psql
psql: FATAL:  role "[username]" does not exist
Time to fix that:
username@postgis:~$ sudo -i
[sudo] password for username:
root@postgis:~# su - postgres
postgres@postgis:~$ createuser --superuser username
postgres@postgis:~$ exit
root@postgis:~# exit

username@postgis:~$ createdb nyc
username@postgis:~$ psql nyc < nyc.sql
It successfully ran the script and populated the nyc database. I could then run:
username@postgis:~$ psql nyc
and connect to the nyc database and run queries.

Now to enable remote connections:
username@postgis:~$ sudo vi /etc/postgresql/9.3/main/pg_hba.conf
and changed this line:
host  all  all  md5
host  all  all     md5
username@postgis:~$ sudo vi /etc/postgresql/9.3/main/postgresql.conf
and changed this line:
#listen_addresses = 'localhost'
listen_addresses = '*'
username@postgis:~$ psql nyc
nyc=# alter user [user] with password '[password]';
This is mostly a repetition of information in my previous post, but since I want to learn it, it bares repeating.

Finally, I'll connect from my desktop machine to the "remote" VirtualBox server, which has IP address on my home network:
username@localmachine:~$ psql -h nyc
psql (9.3.9)
SSL connection (cipher: DHE-RSA-AES256-GCM-SHA384, bits: 256)
Type "help" for help.

Excellent!  I'm all set to explore importing OSM data into my database.