Sunday, April 17, 2016

Moving an ArcGIS File Geodatabase to QGIS

I am taking GGS 553: Geographic Information System this semester at part of my graduate studies at George Mason University.  In a previous post I described how I ended up in this Geographic Information Science graduate certificate program, which I have now been pursuing for almost 2 years.  GGS 553 is a required course, and the first one in the program that has required me to use proprietary software, since much of the course is focused on learning to use ArcGIS.

I am both philosophically and ethically opposed to proprietary software, since it runs dead against the expansion of our shared cultural space, which I believe is vital to the survival of our species. This is a required course, however, and in the large scheme of things I am willing to compromise when I need to. I like to think of it as dancing with the devil, learning the devil's moves in order to be able to freely out dance him in the future. In this case that will mean applying what I learn in GGS 553 to mastering QGIS, the free software alternative to ArcGIS. I had intended to try to do each of our assigned labs this semester in both ArcGIS and QGIS, but when I found it difficult enough just to complete them on time in ArcGIS, I gave up on that idea after the first week.

This week we have a sort of half size assignment, so I thought I would use the extra time available to see if I could do it in QGIS.  The first challenge will be to load the project data into QGIS.  We were given the data in ArcGIS's file geodatabase format. QGIS can not yet read and write to this format directly, but there are tools available to convert it into PostGIS, with which QGIS can work well.

Last Summer I wrote a blog post documenting how I setup a PostGIS server on Ubuntu 14.04.  Since this year I am also needing to learn RHEL, I'll use this guide to setup the server on the little Centos 7 server I have at home for just such purposes, and then connect to it from QGIS running on my Ubuntu desktop.

Installing a PostGIS Server on Centos 7

$ sudo yum install postgis postgresql-server postgresql-contrib
$ sudo postgresql-setup initdb
$ sudo -i -u postgres
$ psql
postgres=# \password postgres
Enter new password: 
Enter it again: 
postgres=# \q
$ exit
$ sudo vi /var/lib/pgsql/data/pg_hba.conf

Change this line (near the bottom):

host    all             all             127.0.0.1/32            ident

to this:

host    all             all             0.0.0.0/0               md5

Next allow database connections from outside:

$ sudo vi /var/lib/pgsql/data/postgresql.conf

Change:

#listen_addresses = 'localhost'

to this:

listen_addresses = '*'

Create a new database user with superuser privileges:

$ sudo su - postgres
$ createuser --superuser [user]
$ psql -c "ALTER ROLE [user] PASSWORD '[password]'"
$ exit

Then as that user create the database and add gis extensions:

$ createdb webster
$ psql -d webster -c 'CREATE EXTENSION postgis'

Then after copying over the Webster.db directory containing the file geodatabase, I ran:

$ ogr2ogr -f "PostgreSQL" PG:"dbname=webster user=[user] password=[password]" Webster.gdb

After which I connected my desktop QGIS to the PostgreSQL server running on my little household server and loaded the three layers I found there:

Resources

Thursday, March 31, 2016

Software Management with YUM

YUM (Yellowdog Updater, Modified) is the package management tool used on Red Hat Enterprise Linux and its derived versions, CentOS and Scientific Linux. It acts as a front end to the RPM Package Manager (RPM), and is used to install, remove, and update software on Red Hat based systems.

I first encountered YUM when installing Yellow Dog Linux on PowerPC based Macintosh computers back at the dawn of the 21st century.  When I switched over to Debian based GNU/Linux systems with the release of Ubuntu in 2004, I completely lost touch with the RPM world until my Spring semester Linux System Administration course's pursuit of RHCSA certification brought me back into the fold.

I am writing this post to use as a handy list of the most common things I need to do when managing software:
  1. Update the software on the system
    $ yum check-update
    $ sudo yum update package_name
    $ sudo yum update [to update all packages]
    $ sudo yum group update group_name
    
  2. List all the currently installed software
    $ yum list installed
    $ yum list installed "global expression"
    
  3. Search for available packages
    $ yum list available "global expression"
    $ yum search term...
    
  4. Display information about a package
    $ yum info package_name
    
  5. Install a new package
    $ sudo yum install package_name
    
  6. Remove an existing package
    $ sudo yum remove package_name
    
  7. List the current repos
    $ yum repolist
    $ yum repolist -v
    
That covers the basics. I also need to learn how to clean up the cruft that accumulates over time as a system is run, in Debian land the kind of thing that would be done with $ sudo apt-get autoremove. It seems that in RPM space that is accomplished with the package-cleanup utility, so I'll look into that.

Resources

Saturday, March 19, 2016

Centos Command-line Tricks and Tips - Getting Rid of the Terminal Beep

Getting Rid of the Terminal Beep:

My terminal was making an annoying beeping (more like a swoosh beep, actually) every time it couldn't match a tab completion.  I like to listen to music while I work, so this was really driving me crazy.  All I needed to do to stop it was to run:
$ echo 'set bell-style none' >> ~/.inputrc
which appends 'set bell-style none' to the .inputrc file in my home directory.  .inputrc didn't exist in my home directory (I checked before running the command), so running this command created it.
After exiting the terminal and starting another, the terminal maintained the silence I wanted it to ;-)

Friday, February 19, 2016

Setting Up a Centos Router - Part 1


In order to run the kind of experiments we will need to run to really learn proper GNU/Linux system administration, we need our own "safe space" in which to play.  In previous years when I had students with the level of skills our ITN 170 group is quickly acquiring, I always used one of our machines as a NAT Router so that we could isolate our own network traffic and setup custom services within our private network space.

The basic idea is captured in the following illustration.
What is required is a machine with two NICs (represented here by Tux) - one which connects to the outside network and the other which connects to the local network.

Setup Process

Here is what I did to setup a basic router using an old desktop PC:

  • Did a minimal install of CentOS 7 on a machine with two NICs, connecting one of the NICs to the outside network and activating this connection using DHCP on the host network during the installation process.
  • Ran yum update after installation to make sure I had the current software.
  • Ran yum install yum-utils vim to get vim and the package-cleanup utility. I then ran package-cleanup --oldkernels --count=1 to remove all but the current kernel package.
  • I ran ip addr and got back information on three network interfaces:
    1. lo - the loopback interface or localhost, with its 127.0.0.1/8 network address.
    2. enp0s25 - the NIC on the motherboard which I had activated with DHCP during installation.
    3. enp3s0 - the addon NIC that was not configured during installation. It had the following information:
      enp3s0:  mtu 1500 qdisc pfifo_fast state UP qlen 1000
          link/ether 00:15:17:20:b6:e6 brd ff:ff:ff:ff:ff:ff
      
  • I edited /etc/sysconfig/network-scripts/ifcfg-enp3s0 adding the following:
    TYPE="Ethernet"
    BOOTPROTO="static"
    IPADDR="192.168.114.1"
    NETMASK="255.255.255.0"
    GATEWAY="x.x.x.x"  (place your gateway adress here)
    
I used the resources linked below to try to enable IP routing and NAT, but I was not successful in getting it to route.  I have a laptop running Centos 7 connected to the router machine.  Before attempting this setup I had installed ClearOS on the router and got it to route for the laptop with a setup process using ClearOS's web interface.  An experienced friend of mine shamed me into removing this, however, by telling me he would never hire a sysadmin who only new how to set this up using a web interface.

So for now I have assigned two of my students to continue looking into it, and I'll get together with that friend who shamed me into this to get his assistance on Tuesday if we haven't figured it out by then.

To be continued...

Resources

Monday, February 8, 2016

Text Processing and Unix History

Preparing for the RHCSA certification is turning out to be a heap of fun! Despite more than 20 years as a free software activist and personal user of GNU/Linux systems for all my personal computing, and despite being a computer science teacher during that same time, there are a wide range of basic Unix CLI skills that I only scratched the surface of in all that time (shame on me!).

Preparing for the RHCSA is providing the opportunity to address that deficit at long last.  Chapter 4 of the book we are using in class to study for the certification is titled "Working with Text Files". The most enjoyable thing about this investigation into Unix text file processing is the view it provides into Unix history.

In the beginning there was eded begat ex, and ex begat vi... Along the way we got cousins grep and sed too.  Since grep, sed, and vi are part of the Unix admin's toolset, I want to learn to use them at least well enough to be able to help prepare students (and myself) for the RHCSA certification and to be able to present them well to future students in my ITN 170: Linux System Administration class.

Since in the beginning there was ed, let me start with that.  I found a very nice blog post, Actually using ed, which I found to be a wonderful introduction to this tool.  I set myself the task of using ed to create a list of fruits in a file named fruits.txt.  The first thing I found out was that trying:
$ ed fruits.txt
did not create the file for me, instead returning a "No such file or directory" error.  So I did the following, which worked:
$ touch fruits.txt
$ ed fruits.txt
0
a
apples
bananas
pears
apricots
kiwis
blueberries
oranges
peaches
cranberries
blackberries
pinapples
teaberries
.
w
110
q
$
After that, I ran $ cat fruits.txt, and saw that everything was as I wanted it:
apples
bananas
pears
apricots
kiwis
blueberries
oranges
peaches
cranberries
blackberries
pinapples
teaberries
Now if I want an alphabetical listing of the fruits in my list, I can run:
$ grep berries fruits.txt | sort
and see this:
blackberries
blueberries
cranberries
teaberries
RegexOne is a nice, interactive tutorial for learning basic regular expressions.  I wanted to do all the exercises using grep on the command-line as well, and in the process setup a new github repo for resources related to our RHCSA study, here.

Next I wanted to learn sed.  Sed - An Introduction and Tutorial by Bruce Barnett is a wonderful tutorial.  With so much awful document out there, it is great to find something written by someone with a grasp of how people actually learn.

Using the fruits.txt file I created with ed, I ran $ sed s/berries/cherries/ fruits.txt and got:
apples
bananas
pears
apricots
kiwis
bluecherries
oranges
peaches
crancherries
blackcherries
pinapples
teacherries
Since sed uses the same substitution syntax that vim uses, learning it will be a big help in becoming a more effective vim user as well.

Saturday, February 6, 2016

QGIS Delivers Functionality and Freedom

I am taking a graduate course this semester, GGS 553 - Geographic Information System, which is required for the Graduate Certificate in Geographic Information Sciences program that I am hoping to complete.  I like the text book we are using for class, and greatly enjoyed the first lecture.  What I am not happy about is that the labs which will make up a large part of the course assignments require the use of proprietary software, specifically ArcGIS, and then by extension, the Windows operating system on which it runs.

I have been a free software activist for more than 20 years. Software for GIS makes it especially easy to state why I believe so strongly in software freedom. To put it simply, I believe software should be part of humanity's shared cultural heritage, and that all efforts to turn it instead into a commodity are immoral.

Installing ArcGIS made this painfully clear to me.  In the first place, using it required that I use a non-free operating system, so I am running Windows just so that I can use ArcGIS.  Going through the gymnastics (registering an on-line account, figuring out where to enter the product code after missing it the first time through the installation, etc.) required to establish that I was "authorized" to use the commidified resource was most unpleasant. It rubs me deeply the wrong way to see human creativity misspent making the world a worse place rather than a better one.

No matter.  I have to do it to complete this required course, so I am determined to make the best of it.  What that means to me is keeping in mind the well known quote from Sun Tzu,
"Know your enemies and know yourself, you will not be imperiled in a hundred battles."
So I'll count learning ArcGIS as knowing my enemy, and time permitting, I will do each lab assignment in QGIS in parallel.

The first thing I wanted to do was to install the latest QGIS on my Ubuntu 14.04 desktop.  To do this, using this web page as a guide, I added the following to the end of my /etc/apt/sources.list file:

# For QGIS 2.12
deb http://qgis.org/ubuntugis trusty main
deb http://ppa.launchpad.net/ubuntugis/ubuntugis-unstable/ubuntu trusty main


Then I ran:
$ sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-key 3FF5FFCAD71472C4
$ sudo aptitude update
$ sudo aptitude install qgis
This is a much easier process than installing ArcGIS. QGIS also runs much faster than ArcGIS, and on the operating system I choose, not the one chosen for me.

It also seems that the wonderful folks who have developed QGIS have modeled its UI after the non-free standard, so the lab notes describing ArcGIS helped me understand QGIS as well. QGIS's Browser is the equivalent of ArcGIS's ArcCatalog. Here is the QGIS Browser showing the shape files from my first lab:
The QGIS Desktop functions like ArcGIS's ArcMap.  Here is QGIS Desktop with my Lab 1 shapefiles in a map:
So far, so good.  I was able to answer all the lab questions using QGIS with the given data, and I learned new things about QGIS through doing the ArcGIS lab exercises.

Thursday, January 28, 2016

Creating a Shared Partition Between Ubuntu and Scientific Linux

Now that I've removed Windows from my desktop computer at work, and installed Scientific Linux in its place (note: it was, Centos 7.2 with LVM partitions, but now it is Scientific Linux 7.1 with standard partitions), I decided I needed a partition that could be shared between the two distros for large user data.

For example, I have 12 Gigabytes in my Music directory, and a number of VirtualBox hard disk images (at 20 to 30 Gigabytes each) that I would like to access from both OS's.  So my plan is to create a new partition which I will mount on /media/share on both Ubuntu and Scientific Linux.  Then I'll make symbolic links from /home/[username]/Music to /media/share/Music from each home directory.

Before I could create a new partition, I needed to shrink one of my existing partitions to free up space.  Here is how the partitions looked when I started:
and here is what they looked like after shrinking /dev/sda3, growing /dev/sda4 and inserting /dev/sda7 into the new space inside it:
I made the change by booting my computer from an Ubuntu 14.04 Live DVD and running GParted. It took about 20 minutes to shrink my home partition, but it worked without incident.

The next step is to add a mount for the new /dev/sda7 partition.  It's been several years since I played around with mounting partitions, but I still remembered that it involved editing the /etc/fstab file and adding the device and the mount point.  So I loaded my /etc/fstab file and noticed something had changed since I last looked at it:
# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
# / was on /dev/sda2 during installation
UUID=4ae26245-fe59-40d2-a380-2c2de57b652b /               ext4    errors=remount-ro 0       1
# /home was on /dev/sda3 during installation
UUID=5c4e86f6-1d94-4eab-9931-5d3aa29e1583 /home           ext4    defaults        0       2
# swap was on /dev/sda1 during installation
UUID=bba8f235-d8b0-4131-9a3e-ec286c3b3837 none            swap    sw              0       0
I was completely unfamiliar with UUID, and had been expecting to see device names (like /dev/sda3 etc.) instead.  I bit of searching led me to several links:
Running $ sudo blkid gave me this:
 /dev/sda1: UUID="b70d7272-e47f-426a-a979-5417bb2f7801" TYPE="swap"
/dev/sda2: UUID="4ae26245-fe59-40d2-a380-2c2de57b652b" TYPE="ext4"
/dev/sda3: UUID="5c4e86f6-1d94-4eab-9931-5d3aa29e1583" TYPE="ext4"
/dev/sda5: UUID="76f8dd02-3671-49ba-b75a-da6d8bb65b19" TYPE="ext4"
/dev/sda6: UUID="e7e58de9-a748-4500-b300-ae1ca10f2056" TYPE="xfs"
/dev/sda7: UUID="44244888-7000-49bf-8ac8-2c32e2f73eb5" TYPE="ext4" 
which I used to add the following line to /etc/fstab:
UUID=44244888-7000-49bf-8ac8-2c32e2f73eb5 /media/share ext4 defaults
and then ran:
$ sudo mkdir /media/share
$ sudo mount -a
which mounted /dev/sda7 on /media/share. Next I moved all my music files to /media/share/Music, deleted the Music directory in my home directory, and replaced it with a symbolic link (note: run from my user's home directory):
$ ln -s /media/share/Music Music
I started Rhythmbox and it worked as if nothing had changed.

Rebooting into Scientific Linux, I added the same line to /etc/fstab and ran the same mkdir and mount commands, then removed my still empty Music directory, made the same sym link, and voila, I had access to all my music from Scientific Linux (after installing Rhythmbox, that is).

For VirtualBox VM sharing, things are a bit more complicated.  Ubuntu makes installing VirtualBox trivial, since it is in the main repository, but on Ubuntu 14.04 version 4.3 is what you get.  On Scientific Linux I installed version 5.0 using the instruction from an earlier post.

Fearing there might be meta-data conflicts between the two versions, but feeling confident the virtual hard disk image files (.vdi) could be shared between them without conflict (since I regularly copy these files back and forth between distros without problems), I did the following:
  1. On the Ubuntu side, I moved my entire VirtualBox VMs directory from my home directory to /media/share and sym linked to it as I had done with the Music directory.
  2. On the Scientific Linux side, I kept VirtualBox VMs in my home directory, using sym links only for the virtual disk image files.  For example, from inside $HOME/VirtualBox VMs/Server1 I ran:
    $ ln -s /media/share/VirtualBox\ VMs/Server1/Server1.vdi Server1.vdi
This worked nicely.  Just for fun, I ran yum update on Server1 launched from VirtualBox 5.0 on Scientific Linux, then rebooted into Ubuntu 14.04 and relaunched Server1 from there, seeing changes I had made.

Finally, after noticing that VirtualBox-4.3 was available for Scientific Linux 7.1, I ran # yum remove VirtualBox-5.0 and then # yum install VirtualBox-4.3, made the VirtualBox VMs directory in my home directory a sym link to /media/share/VirtualBox VMs and quickly added all the VMs back.  Now even the VMs with the VirtualBox extensions installed (for full screen GUI and auto mouse capture) work on both OS's.