Page 1 of 1 (3 posts)

  • talks about »
  • case study

Tags

Last update:
Thu Oct 30 10:50:07 2014

A Django site.

QGIS Planet

How the West was Won (or QGIS rocks!)

gazetteer search list!<tl><dr>

Local Authorities work with people and assets. Most people have an address and assets are mostly located somewhere. As my old geography teacher used to say, “everything has geography, geography is everything”. For people who work in Local Authorities being able to find an accurate and up-to-date address easily is key to being able to deliver a service quickly and efficiently. If GIS officers had their way even more of the Council back office systems across the country would hook up to the corporate GI database and do cool stuff with spatial information.

At the recent AGI GeoCom and FOSS4G conferences in Nottingham during Maptember QGIS 2.0 was launched and garnered a lot of attention from people interested in finding solutions to save money and time and still deliver great results. I came away with a resolution to push for the use of QGIS at work in an attempt to get a desktop GIS onto more desktops while not breaking the bank. I had also met Simon Miles (@geosmiles) and eagerly followed his talk of creating a hybrid GI infrastructure using a mix of open-source and proprietary software.

QGIS fitted the bill with its extensive range of functionality, growing list of available plugins, super easy connectivity to a range of data sources including Oracle, SQL Server and PostGIS, all of which we use. The one thing it was lacking was a search plugin to let the users search the corporate address gazetteer for postal address and street names. Our existing GIS has such a plugin and it works well but not with the imminent upgrades. There had to be something that could connect to the gazetteer table in our PostGIS database, search and display an address.

qgis gazetteer pluginEnter the QGIS Gazetteer Plugin developed by Astun Technology, Nathan Woodrow and Matt Walker. Add some keen peeps from the UK QGIS User Group and some social networking and stuff begins to happen.

After downloading and installing the plugin I soon hit a brick wall – or rather a firewall – the security at work wouldn’t let the Yahoo and Geonames gazetteers work and we don’t have Astun’s iShareGIS software stack. I delved into the code to see if I could work out what was happening and hit another wall – how does this Python stuff work? Nought for two after one over.

After some chat on the UK QGIS Google+ group I found some people in Local Government across the UK who have managed to get the plugin to work and work well.  Simon Miles (Royal Borough of Windsor and Maidenhead), Kevin Williams (Neath-Port Talbot) and Matt Travis (Dartmoor National Park) rallied to my call and soon I had some code in my grubby mitts.  An hour after that I had a working plugin searching our address and street gazetteers, postcodes and planning applications.  Listed buildings, building warrants and others soon followed.  I shared the plugin with some of our power users and they were very impressed and have even started asking for enhancements.

This whole process is a great example of how a User Group can work when coupled with FOSS4G fans, social media, open-source software and a common problem to solve.  Being able to present a working solution to users and managers in just a few days certainly strengthens the case for using QGIS alongside the traditional GIS tools and maybe, in time, replacing them.

What follows is a rather more technical description of the installation and configuration of just about the single most useful application a Council officer could ask for. If you want the short version and can do the tech stuff then go to https://github.com/mixedbredie/qgis-gazetteer-search and get it, otherwise hang in there to the end…

What you need:

1. QGIS 2.0 installed

2. the QGIS Gazetteer plugin installed

3. a PostGIS database with some gazetteer tables

4. an Apache2 web server

how-it-works

1. QGIS

Install QGIS 2.0 – I used the standalone installer from http://www.qgis.org/en/site/forusers/download.html

2. QGIS Gazetteer Plugin

Install the QGIS Gazetteer plugin from https://github.com/AstunTechnology/QGIS-Gazetteer-Plugin by downloading the zip file and extracting into your .qgis2 folder. This is located at

C:\Documents and Settings\<Username>\.qgis2\python\plugins\ (Windows XP) or 
C:\Users\<Username>\.qgis2\python\plugins\ (Windows 7+).

Make sure that the extracted directory is called “gazetteersearch”. Matt Walker (@_walkermatt) has upgraded the plugin to work with QGIS 2.0 and added a few enhancements like an OpenStreetMap search.

In the “gazetteers” sub directory there a “config.ini” file which needs to be edited to list the datasets we want to search. I’ll show how to connect to the address and street gazetteers and a table of postcode areas.

I deleted the GeoNames, Yahoo and AstunTechnology lines and inserted the following:

[LLPG]
gazetteer:llpg
[LSG]
gazetteer:lsg
[Postcode]
gazetteer:postcodes

Create three files in the gazetteers folder – llpg.py, lsg.py and postcodes.py

In the llpg.py file put:

import json
import collections
url = "http://10.135.1.69/cgi-bin/llpg_pg.py"
params = {
    'address': '##searchstring##'
}
def parseRequestResults (data):
json_result = json.loads (data)
    for item in json_result:
        result = collections.namedtuple ('Result', ['description','x','y','zoom', 'epsg'])
        result.description = item['address']
        result.x = float (item['easting'])
        result.y = float (item['northing'])
        result.zoom = 1250
        result.epsg = 27700
        yield result

In the lsg.py file put:

import json
import collections
url = "http://10.12.345.678/cgi-bin/lsg_pg.py"
params = {
    'address': '##searchstring##'
}
def parseRequestResults (data):
json_result = json.loads (data)
for item in json_result:
        result = collections.namedtuple ('Result', ['description','x','y','zoom', 'epsg'])
        result.description = item['address']
        result.x = float (item['easting'])
        result.y = float (item['northing'])
        result.zoom = 2500
        result.epsg = 27700
        yield result

In the postcodes.py file put:

import json
import collections
url = "http://10.12.345.678/cgi-bin/postcodes_pg.py"
params = {
    'postcode': '##searchstring##'
}
def parseRequestResults (data):
    json_result = json.loads (data)
    for item in json_result:
        result = collections.namedtuple ('Result', ['description','x','y','zoom', 'epsg'])
        result.description = item['postcode']
        result.x = float (item['easting'])
        result.y = float (item['northing'])
        result.zoom = 2500
        result.epsg = 27700
        yield result

You need to set the correct server IP address in the URL and link to the correct file in the cgi-bin. You can also set the zoom scale for the results and I use ESPG:27700 as everything we have is in British National Grid.

If you restart QGIS and enable the plugin you’ll see there are now four search options in the drop-down list. Right, now to make sure PostGIS has the correct information.

3. PostGIS Database

The installation and configuration of PostgreSQL and PostGIS is more than this post is going to address but you’ll need to have one. With some tables in it with some data that can be searched. We load our address and street gazetteers into PostGIS on a daily basis and these tables are used by the plugin for searching and displaying records. Our tables have a UPRN (Unique Property Reference Number) field and an ADDRESS field with a full address in it (name number street town locality postcode). The address gazetteer has a point geometry. The street gazetteer has a USRN (Unique Street Reference Number) and a NAME field with the full descriptive name of the street in it. It has a line geometry. Almost any table can be searched as long as it has a unique id, a text field for searching, and some geometry.

Our address gazetteer already had the correct fields in so nothing had to be done but the street gazetteer needed some changes. I created a view of the streets to change the name of the fields to fit with what the plugin was expecting:

CREATE OR REPLACE VIEW angusdata.search_lsg_streets AS
SELECT a.usrn AS uprn,
btrim(pg_catalog.concat('usrn:', btrim(a.usrn::text), ' ', btrim(a.street::text), ' ', btrim(a.locality::text), ' ', btrim(a.town::text))) AS address,a.geometry
FROM lsg_streets a;

This changes the USRN field to UPRN and concatenates USRN, STREET, LOCALITY and TOWN to create an ADDRESS field and then adds the geometry.

The Postcodes table has polygon geometry and a postcode field and the python script that creates the web service has been tweaked to work with this. The next section will explain how to create the web services on Apache2.

4. Apache

I have a local Apache2 webserver running on my PC but it’s pretty easy to get one set up on server if you chat to the right people :-) so get one installed (http://www.apachehaus.com/cgi-bin/download.plx). The plugin uses the web services running in the Apache2 cgi-bin to connect to the PostGIS database to query the gazetteers. You’ll also need to have Python (http://www.python.org/download/releases/2.7.5/) installed on the same machine. And the Python psycopg2 module (http://www.stickpeople.com/projects/python/win-psycopg/).

First up, the Apache2 server needs to be configured to allow use of the cgi-bin. Edit the httpd.conf file and make the following changes.

Uncomment the following lines to enable the modules:

LoadModule access_compat_module modules/mod_access_compat.so
LoadModule cgi_module modules/mod_cgi.so
LoadModule proxy_fcgi_module modules/mod_proxy_fcgi.so
LoadModule rewrite_module modules/mod_rewrite.so

Set the server name to its IP address:

ServerName 10.12.345.678:80

Allow access to the server:

<Directory />
  Options Indexes FollowSymLinks ExecCGI
  AllowOverride All
  Order allow,deny
  Allow from all
  Require all granted
</Directory>

Set the file that Apache will serve if a directory is requested:

<IfModule dir_module>
  DirectoryIndex index.html index.htm index.php index.php3
</IfModule>

Allow access to the CGI directory and enable it to handle python scripts:

<Directory "${SRVROOT}/cgi-bin">
  Options Indexes FollowSymLinks ExecCGI
  AllowOverride All
  Order allow,deny
  Allow from all
  Require all granted
  AddHandler cgi-script .cgi .py
</Directory>

This is a pretty basic configuration and it could probably be improved but it works on my system.

Then in the CGI-BIN directory create some files, one for each web service and to match the gazetteers listed in the config file. I created three files called llpg_pg.py, lsg_pg.py and postcodes_pg.py.

In the llpg_pg.py file the SQL query selects the records that match the search string and uses the point geometry of the records to return to the plugin.

#!D:/Python27/python.exe
# -*- coding: UTF-8 -*-
import cgi
import json
import psycopg2
sql = """select a.uprn, a.address, ST_X(a.geometry), ST_Y(a.geometry)
from angusdata.address_gazetteer a
where a.address ilike '%%' || (%(p_address)s) || '%%'
order by address"""
form = cgi.FieldStorage ()
connection = psycopg2.connect ("host='10.12.345.678' port='5432' dbname='postgisdb' user='username' password='password'")
cursor = connection.cursor ()
cursor.execute (sql, {"p_address": form["address"].value})
list = []
for record in cursor:
    data = dict (zip (["uprn", "address", "easting", "northing"], record))
    list.append (data)
print "Content-Type: application/json\n"
print json.dumps (list, indent = 4)

In the lsg_pg.py file the SQL query selects the records from the view and converts the line geometry to a centroid point.

#!D:/Python27/python.exe
# -*- coding: UTF-8 -*-
import cgi
import json
import psycopg2
sql = """select a.uprn, a.address, ST_X(ST_Centroid(a.geometry)), ST_Y(ST_Centroid(a.geometry))
from angusdata.search_lsg_streets a
where a.address ilike '%%' || (%(p_address)s) || '%%'
order by address"""
form = cgi.FieldStorage ()
connection = psycopg2.connect ("host='10.12.345.678' port='5432' dbname='postgisdb' user='username' password='password'")
cursor = connection.cursor ()
cursor.execute (sql, {"p_address": form["address"].value})
list = []
for record in cursor:
    data = dict (zip (["uprn", "address", "easting", "northing"], record))
    list.append (data)
print "Content-Type: application/json\n"
print json.dumps (list, indent = 4)

In the postcodes_pg.py file the SQL query turns the polygon geometry into a point geometry using the PostGIS ST_PointOnSurface function. The psycopg2 parameters have been tweaked to use the postcode tables fields.

#!D:/Python27/python.exe
# -*- coding: UTF-8 -*-
import cgi
import json
import psycopg2
sql = """select a.postcode, ST_X(ST_Pointonsurface(a.geometry)), ST_Y(ST_Pointonsurface(a.geometry))
from thirdparty.os_codepointpoly a
where a.postcode ilike '%%' || (%(p_postcode)s) || '%%'
order by postcode"""
form = cgi.FieldStorage ()
connection = psycopg2.connect ("host='10.12.345.678' port='5432' dbname='postgisdb' user='username' password='password'")
cursor = connection.cursor ()
cursor.execute (sql, {"p_postcode": form["postcode"].value})
list = []
for record in cursor:
    data = dict (zip (["postcode", "easting", "northing"], record))
    list.append (data)
print "Content-Type: application/json\n"
print json.dumps (list, indent = 4)

Make sure the #!shebang at the top of the python script has the correct path to your Python installation.

Check the Apache2 log files if there are any errors.

Right, we have QGIS installed with the gazetteer search plugin configured to use the new gazetteers.  The PostGIS tables and views are presenting the data in format required by the web services. The Apache2 server is configure to allow access and the python scripts are in the cgi-bin. Fire up QGIS, refresh the plugin and enter a search term and hit GO!  Awesome!  Mine works, at least.

None of this would have been possible without the UK QGIS group and the input from Matt, Kevin and Simon.  Big thanks to Matt Walker and Jo Cook for recent updates to the plugin.  Use the comments to tell us how to do it better.


Using QGIS in local government

Something that I always find interesting is how people are using different open source tools to get their work done.  This post attempts to outline how I/we are using QGIS at work for different projects.

Kerb mapping, condition, and defect pickup

This project is currently being done by a 67 year old foreman who has worked for the council for a very long time and has great knowledge of the town.   QGIS, with the main working layers stored in PostGIS, was setup so that he can:

  • Digitize kerb lines from aerial photos.
  • Split the existing kerb lines into segments depending on different asset rules.
  • Give each segment an overall condition rating.
  • Add defect points along the each kerb segment e.g. broken, lifted, etc,

Each defect point is snapped to the underlying  kerb line and chainages (distance along line) is generated using a update statement at the end of the project (could be done using a insert trigger if needed) using ST_line_locate_point(line, point).

Defect points coloured by risk captured against the kerb line

Overall QGIS has been great for this project.  The built in data entry forms have been a great help to allow fast and correct data entry. Each form has four drop downs all with present values and descriptions to aid in data entry.

Flood damaged claim maps

We recently suffered, like the rest of Queensland, some really major flooding which caused large amounts of damage to our road infrastructure. We got off pretty light compared to some places, nevertheless we still had a lot of damaged assets.  And so began the process of collecting data that could be used for state government funding claims.

Anyway, onto the QGIS bit.  QGIS was installed on one of the main engineers computers in order for him to make maps for each claim.  Having the ability for him to have one map window but multiple frames in the composer helped him to create multiple  views of the same data with ease.

In total there are 42 QGIS project files with a main project file which served the base layers to the other projects, using the cool Embed Layers and Groups feature.  This means any change in main base project was reflected up(down?) to the other projects next time they are opened.  The main project file has things like, property layer; normal road layers, with labels; road layer with roads for claims.   The other 42 projects have a filtered, and styled, road layer to only show roads in that batch, and its composers (print layouts).

Normally we would use MapInfo for this kind of thing but consider this: There are at least 3 print layouts per claim, each layout could have more then one map frame.  Now with MapInfo only being able to have a 1:1 ratio between the map window and the map frame in the layout you would need at least 3 map windows per claim.  Quick calc:

42 * 3 =  126+n map windows + 126 print layouts (n = extra map frames in layouts)

Each map window has its own copy of every layer, making change once apply every where changes hard.  This of course doesn’t apply to styles as they are stored in the .map (tab) file, but does for labels, style overrides, etc.   I’ll pass.

QGIS is no means perfect for printing or print layouts but the 1:N map window to map frame ratio worked really well for this project.  The styling options in QGIS also helped to change the display of the map depending on what was needed to be shown quickly, one even used the rule-based rendering.

You get the point.
Moving on.

Processing GPS photos with road chainages

This one I am quite proud of.  It’s nothing fancy but still saves a lot of time.  While not really QGIS only but a combination of QGIS+Spatialite it process GPS photos and assigns them a road name and chainage.

The issue: A large influx of GPS photos for the different flood damage projects and the need to process them quickly so that they got assigned to the correct road and chainage.  Now you can map GPS photos easy enough but then you still have to go to each one and assign a road name, chainage, and move it into the correct folder.  To hell with doing that by hand, this is why we invented GIS.

The result is a little (140 line) python script that:

  1. extracts the coordinates from each photos,
  2. finds the closest (within tolerance) road distance node (distance nodes are generated at 5m intervals along the road, around 800,000 in total for the whole shire),
  3. gets the road name, locality, and chainage for that node,
  4. creates a folder with that road name,
  5. renames the photo with {name} @ {chainage},
  6. moves it into the road name folder it is assign to.
  7. inserts a record for that photo into the spatialite database that can be viewed in QGIS.
The Spatialite database has a spatial index on the road distance nodes and with that in place it can process 148 photos in 8 seconds.  Not too bad at all. Now all we have to do to process the photos is stick them into a special folder and run process.bat.

Porting our planning scheme maps

I have been involved in creating, and maintaining, our planning scheme maps for the council.  It’s been a pretty fun project, apart from the constant moving target that is the state planning specifications, but I digress.

Planning scheme in QGIS

This project was done, and still is, in MapInfo. While there is nothing technically wrong with that, it has become a bit more of a pain to maintain then one would hope.  The planning scheme is not just one map but rather a series of different maps all with different scales and requirements.  I’m sure by now you can start to see the issues that can arise:

  1. No dynamic scale bar for layouts (not even a scale bar object rather just text and boxes made to look like a scale bar. With no group items feature moving these around is a pain).
  2. 1:1 map window to map frame means excessive map windows when the data is all the same with just different views.
  3. Legends don’t support ordering, adding items, or groups.
  4. With no embedding base maps feature like in QGIS it’s hard to change one thing and apply it to all the map windows/workspaces.
The specifications also ask for lines with symbols along them to show things like bikeways, footpaths etc, something that can’t be done in MapInfo, well it can by using the line style editor but I would rather stab myself in the eye.
The one thing I haven’t fully worked out how to do in QGIS yet is fully automate the printing process. Currently I open MapInfo using a batch file and pass it a workspace and MBX which prints the layouts and exits. I do this for each map type.    In QGIS I have a few options:
  1. Create a plugin that runs though each project and prints off its composers.
  2. Create a python script that runs from a batch file using qgis.core and qgis.gui QGIS python bindings.
  3. add a –code option to the command line of QGIS so that you could run: qgis.exe –noplugins –code “print.py”, which would open QGIS and run the python code and exit.
I’m yet to explore what option is the best for this project but I’ll get back to you on that.  Once I have the above issue sorted I plan on creating the maps in QGIS to see how it would turn out (time permitting)

Custom asset data collection program

This one would have to be my favourite.  I really love programming (most days), and being able to create our own data collection program using QGIS and MS SQL 2008 has been great.

While it is only very very young I’m already seeing some great potential.  Using an open source base (apart from MS SQL) has given us a lot of power, power to change stuff that we don’t like (which so far has been one minor bug), and the power to get exactly what we need.

I can’t talk about this project a lot as it is only very new and still only in design/testing/prototyping stage.

The main things for me are:

  • Ease of use. If I get asked how to do something over and over I have failed the users. And no 100 page training manuals.
  • Fast
  • No menus, or right-click menus! I’m a power user and even I hate navigating menus on a tablet.
  • Easy to build custom forms
  • Online/Offline syncing
  • Ease of use. Oh did I say this already!? Well it’s important.
  • Easy to configure by admins.
  • Limited use of dialogs. It’s NOT ok for an app to ask users to confirm 100 dialogs to do one thing.

Overall I think using QGIS and PyQt I can hit all the targets listed above quite well. In fact I know I can because I have already hit most of the them in the last couple of weeks.

Summary

So that is my list of QGIS uses in my local government situation, hopefully it wasn’t TL;DR and you found it interesting.  I’m sure there will be plenty more to add at the end of 2012.


Filed under: Open Source, qgis Tagged: case study, FOSSGIS, gis, local gov, Local government, mapinfo, Open Source, osgeo, qgis, Quantum GIS

QGIS and GRASS in Local Government Bushfire Hazard Mapping – A Case Study

QGIS and GRASS in Local Government Bushfire Hazard MappingIntroduction

The Southern Downs Regional Council (SDRC) is a small-to-medium sized local government in south east Queensland, Australia. The council region, mainly the southern part, suffers from major bushfires.  Bushfire is a real and present concern for the residents and landowners in the Southern Downs Region, and has resulted in the loss of life and property.

This project will allow the council and the people of the region to be more aware of the risk and to allow for better decision making in the future.

The Project
As bushfire is not only a problem for SDRC but also for the whole of Queensland, the state government requires that each local government identifies the bushfire hazard in their area via the State Planning Policy 1/03 Mitigating the Adverse Impacts of Flood, Bushfire and Landslide [1].  This kind of job would normally be done using consultants but was instead done by the council itself using a combination of QGIS and GRASS.

The GIS side of the project project was broken down into 6 main steps

  1. Slope assessment and mapping
  2. Aspect assessment and mapping
  3. Vegetation assessment and mapping
  4. Combining scores to identify the severity of bushfire hazard
  5. Field verification and qualitative assessment
  6. Final Maps

The use of QGIS and GRASS
QGIS, using the GRASS plugin, was selected as it provided the tools needed to complete the job and the interaction between QGIS and GRASS made it easy to process the raster maps and present them in a meaningful way to users.  SDRC uses MapInfo for its main GIS system, however MapInfo’s addons were not as powerful as GRASS GIS for raster processing.

The QGIS GRASS plugin was used to import 5 meter contours of the whole region into GRASS which were then converted into a contour raster map using r.surf.contour.  A slope and aspect map were then generated using r.slope.aspect from the raster contour map.  Categories groups were assigned to different slope and aspect ranges and given a risk sore.  Vegetation areas were also given different risk scores.  All the resulting raster maps were then combined using mapcalc and given a final risk hazard score.  The risk scores are then divided into three main categories: high; medium; and low.  

The final part of the process was field verification via the rural fire service. After the review process, QGIS was used to print the final maps for presentation.

As all GRASS commands can be run from the command line, all the commands that were needed to generate the bushfire hazard maps were recorded, for documentation purposes and for if the maps needed to be regenerated some time in the future.

Conclusion.
Overall QGIS, together with the GRASS plugin, provided a great experience and a great final outcome for the council doing their own bushfire hazard mapping.  The GRASS plugin provides a very easy to use interface to GRASS through QGIS.  As QGIS is able to open the GRASS raster format natively, integration is very seamless and maps can be made with ease.

The project won an encouragement award at the Queensland Planning Institute of Australia state planning awards in 2010 [2]

References
[1] http://www.emergency.qld.gov.au/publications/spp/
[2]http://digital.crowtherblayne.com.au/default.aspx?xml=crowther_pia.xml


Filed under: Open Source, qgis Tagged: bushfire, case study, GRASS GIS, Open Source, qgis, Quantum GIS

  • Page 1 of 1 ( 3 posts )
  • case study

Back to Top

Sponsors