Page 1 of 1 (9 posts)

  • talks about »
  • plugin

Tags

Last update:
Fri Oct 24 04:30:09 2014

A Django site.

QGIS Planet

How the West was Won (or QGIS rocks!)

gazetteer search list!<tl><dr>

Local Authorities work with people and assets. Most people have an address and assets are mostly located somewhere. As my old geography teacher used to say, “everything has geography, geography is everything”. For people who work in Local Authorities being able to find an accurate and up-to-date address easily is key to being able to deliver a service quickly and efficiently. If GIS officers had their way even more of the Council back office systems across the country would hook up to the corporate GI database and do cool stuff with spatial information.

At the recent AGI GeoCom and FOSS4G conferences in Nottingham during Maptember QGIS 2.0 was launched and garnered a lot of attention from people interested in finding solutions to save money and time and still deliver great results. I came away with a resolution to push for the use of QGIS at work in an attempt to get a desktop GIS onto more desktops while not breaking the bank. I had also met Simon Miles (@geosmiles) and eagerly followed his talk of creating a hybrid GI infrastructure using a mix of open-source and proprietary software.

QGIS fitted the bill with its extensive range of functionality, growing list of available plugins, super easy connectivity to a range of data sources including Oracle, SQL Server and PostGIS, all of which we use. The one thing it was lacking was a search plugin to let the users search the corporate address gazetteer for postal address and street names. Our existing GIS has such a plugin and it works well but not with the imminent upgrades. There had to be something that could connect to the gazetteer table in our PostGIS database, search and display an address.

qgis gazetteer pluginEnter the QGIS Gazetteer Plugin developed by Astun Technology, Nathan Woodrow and Matt Walker. Add some keen peeps from the UK QGIS User Group and some social networking and stuff begins to happen.

After downloading and installing the plugin I soon hit a brick wall – or rather a firewall – the security at work wouldn’t let the Yahoo and Geonames gazetteers work and we don’t have Astun’s iShareGIS software stack. I delved into the code to see if I could work out what was happening and hit another wall – how does this Python stuff work? Nought for two after one over.

After some chat on the UK QGIS Google+ group I found some people in Local Government across the UK who have managed to get the plugin to work and work well.  Simon Miles (Royal Borough of Windsor and Maidenhead), Kevin Williams (Neath-Port Talbot) and Matt Travis (Dartmoor National Park) rallied to my call and soon I had some code in my grubby mitts.  An hour after that I had a working plugin searching our address and street gazetteers, postcodes and planning applications.  Listed buildings, building warrants and others soon followed.  I shared the plugin with some of our power users and they were very impressed and have even started asking for enhancements.

This whole process is a great example of how a User Group can work when coupled with FOSS4G fans, social media, open-source software and a common problem to solve.  Being able to present a working solution to users and managers in just a few days certainly strengthens the case for using QGIS alongside the traditional GIS tools and maybe, in time, replacing them.

What follows is a rather more technical description of the installation and configuration of just about the single most useful application a Council officer could ask for. If you want the short version and can do the tech stuff then go to https://github.com/mixedbredie/qgis-gazetteer-search and get it, otherwise hang in there to the end…

What you need:

1. QGIS 2.0 installed

2. the QGIS Gazetteer plugin installed

3. a PostGIS database with some gazetteer tables

4. an Apache2 web server

how-it-works

1. QGIS

Install QGIS 2.0 – I used the standalone installer from http://www.qgis.org/en/site/forusers/download.html

2. QGIS Gazetteer Plugin

Install the QGIS Gazetteer plugin from https://github.com/AstunTechnology/QGIS-Gazetteer-Plugin by downloading the zip file and extracting into your .qgis2 folder. This is located at

C:\Documents and Settings\<Username>\.qgis2\python\plugins\ (Windows XP) or 
C:\Users\<Username>\.qgis2\python\plugins\ (Windows 7+).

Make sure that the extracted directory is called “gazetteersearch”. Matt Walker (@_walkermatt) has upgraded the plugin to work with QGIS 2.0 and added a few enhancements like an OpenStreetMap search.

In the “gazetteers” sub directory there a “config.ini” file which needs to be edited to list the datasets we want to search. I’ll show how to connect to the address and street gazetteers and a table of postcode areas.

I deleted the GeoNames, Yahoo and AstunTechnology lines and inserted the following:

[LLPG]
gazetteer:llpg
[LSG]
gazetteer:lsg
[Postcode]
gazetteer:postcodes

Create three files in the gazetteers folder – llpg.py, lsg.py and postcodes.py

In the llpg.py file put:

import json
import collections
url = "http://10.135.1.69/cgi-bin/llpg_pg.py"
params = {
    'address': '##searchstring##'
}
def parseRequestResults (data):
json_result = json.loads (data)
    for item in json_result:
        result = collections.namedtuple ('Result', ['description','x','y','zoom', 'epsg'])
        result.description = item['address']
        result.x = float (item['easting'])
        result.y = float (item['northing'])
        result.zoom = 1250
        result.epsg = 27700
        yield result

In the lsg.py file put:

import json
import collections
url = "http://10.12.345.678/cgi-bin/lsg_pg.py"
params = {
    'address': '##searchstring##'
}
def parseRequestResults (data):
json_result = json.loads (data)
for item in json_result:
        result = collections.namedtuple ('Result', ['description','x','y','zoom', 'epsg'])
        result.description = item['address']
        result.x = float (item['easting'])
        result.y = float (item['northing'])
        result.zoom = 2500
        result.epsg = 27700
        yield result

In the postcodes.py file put:

import json
import collections
url = "http://10.12.345.678/cgi-bin/postcodes_pg.py"
params = {
    'postcode': '##searchstring##'
}
def parseRequestResults (data):
    json_result = json.loads (data)
    for item in json_result:
        result = collections.namedtuple ('Result', ['description','x','y','zoom', 'epsg'])
        result.description = item['postcode']
        result.x = float (item['easting'])
        result.y = float (item['northing'])
        result.zoom = 2500
        result.epsg = 27700
        yield result

You need to set the correct server IP address in the URL and link to the correct file in the cgi-bin. You can also set the zoom scale for the results and I use ESPG:27700 as everything we have is in British National Grid.

If you restart QGIS and enable the plugin you’ll see there are now four search options in the drop-down list. Right, now to make sure PostGIS has the correct information.

3. PostGIS Database

The installation and configuration of PostgreSQL and PostGIS is more than this post is going to address but you’ll need to have one. With some tables in it with some data that can be searched. We load our address and street gazetteers into PostGIS on a daily basis and these tables are used by the plugin for searching and displaying records. Our tables have a UPRN (Unique Property Reference Number) field and an ADDRESS field with a full address in it (name number street town locality postcode). The address gazetteer has a point geometry. The street gazetteer has a USRN (Unique Street Reference Number) and a NAME field with the full descriptive name of the street in it. It has a line geometry. Almost any table can be searched as long as it has a unique id, a text field for searching, and some geometry.

Our address gazetteer already had the correct fields in so nothing had to be done but the street gazetteer needed some changes. I created a view of the streets to change the name of the fields to fit with what the plugin was expecting:

CREATE OR REPLACE VIEW angusdata.search_lsg_streets AS
SELECT a.usrn AS uprn,
btrim(pg_catalog.concat('usrn:', btrim(a.usrn::text), ' ', btrim(a.street::text), ' ', btrim(a.locality::text), ' ', btrim(a.town::text))) AS address,a.geometry
FROM lsg_streets a;

This changes the USRN field to UPRN and concatenates USRN, STREET, LOCALITY and TOWN to create an ADDRESS field and then adds the geometry.

The Postcodes table has polygon geometry and a postcode field and the python script that creates the web service has been tweaked to work with this. The next section will explain how to create the web services on Apache2.

4. Apache

I have a local Apache2 webserver running on my PC but it’s pretty easy to get one set up on server if you chat to the right people :-) so get one installed (http://www.apachehaus.com/cgi-bin/download.plx). The plugin uses the web services running in the Apache2 cgi-bin to connect to the PostGIS database to query the gazetteers. You’ll also need to have Python (http://www.python.org/download/releases/2.7.5/) installed on the same machine. And the Python psycopg2 module (http://www.stickpeople.com/projects/python/win-psycopg/).

First up, the Apache2 server needs to be configured to allow use of the cgi-bin. Edit the httpd.conf file and make the following changes.

Uncomment the following lines to enable the modules:

LoadModule access_compat_module modules/mod_access_compat.so
LoadModule cgi_module modules/mod_cgi.so
LoadModule proxy_fcgi_module modules/mod_proxy_fcgi.so
LoadModule rewrite_module modules/mod_rewrite.so

Set the server name to its IP address:

ServerName 10.12.345.678:80

Allow access to the server:

<Directory />
  Options Indexes FollowSymLinks ExecCGI
  AllowOverride All
  Order allow,deny
  Allow from all
  Require all granted
</Directory>

Set the file that Apache will serve if a directory is requested:

<IfModule dir_module>
  DirectoryIndex index.html index.htm index.php index.php3
</IfModule>

Allow access to the CGI directory and enable it to handle python scripts:

<Directory "${SRVROOT}/cgi-bin">
  Options Indexes FollowSymLinks ExecCGI
  AllowOverride All
  Order allow,deny
  Allow from all
  Require all granted
  AddHandler cgi-script .cgi .py
</Directory>

This is a pretty basic configuration and it could probably be improved but it works on my system.

Then in the CGI-BIN directory create some files, one for each web service and to match the gazetteers listed in the config file. I created three files called llpg_pg.py, lsg_pg.py and postcodes_pg.py.

In the llpg_pg.py file the SQL query selects the records that match the search string and uses the point geometry of the records to return to the plugin.

#!D:/Python27/python.exe
# -*- coding: UTF-8 -*-
import cgi
import json
import psycopg2
sql = """select a.uprn, a.address, ST_X(a.geometry), ST_Y(a.geometry)
from angusdata.address_gazetteer a
where a.address ilike '%%' || (%(p_address)s) || '%%'
order by address"""
form = cgi.FieldStorage ()
connection = psycopg2.connect ("host='10.12.345.678' port='5432' dbname='postgisdb' user='username' password='password'")
cursor = connection.cursor ()
cursor.execute (sql, {"p_address": form["address"].value})
list = []
for record in cursor:
    data = dict (zip (["uprn", "address", "easting", "northing"], record))
    list.append (data)
print "Content-Type: application/json\n"
print json.dumps (list, indent = 4)

In the lsg_pg.py file the SQL query selects the records from the view and converts the line geometry to a centroid point.

#!D:/Python27/python.exe
# -*- coding: UTF-8 -*-
import cgi
import json
import psycopg2
sql = """select a.uprn, a.address, ST_X(ST_Centroid(a.geometry)), ST_Y(ST_Centroid(a.geometry))
from angusdata.search_lsg_streets a
where a.address ilike '%%' || (%(p_address)s) || '%%'
order by address"""
form = cgi.FieldStorage ()
connection = psycopg2.connect ("host='10.12.345.678' port='5432' dbname='postgisdb' user='username' password='password'")
cursor = connection.cursor ()
cursor.execute (sql, {"p_address": form["address"].value})
list = []
for record in cursor:
    data = dict (zip (["uprn", "address", "easting", "northing"], record))
    list.append (data)
print "Content-Type: application/json\n"
print json.dumps (list, indent = 4)

In the postcodes_pg.py file the SQL query turns the polygon geometry into a point geometry using the PostGIS ST_PointOnSurface function. The psycopg2 parameters have been tweaked to use the postcode tables fields.

#!D:/Python27/python.exe
# -*- coding: UTF-8 -*-
import cgi
import json
import psycopg2
sql = """select a.postcode, ST_X(ST_Pointonsurface(a.geometry)), ST_Y(ST_Pointonsurface(a.geometry))
from thirdparty.os_codepointpoly a
where a.postcode ilike '%%' || (%(p_postcode)s) || '%%'
order by postcode"""
form = cgi.FieldStorage ()
connection = psycopg2.connect ("host='10.12.345.678' port='5432' dbname='postgisdb' user='username' password='password'")
cursor = connection.cursor ()
cursor.execute (sql, {"p_postcode": form["postcode"].value})
list = []
for record in cursor:
    data = dict (zip (["postcode", "easting", "northing"], record))
    list.append (data)
print "Content-Type: application/json\n"
print json.dumps (list, indent = 4)

Make sure the #!shebang at the top of the python script has the correct path to your Python installation.

Check the Apache2 log files if there are any errors.

Right, we have QGIS installed with the gazetteer search plugin configured to use the new gazetteers.  The PostGIS tables and views are presenting the data in format required by the web services. The Apache2 server is configure to allow access and the python scripts are in the cgi-bin. Fire up QGIS, refresh the plugin and enter a search term and hit GO!  Awesome!  Mine works, at least.

None of this would have been possible without the UK QGIS group and the input from Matt, Kevin and Simon.  Big thanks to Matt Walker and Jo Cook for recent updates to the plugin.  Use the comments to tell us how to do it better.


Video: how to create maps with webservices of the Dutch PDOK infrastructure in QGis throught the PDOK Plugin

On Eenvoudige PDOK services plugin voor QGIS (in Dutch) Richard already described how to use the PDOK plugin to connect to webservices of the Dutch PDOK infrastructure. This infrastructure offers nationwide and publicly available WMS, WFS and some tiling services of Dutch governments. This video http://www.youtube.com/watch?v=NjLG4KtpdYk shows how to use the plugin to create a [...]

3 Reasons to Move Your QGIS Plugins to the New Repository

  1. Since the release of QGIS 1.8, Plugin Installer no longer includes the “add 3rd party repositories” button. This was an intentional design choice!
  2. The new official plugin repository at plugins.qgis.org keeps everything in one place making it easier for users to find documentation and report issues. It will also provide many long-wanted features such as a rating system for plugins. You can already sort by number of downloads to discover the most popular plugins.
  3. Last but not least: New users will not be able to discover your plugin if it is not in the repository.

Go ahead to plugins.qgis.org and upload your plugin now!


A new QGIS plugin: Python Script Runner

Gary Sherman has just published a new Python plugin for QGIS that I think people will find very handy, I know I will.  The plugin allows you to run Python scripts inside QGIS for tasks that don’t really require, or warrant, a whole plugin.

Go check out Gray’s post about the new plugin at http://spatialgalaxy.net/2012/01/29/script-runner-a-plugin-to-run-python-scripts-in-qgis/

The new plugin can be installed via the Plugin Installer using the “runner” or “script”.  The Plugin Installer is another one of my favorite plugins for QGIS, being able to push out a new plugin and know that everyone can get it is a good feeling :)


Filed under: Open Source, qgis Tagged: FOSSGIS, gis, Open Source, osgeo, plugin, python, qgis, Quantum GIS

Easy Rectangles, Circles and Ellipses in QGIS

“Rectangles ovals digitizing” plugin by Pavol Kapusta adds editing tools that make it really easy to create rectangles, squares, circles and ellipses. These are the tools provided by the new plugin:

Tools in "Rectangles ovals digitizing"

Give it a try!


Fast SQL Layer for QGIS

For everyone working with spatial databases in QGIS there comes a time when “Add PostGIS/SpatiaLite Layer” and “RT Sql Layer” start to be annoying. You always have to retype or copy-paste your SQL queries into the user interface if you need to change the tiniest thing in the layer’s definition.

This is where “Fast SQL Layer” can be a real time saver. Fast SQL Layer is a new plugin for QGIS by Pablo T. Carreira. It basically adds an SQL console for loading layers from PostGIS/SpatiaLite into QGIS. And it even comes with syntax highlighting!

Installation

Fast SQL Layer comes with one dependency: Pygments, which is used for syntax highlighting.

On Ubuntu, all you have to do is install it with apt-get:

sudo apt-get install python-pygments

For Windows with OSGeo4W, @Mike_Toews posted this on gis.stackexchange:

I downloaded and extracted Pygments-1.4.tar.gz, then in an OSGeo4W shell within the Pygments-1.4 directory, type python setup.py build then python setup.py install

Usage

When you activate the plugin in plugin manager, a dock widget will appear which contains the console and some fields for specifying the database connection that should be used. Then, you can simply write your SQL query and load the results with one click.

Fast SQL plugin

In this example, I renamed “gid” to “id”, but you can actually edit the values in the drop down boxes to adjust the column names for id and geometry:

A second layer loaded using Fast SQL plugin

It certainly needs some polishing on the user interface side but I really like it.


OpenLayers plugin visits code sprint

A short visit and 7 hours train ride to the OpenLayers code sprint mainly for a presentation at the Swiss MapFish user group meeting in Lausanne, resulted in a new release of the QGIS OpenLayers plugin. The OpenLayers plugin adds WebKit based layers to QGIS and ships with OpenStreetMap-, Google- and Yahoo-Layers.

Changes in this release:

  • Update to OpenLayers trunk
  • Google Layers using API V3 (no API key necessary)
  • Code refactoring for adding new layer types with one line of code (and some HTML…)

The next planned step is integrating this plugin with the very nice Openlayers Overview plugin from Luiz Motta.

Information for adding your own layers and a bug tracker is now available at hub.qgis.org/projects/openlayers

How to Sort a Shapefile by Attributes

If you ever need to sort the entries in a shapefile by one of its attributes, you might be happy to learn that somebody already did all the work for you and wrote a plugin for QGIS that can perform such sorting tasks: MMQGIS Sort plugin by Michael Minn

MMQGIS Sort plugin


Testing UMN Mapfiles with QGIS

The Sunday night session of the QGIS hackfest resulted in a new release of the Mapfile Tools plugin.

This QGIS plugin allows you to display an UMN Mapserver mapfile in QGIS without running a Mapserver instance. It depends only on Mapscript (apt-get install python-mapscript on Debian/Ubuntu) and allows you to zoom and pan on the mapfile layer.

In release 0.6, an output window has been added, which shows error messages and detailed layer information. This makes it a convenient tool to test your mapfiles.

  • Page 1 of 1 ( 9 posts )
  • plugin

Back to Top

Sponsors