Page 1 of 99 (1967 posts)

  • talks about »

Tags

Last update:
Sun Oct 21 13:45:13 2018

A Django site.

QGIS Planet

Mesh Data Abstraction Library (MDAL)

In the world of GIS, vector and raster are the most common data types to represent real world features. But not always, vectors and rasters can describe the complexity and nature of the data.

MDAL Mesh

Example of those datasets are:

  • Metrological data: e.g. temperature and wind speed
  • Hydrological data: flood depth and flow velocity
  • Oceanography: salinity and wave flux

MDAL arrows and contours

In the past, we introduced Crayfish plugin to handle unstructured grids, usually with temporal and other components. The main issues with Crayfish plugin were:

  • Crayfish plugin had its own renderer. Therefore, we had to create binaries and ship them with the plugin. This was not ideal, as not all the platforms (e.g. MacOS) was not supported.
  • Crayfish plugin loaded all the data in the memory. For small files, this was not an issue. But there was not a solution for larger files.

Why MDAL?

For more details of the rational behind introducing MDAL, see the QGIS Enhancement Proposal with great suggestions and feedback from the developers.

TL;DR: we needed a new abstraction library to handle the mesh data formats similar to GDAL/OGR dealing with rasters and vectors.

Current state

After the feedback from the QGIS community, we have introduced MDAL library earlier this year. It took us a bit of time to put the infrastructure in place. There are already some formats available from MDAL. The library was integrated to QGIS from 3.2. But extensive improvements and new features were added during 3.4.

There are still more formats to support. In QGIS, we will also need to enhance the spatial indexing and performance of the driver in general. For more information visit MDAL website.

How to work with MDAL layer in QGIS

Here is a mesh layer you can use to load in QGIS. For more data, you can visit the ECMWF(or Copernicus) or NASA websites.

In QGIS, from the main menu > Layer > Data Source Manager. A new window will appear. From the left panel, click on Mesh layer Mesh and point to the downloaded file.

You should be able to see a figure similar to the one below:

Mesh layer in QGIS

To view quantities within the mesh, you can open the Layer Styling Panel. Click on symbology Styling tab:

  • First select symbology and deactivate Native Mesh Rendering
  • Click on general and under groups, towards the right of 10 metre wind click on contour and vector to view the quantities.
  • Below the Groups, you can move the slider to browse through time of the dateset(s).
  • To change style of the contours and vectors you can click on contour and vector tabs and adjust the color ramps and vector sizes accordingly.

Mesh layer in QGIS

What happens to Crayfish plugin?

We have recently released a new version of Crayfish plugin to work directly with mesh layer in QGIS. Crayfish is now based on Python only and can be installed on all main platforms (including MacOS).

Future developments

We are planning to add mesh calculator (either as a core QGIS feature or Crayfish/Processing plugin). There will be more formats in the pipeline to support in MDAL too.

If you’d like to add support for your mesh layer to MDAL, you can follow the examples on the github repository. We are always happy to help.

New 3D features in QGIS 3.4

Thanks to the great response from the QGIS community to our crowdfunding call, we have added several new features to the upcoming release (3.4).

Here are the highlights of the features:

Print layout

With this new feature, you can embed a 3D scene to the print layout. In addition, there are camera and view setting tools available to adjust the scene from within the map frame. This will allow you to generate high resolution outputs similar to the other map frames.

QGIS 3D configuration

Animation

You can create an animation based on a set of keyframes - camera positions at particular times. QGIS 3D then interpolates the camera positions/rotations in between the keyframes.

To create the keyframes, first set the scene for your map, by rotating, zooming or moving the camera. Then assign a time to the frame. There are several methods for interpolations between keyframes.

QGIS 3D configuration

Identify tool in 3d map

A new Identify Tool was introduced to 3D map. Using this tool, you can inspect features from the 3D scene. Thanks to our friends from Faunalia for funding this feature.

3D lines

If you have a 3D linestring, you can now use the elevation of vertices to display it in 3D map.

QGIS 3D configuration

Camera control

There are more control on how you can pan, rotate and zoom the camera within the scene:

  • Page up/down: to move the camera up/down: Page up/down keys for moving the camera
  • Free camera movements: also possible to look from below: Page up/down keys for moving the camera
  • Shift key and drag: the camera rotates around a point on terrain: Page up/down keys for moving the camera
  • Ctrl key and drag: this will rotate camera while it stays in one position Page up/down keys for moving the camera

In addition, the point towards which the camera is looking is dynamically adjusted based on the terrain. The rotation in scenes where a DEM used for terrain has been enhanced significantly. This is more pronounced with presence of higher hills in the terrain.

Further development

There are still some exciting features from the crowdfunding which will be incorporated in QGIS 3.6. You can check the QGIS development or follow us on Twitter.

Add Realistic Mist and Fog to Topography in QGIS 3.2

I recently came across a great tutorial by in which he demonstrated how to create map of Switzerland in the style of Edward Imhof, the famed Swiss cartographer renowned for his hand painted maps of Switzerland and other mountainous regions of the world. John’s map used traditional hillshading, multidirectional hillshading and crucially, a translucent topographic layer that created a mist like appearance he likened to the sfumato technique used by painters since the Renascence.

I followed John’s tutorial in QGIS 3.2 and I was quite pleased with the initial result below. However, the process creating it is a bit too complicated for a tutorial so I set about simplifying the process and rather than imitating Imhof’s distinct style, my goal this time is realism.

The heart of the effect involves the very clever idea of using the topographic layer as a subtle opacity mask to simulate mist, fog and atmospheric haze. Have a look at the image below taken on March 17th, 2005 by NASA’s Terra satellite. This is the industrialised Po valley of northern Italy, surrounded by the Alps and Apennine Mountains that rise above the valley’s hazy pollution. The haze adds a sense of depth to the surrounding hills and mountains. It’s not uncommon to see fog and pollution in satellite imagery that gives way to the clear air in high mountains e.g. northern India and Nepal, China, Pakistan and India. Creating a similar mist effect in QGIS is actually quite simple.

First download topography for the Alps and Po region (a 68.55 Mb GeoTiff file derived from freely available EU-DEM data I resampled from 25 to 100m resolution). Next, make sure you have the plugin QuickMapServics (QMS) installed (menu Plugins – Manage and Install Plugins). This great plugin provides access to over 1000 basemaps.

Load the GeoTiff file into QGIS (Raster – Load) and rename the layer Hillshade. Right click the layer to open the Layer Properties window. In the Symbology panel, next to Render Type, choose Hillshade. Change the altitude to 35 degrees, Azimuth to 300 degrees and Z Factor of 1.5 (illuminating the landscape from the top left). Finally, change the Blending mode to Multiply. Click OK to close the dialogue.

To add the basemap layer, Esri World Imagery (Clarity), type “ESRI clarity” in the QMS search bar to find and add the basemap; Go to View – Panels and activate the QMS search bar if it isn’t initially visible. Make sure it’s the bottommost layer.

Oh, that’s a bit disappointing, we only increased the relief little a bit. It’s missing the vitally important mist layer.

To create mist, right click the Hillshade layer and choose Duplicate. Rename the new layer Mist and make sure it’s above the Hillshade layer. Now open the Layer Properties window of the layer, we’re going edit it’s attributes to make it look like mist.

Change the Render type to Singleband Pseudocolor and use 0 and 3000 for the min and max values (limiting maximum latitude of the mist to 3000 meters). Then open the colour ramp window by clicking on the Color ramp and enter these values:

  • Left Gradient – HSV 215 15 50 and 75% transparency
  • Right Gradient – HSV 215 15 50 and 0% transparency

Close the Color Ramp dialogue. In the Layer Properties window, and this is very important, change the Blending mode to Lighten. Click OK to close the Layer Properties window.

Wow, we have mist!

The mist effect looks great. It certainly adds a lot of realism to the topographic map, it now looks quite like NASA’s images. This is just a quick and basic map so there’s lots of scope to improve the effect. Play around with the colour of the mist layer and its opacity, or even brighten the Hillshade layer underneath. See what effects these changes have.

Here’s another example below. In this example I duplicated the hillshade layer and set the second hillshade layer to Multidirectional Hillshading (yes, QGIS 3.2 has Multidirectional Hillshading). I then adjusted the transparency of both hillshade layers so they blended together nicely. I then replaced the basemap with another duplicated topography layer that I coloured using the gradient sd-a (by Jim Mossman, 2005) using the cpt-city plugin. And lastly, I doubled the opacity of the mist layer turning it into a milky fog. I think it looks great!

What next? Well, there’s lots of possibilities. Perhaps download Martian topography and add mist to the bottom of Valles Marineris?

References:

Eduard Imhof – Biography

TV documentary about Eduard Imhof

The Map as an Artistic Territory: Relief Shading Works and Studies by Eduard Imhof

Haze in northern Italy – NASA Terra Satellite

Tzvetkov, J., 2018. Relief visualization techniques using free and open source GIS tools. Polish Cartographical Review, 50(2), pp.61-71.

Programme change…

See the Dutch website for more info on the programme.

Edit Features “In Place” crowdfund — made it to QGIS 3.4!

Well, thanks to the resounding success of our QGIS edit-in-place crowdfunding campaign, we’ve been frantically smashing away at our keyboards in an attempt to reward the QGIS community by sneaking this feature in a whole 4 months earlier than originally promised! And, we’re very proud to announce, that this exciting new feature has been implemented and will be included in the upcoming QGIS 3.4 release (due late October 2018). So go ahead — grab one of the nightly pre-release of QGIS 3.4 and checkout the results.

This wouldn’t have been possible without the rapid response to the campaign and the generosity of our wonderful backers:

(In addition to these backers, we’ve also received numerous anonymous donations to this feature from many other individuals — while we can’t list you all publicly, you’re also in our thanks!)

 

Keep an eye on this blog for other upcoming QGIS crowdfunding campaigns targeted at QGIS 3.6 and beyond… we’ve got lots more exciting work planned for these releases!

 

Dutch QGIS usergroup meeting

All presentations will be in Dutch. Please have a look at the Dutch version of this page to see more…

8th Scottish QGIS UK user group meeting plans

It’s been a long time since we posted anything up here but hopefully the news of another user group happening in Stirling on 7th November will hopefully assuage your distress.

The event is very kindly hosted by Historic Environment Scotland in their restored Engine Shed and is also supported by Ordnance Survey, Cawdor Forestry, thinkWhere, SEPA and Registers of Scotland.

The programme is still being worked out but plan on attending one of two workshop sessions in the morning run by Ordnance Survey and thinkWhere and then a full afternoon of presentations and lightning talks.

Tickets are available through Eventbrite.

8th Scottish QGIS @UK user group

MacOS specific bug fixing campaign

If you are a MacOS QGIS user, you are probably bothered by some MacOS specific bugs. These are due to the fact that we have fewer QGIS developers working on the MacOS platform and there are additional MacOS specific issues in the underlying qt5 library.

Nevertheless, we found a developer, Denis Rouzaud, who wants to specifically look into investigating and hopefully solving several of these issues. If you are a MacOS user and care about a better QGIS experience on this platform, we invite you to financially support this effort. As a private person, and for smaller amounts, please use the usual donation channel – if you are a company or organization and want to contribute to this specific effort, please consider becoming a sponsor. In any case – please add “MacOS bug fixing campaign” as a remark when donating/sponsoring or inform [email protected] about your earmarked donation.

This effort runs from the 14th September 2018 until the 3.4 release date, due on October 26, 2018. See the QGIS road map for more details about our release schedule.

Specific issues that are looked into, are:

issue priority subject
11103 1 Support for retina displays (HiDPI)
17773 1 No Retina / HiDPI support in 2.99 on osx
19546 1 QGIS 3 slow on macOS at high resolutions
19524 1 [macOS] Map canvas with wrong size on QGIS 3.2.1 start up
19321 2 Map Tips on Mac doesn’t display the content correctly
19314 1 3.2 crashes on startup on a Mac
19092 2 Measure tool on a Mac uses the top right corner of the cross hair cursor instead of the centre
18943 3 QGIS Server on MacOS X High Sierra
18914 3 [macOS] Plugin list corrupted by wrongly placed checkboxes on Mac
18720 2 QGIS 3.0.1 crashes on Mac
18452 3 Snapping options missing on Mac
18418 2 Scroll zoom erratic on Mac trackpad
16575 3 QGIS 2.18.7 crashes on macOS 10.12.4 when undocking the label panel
16025 2 [macOS] Control feature rendering order will crash QGIS
3975 2 PDF exports on OSX always convert text to outlines

Thank you for considering to support this effort! Please note that some issues may also exist due to up-stream issues in the qt library. In such a case, it can’t be guaranteed if and how fast, such an issue can be fixed.

Andreas Neumann, QGIS.ORG treasurer

Edit Features “In Place” crowdfund — target reached!

Well, the final pledges have been tallied and we’re very proud to announce that our latest crowd funding campaign has been a roaring success!

We’ve been completely blown away by the response to this campaign. Thanks to some incredibly generous backers and donors, we’ve been able to hit the campaign target with plenty of time to spare. As a result, we’ll be pushing hard to reward the generosity of the community by trying to sneak this feature in for the upcoming QGIS 3.4 release (instead of the originally promised 3.6 release)! You can read more about what we’re adding at the campaign page.

We’d like to take this opportunity to extend our heartfelt thanks to all the backers who have pledged to support this project:

In addition to these backers, we’ve also received numerous anonymous donations to this feature from many other individuals — while we can’t list you all publicly, you’re also in our thanks!

Stay tuned for more updates to come as work proceeds on this feature…

Geocoding with Geopy

Need to geocode some addresses? Here’s a five-lines-of-code solution based on “An A-Z of useful Python tricks” by Peter Gleeson:

from geopy import GoogleV3
place = "Krems an der Donau"
location = GoogleV3().geocode(place)
print(location.address)
print("POINT({},{})".format(location.latitude,location.longitude))

For more info, check out geopy:

geopy is a Python 2 and 3 client for several popular geocoding web services.
geopy includes geocoder classes for the OpenStreetMap Nominatim, ESRI ArcGIS, Google Geocoding API (V3), Baidu Maps, Bing Maps API, Yandex, IGN France, GeoNames, Pelias, geocode.earth, OpenMapQuest, PickPoint, What3Words, OpenCage, SmartyStreets, GeocodeFarm, and Here geocoder services.

QGIS speaks a lot of languages

QGIS is a real cosmopolitan. Born in Alaska sixteen years ago, it has spread all over the world since. Thanks to its open source mentality, it finds not only in economically strong countries big usergroups. No question, that beside all

(Nederlands) 3 oktober QGIS gebruikersdag in Nieuwegein

Sorry, this entry is only available in the Dutch language

Plotting GPS Trajectories with error ellipses using Time Manager

This is a guest post by Time Manager collaborator and Python expert, Ariadni-Karolina Alexiou.

Today we’re going to look at how to visualize the error bounds of a GPS trace in time. The goal is to do an in-depth visual exploration using QGIS and Time Manager in order to learn more about the data we have.

The Data

We have a file that contains GPS locations of an object in time, which has been created by a GPS tracker. The tracker also keeps track of the error covariance matrix for each point in time, that is, what confidence it has in the measurements it gives. Here is what the file looks like:

data.png

Error Covariance Matrix

What are those sd* fields? According to the manual: The estimated standard deviations of the solution assuming a priori error model and error parameters by the positioning options. What it basically means is that the real GPS location will be located no further than three standard deviations across north and east from the measured location, most of (99.7%) the time. A way to represent this visually is to create an ellipse that maps this area of where the real location can be.ellipse_ab

An ellipse can be uniquely defined from the lengths of the segments a and b and its rotation angle. For more details on how to get those ellipse parameters from the covariance matrix, please see the footnote.

Ground truth data

We also happen to have a file with the actual locations (also in longitudes and latitudes) of the object for the same time frame as the GPS (also in seconds), provided through another tracking method which is more accurate in this case.

actual_data

This is because, the object was me running on a rooftop in Zürich wearing several tracking devices (not just GPS), and I knew exactly which floor tiles I was hitting.

The goal is to explore, visually, the relationship between the GPS data and the actual locations in time. I hope to get an idea of the accuracy, and what can influence it.

First look

Loading the GPS data into QGIS and Time Manager, we can indeed see the GPS locations vis-a-vis the actual locations in time.

actual_vs_gps

Let’s see if the actual locations that were measured independently fall inside the ellipse coverage area. To do this, we need to use the covariance data to render ellipses.

Creating the ellipses

I considered using the ellipses marker from QGIS.

ellipse_marker.png

It is possible to switch from Millimeter to Map Unit and edit a data defined override for symbol width, height and rotation. Symbol width would be the a parameter of the ellipse, symbol height the b parameter and rotation simply the angle. The thing is, we haven’t computed any of these values yet, we just have the error covariance values in our dataset.

Because of the re-projections and matrix calculations inherent into extracting the a, b and angle of the error ellipse at each point in time, I decided to do this calculation offline using Python and relevant libraries, and then simply add a WKT text field with a polygon representation of the ellipse to the file I had. That way, the augmented data could be re-used outside QGIS, for example, to visualize using Leaflet or similar. I could have done a hybrid solution, where I calculated a, b and the angle offline, and then used the dynamic rendering capabilities of QGIS, as well.

I also decided to dump the csv into an sqlite database with an index on the time column, to make time range queries (which Time Manager does) run faster.

Putting it all together

The code for transforming the initial GPS data csv file into an sqlite database can be found in my github along with a small sample of the file containing the GPS data.

I created three ellipses per timestamp, to represent the three standard deviations. Opening QGIS (I used version: 2.12, Las Palmas) and going to Layer>Add Layer>Add SpatialLite Layer, we see the following dialog:

add_spatialite2.png

After adding the layer (say, for the second standard deviation ellipse), we can add it to Time Manager like so:

add_to_tm

We do the process three times to add the three types of ellipses, taking care to style each ellipse differently. I used transparent fill for the second and third standard deviation ellipses.

I also added the data of my  actual positions.

Here is an exported video of the trace (at a place in time where I go forward, backwards and forward again and then stay still).

gps

Conclusions

Looking at the relationship between the actual data and the GPS data, we can see the following:

  • Although the actual position differs from the measured one, the actual position always lies within one or two standard deviations of the measured position (so, inside the purple and golden ellipses).
  • The direction of movement has greater uncertainty (the ellipse is elongated across the line I am running on).
  • When I am standing still, the GPS position is still moving, and unfortunately does not converge to my actual stationary position, but drifts. More research is needed regarding what happens with the GPS data when the tracker is actually still.
  • The GPS position doesn’t jump erratically, which can be good, however, it seems to have trouble ‘catching up’ with the actual position. This means if we’re looking to measure velocity in particular, the GPS tracker might underestimate that.

These findings are empirical, since they are extracted from a single visualization, but we have already learned some new things. We have some new ideas for what questions to ask on a large scale in the data, what additional experiments to run in the future and what limitations we may need to be aware of.

Thanks for reading!

Footnote: Error Covariance Matrix calculations

The error covariance matrix is (according to the definitions of the sd* columns in the manual):

sde * sde sign(sdne) * sdne * sdne
sign(sdne) * sdne * sdne sdn * sdn

It is not a diagonal matrix, which means that the errors across the ‘north’ dimension and the ‘east’ dimension, are not exactly independent.

An important detail is that, while the position is given in longitudes and latitudes, the sdn, sde and sdne fields are in meters. To address this in the code, we convert the longitude and latitudes using UTM projection, so that they are also in meters (northings and eastings).

For more details on the mathematics used to plot the ellipses check out this article by Robert Eisele and the implementation of the ellipse calculations on my github.

FOSS4G 2018 Dar es Salaam

This year’s FOSS4G edition took place in Dar es Salaam, Tanzania. As every year, Sourcepole was supporting this major event as a sponsor. We would like to thank for all the interesting discussions and feedback to our presentations!

image

QGIS Web Client 2 Update

Styling and publication of vector tiles

Using GeoPackage as work and exchange format

Thanks to the LOC for organizing another great FOSS4G!

Pirmin Kalberer (@implgeo)

OpenCL acceleration now available in QGIS

What is OpenCL?

From https://en.wikipedia.org/wiki/OpenCL:

OpenCL (Open Computing Language) is a framework for writing programs that execute across heterogeneous platforms consisting of central processing units (CPUs), graphics processing units (GPUs), digital signal processors (DSPs), field-programmable gate arrays (FPGAs) and other processors or hardware accelerators. OpenCL specifies programming languages (based on C99 and C++11) for programming these devices and application programming interfaces (APIs) to control the platform and execute programs on the compute devices. OpenCL provides a standard interface for parallel computing using task- and data-based parallelism.

Basically, you write a program and you execute it on a GPU (or, less frequently, on a CPU or on a DSP) taking advantage of the huge parallel programming capabilities of the modern graphic cards.

Depending on many different factors, the speed gain can vary to a great extent, but it is typically around one order of magnitude.

How QGIS benefits from OpenCL?

The work I’ve done consisted in integrating OpenCL support into QGIS and writing all the utilities to load, build and run OpenCL programs.

For now, I’ve ported the following QGIS core algorithms, all of them are availabe in processing:

  • slope
  • aspect
  • hillshade
  • ruggedness

Since the framework to support OpenCL is now in place, I think that more algorithms will be ported over the time.

During this development, even if was not in scope, the hillshade renderer has been optimized for speed and it can also benefit of OpenCL acceleration.

How to activate OpenCL support

OpenCL support is optional and opt-in, to use it, you need to activate it into the QGIS options dialog like shown in the screenshot below:

How much performance gain can I expect?

Well, YMMV, but here are some figures for a big DEM raster, low values mean faster execution.

GDAL means CPU execution using the GDAL processing algorithm.

How to install the OpenCL drivers?

Of course it depends on your specific hardware and on your O.S., AMD, NVidia and Intel have different distributions channels, in general the driver for your graphic card will also provide the OpenCL driver, if your GPU is compatible, if OpenCL is not available on your current machine, try to Google for OpenCL, your O.S. and graphic card.

If there is no OpenCL support for your graphic card, you might try to install a driver for your GPU (Intel for example provides them) and you will probably have a decent acceleration even if not as much as you can get on a real graphic card.

This fact worths some more explanation: you might ask your self why running and algorithm directly on the CPU and running it on the same CPU but using OpenCL would make any difference and the reason why it is generally faster by using OpenCL is that OpenCL will run the algorithm in parallel on all cores of your CPU, while a normal application (and QGIS does not make an exception here) will use a single core.

Credits

I started this work as a proof of concept in my spare time (that it is not much, lately) and when I realized that it was promising, I submitted a QGIS grant proposal in order to allocate some working time to port more algorithms, write tests and polish the implementation.

This work would not be possible without all the generous sponsors and donors that feed the QGIS grant program year after year, many thanks to the QGIS community for this amazing support!

Jürgen Fischer was as usual very helpful and took care of the windows builds, now available in OSGeo4W packages.

Nyall Dawson helped with the code review and with testing the implementation on different cards and machines.

Matthias Kuhn reviewed the code.

Even Rouault pointed me to some highly efficient GDAL algorithm optimizations that I’ve been able to integrate in QGIS.

 

 

Edit Features “In Place” Using QGIS Spatial Operations crowdfund launched!

We’ve just launched a new QGIS crowd funding campaign which we’re super-excited about! This time, we’re addressing what we see as the major shortcoming within QGIS vector layer editing tools, and bridging the gap between the vast power of QGIS’ Processing algorithms and easy-to-use operations which modify layer features “in place”. Here’s a quick sneak preview of what we have planned:

 

QGIS is already a vector editing powerhouse, and we believe that this improvement will boost the current functionality up an order of magnitude! To make it possible we need 6500€ pledged before 30 September 2018.

This is also our first crowdfunding campaign in which we’re running a “dual funding” approach, which we think should make things friendly and easy for both corporate backers and end user contributions alike. Read more about this at the full campaign page.

You can help make this a reality by supporting the campaign or by sharing the page and increasing exposure to the campaign. Updates to follow!

 

Celebrating 35 years of GRASS GIS!

Today marks 35 years of GRASS GIS development – with frequent releases the project keeps pushing the limits in terms of geospatial data processing quality and performance.

GRASS (Geographic Resources Analysis Support System) is a free and open source Geographic Information System (GIS) software suite used for geospatial data management and analysis, image processing, graphics and map production, spatial modeling, and 3D visualization. Since the major GRASS GIS 7 version, it also comes with a feature rich engine for space-time cubes useful for time series processing of Landsat and Copernicus Sentinel satellite data and more. GRASS GIS can be either used as a desktop application or as a backend for other software packages such as QGIS and R. Furthermore, it is frequently used on HPC and cloud infrastructures for massive parallelized data processing.

Brief history
In 1982, under the direction of Bill Goran at the U.S. Army Corps of Engineers Construction Engineering Research Laboratory (CERL), two GIS development efforts were undertaken. First, Lloyd Van Warren, a University of Illinois engineering student, began development on a new computer program that allowed analysis of mapped data.  Second, Jim Westervelt (CERL) developed a GIS package called “LAGRID – the Landscape Architecture Gridcell analysis system” as his master’s thesis. Thirty five years ago, on 29 July 1983, the user manual for this new system titled “GIS Version 1 Reference Manual” was first published by J. Westervelt and M. O’Shea. With the technical guidance of Michael Shapiro (CERL), the software continued its development at the U.S. Army Corps of Engineers Construction Engineering Research Laboratory (USA/CERL) in Champaign, Illinois; and after further expansion version 1.0 was released in 1985 under the name Geographic Resources Analysis Support System (GRASS). The GRASS GIS community was established the same year with the first annual user meeting and the launch of GRASSnet, one of the internet’s early mailing lists. The user community expanded to a larger audience in 1991 with the “Grasshopper” mailing list and the introduction of the World Wide Web. The users’ and programmers’ mailing lists archives for these early years are still available online.
In the mid 1990s the development transferred from USA/CERL to The Open GRASS Consortium (a group who would later generalize to become today’s Open Geospatial Consortium — the OGC). The project coordination eventually shifted to the international development team made up of governmental and academic researchers and university scientists. Reflecting this shift to a project run by the users, for the users, in 1999 GRASS GIS was released under the terms of the GNU General Public License (GPL). A detailed history of GRASS GIS can be found at https://grass.osgeo.org/history/.

Where to next?
The development on GRASS GIS continues with more energy and interest than ever. Parallel to the long-term maintenance of the GRASS 7.4 stable series, effort is well underway on the new upcoming cutting-edge 7.6 release, which will bring many new features, enhancements, and cleanups. As in the past, the GRASS GIS community is open to any contribution, be it in the form of programming, documentation, testing, and financial sponsorship. Please contact us!

About GRASS GIS

The Geographic Resources Analysis Support System (https://grass.osgeo.org/), commonly referred to as GRASS GIS, is an Open Source Geographic Information System providing powerful raster, vector and geospatial processing capabilities in a single integrated software suite. GRASS GIS includes tools for spatial modeling, visualization of raster and vector data, management and analysis of geospatial data, and the processing of satellite and aerial imagery. It also provides the capability to produce sophisticated presentation graphics and hardcopy maps. GRASS GIS has been translated into about twenty languages and supports a huge array of data formats. It can be used either as a stand-alone application or as backend for other software packages such as QGIS and R geostatistics. It is distributed freely under the terms of the GNU General Public License (GPL). GRASS GIS is a founding member of the Open Source Geospatial Foundation (OSGeo).

The GRASS Development Team, July 2018

The post Celebrating 35 years of GRASS GIS! appeared first on GFOSS Blog | GRASS GIS and OSGeo News.

blog:polska_grupa_uzytkownikow_qgis_na_facebooku

Na Facebooku uruchomiliśmy grupę o nazwie Polska Grupa Użytkowników QGIS, która jest odpowiedzią na rosnące zapotrzebowanie na takie miejsce w sieciach społecznościowych. Celem grupy jest promowanie QGIS jako wolnego i otwartego oprogramowania oraz wymiana wiedzy, umiejętności i doświadczeń. Zapraszamy do grupy wszystkich posiadających konto na FB i korzystających z QGIS. Zachęcamy też do aktywnego w niej uczestnictwa.

blog:polska_grupa_uzytkownikow_qgis_na_facebooku

Na Facebooku uruchomiliśmy grupę o nazwie Polska Grupa Użytkowników QGIS, która jest odpowiedzią na rosnące zapotrzebowanie na takie miejsce w sieciach społecznościowych. Celem grupy jest promowanie QGIS jako wolnego i otwartego oprogramowania oraz wymiana wiedzy, umiejętności i doświadczeń. Zapraszamy do grupy wszystkich posiadających konto na FB i korzystających z QGIS. Zachęcamy też do aktywnego w niej uczestnictwa.

Movement data in GIS #15: writing a PL/pgSQL stop detection function for PostGIS trajectories

Do you sometimes start writing an SQL query and around at line 50 you get the feeling that it might be getting out of hand? If so, it might be useful to start breaking it down into smaller chunks and wrap those up into custom functions. Never done that? Don’t despair! There’s an excellent PL/pgSQL tutorial on postgresqltutorial.com to get you started.

To get an idea of the basic structure of a PL/pgSQL function and to proof that PostGIS datatypes work just fine in this context, here’s a basic function that takes a trajectory geometry and outputs its duration, i.e. the difference between its last and first timestamp:

CREATE OR REPLACE FUNCTION AG_Duration(traj geometry) 
RETURNS numeric LANGUAGE 'plpgsql'
AS $BODY$ 
BEGIN
RETURN ST_M(ST_EndPoint(traj))-ST_M(ST_StartPoint(traj));
END; $BODY$;

My end goal for this exercise was to implement a function that takes a trajectory and outputs the stops along this trajectory. Commonly, a stop is defined as a long stay within an area with a small radius. This leads us to the following definition:

CREATE OR REPLACE FUNCTION AG_DetectStops(
   traj geometry, 
   max_size numeric, 
   min_duration numeric)
RETURNS TABLE(sequence integer, geom geometry) 
-- implementation follows here!

Note how this function uses RETURNS TABLE to enable it to return all the stops that it finds. To add a line to the output table, we need to assign values to the sequence and geom variables and then use RETURN NEXT.

Another reason to use PL/pgSQL is that it enables us to write loops. And loops I wanted for my stop detection function! Specifically, I wanted to go through all the points in the trajectory:

FOR pt IN SELECT (ST_DumpPoints(traj)).geom LOOP
-- here comes the magic!
END LOOP;

Eventually the function should go through the trajectory and identify all segments that stay within an area with max_size diameter for at least min_duration time. To test for the area size, we can use:

IF ST_MaxDistance(segment,pt) <= max_size THEN is_stop := true; 

Putting everything together, my current implementation looks like this:

CREATE OR REPLACE FUNCTION AG_DetectStops(
   traj geometry,
   max_size numeric,
   min_duration numeric)
RETURNS TABLE(sequence integer, geom geometry) 
LANGUAGE 'plpgsql'
AS $BODY$
DECLARE 
   pt geometry;
   segment geometry;
   is_stop boolean;
   previously_stopped boolean;
   stop_sequence integer;
   p1 geometry;
BEGIN
segment := NULL;
sequence := 0;
is_stop := false;
previously_stopped := false;
p1 := NULL;
FOR pt IN SELECT (ST_DumpPoints(traj)).geom LOOP
   IF segment IS NULL AND p1 IS NULL THEN 
      p1 := pt; 
   ELSIF segment IS NULL THEN 
      segment := ST_MakeLine(p1,pt); 
      p1 := NULL;
      IF ST_Length(segment) <= max_size THEN is_stop := true; END IF; ELSE segment := ST_AddPoint(segment,pt); -- if we're in a stop, we want to grow the segment, otherwise we remove points to the specified min_duration IF NOT is_stop THEN WHILE ST_NPoints(segment) > 2 AND AG_Duration(ST_RemovePoint(segment,0)) >= min_duration LOOP
            segment := ST_RemovePoint(segment,0); 
         END LOOP;
      END IF;
      -- a stop is identified if the segment stays within a circle of diameter = max_size
      IF ST_Length(segment) <= max_size THEN is_stop := true; ELSIF ST_Distance(ST_StartPoint(segment),pt) > max_size THEN is_stop := false;
      ELSIF ST_MaxDistance(segment,pt) <= max_size THEN is_stop := true; ELSE is_stop := false; END IF; -- if we found the end of a stop, we need to check if it lasted long enough IF NOT is_stop AND previously_stopped THEN IF ST_M(ST_PointN(segment,ST_NPoints(segment)-1))-ST_M(ST_StartPoint(segment)) >= min_duration THEN
            geom := ST_RemovePoint(segment,ST_NPoints(segment)-1); 
            RETURN NEXT;
            sequence := sequence + 1;
            segment := NULL;
            p1 := pt;
         END IF;
      END IF;
   END IF;
   previously_stopped := is_stop;
END LOOP;
IF previously_stopped AND AG_Duration(segment) >= min_duration THEN 
   geom := segment; 
   RETURN NEXT; 
END IF;
END; $BODY$;

While this function is not really short, it’s so much more readable than my previous attempts of doing this in pure SQL. Some of the lines for determining is_stop are not strictly necessary but they do speed up processing.

Performance still isn’t quite where I’d like it to be. I suspect that all the adding and removing points from linestring geometries is not ideal. In general, it’s quicker to find shorter stops in smaller areas than longer stop in bigger areas.

Let’s test! 

Looking for a testing framework for PL/pgSQL, I found plpgunit on Github. While I did not end up using it, I did use its examples for inspiration to write a couple of tests, e.g.

CREATE OR REPLACE FUNCTION test.stop_at_beginning() RETURNS void LANGUAGE 'plpgsql'
AS $BODY$
DECLARE t0 integer; n0 integer;
BEGIN
WITH temp AS ( SELECT AG_DetectStops(
   ST_GeometryFromText('LinestringM(0 0 0, 0 0 1, 0.1 0.1 2, 2 2 3)'),
   1,1) stop 
)
SELECT ST_M(ST_StartPoint((stop).geom)), 
       ST_NPoints((stop).geom) FROM temp INTO t0, n0;	
IF t0 = 0 AND n0 = 3
   THEN RAISE INFO 'PASSED - Stop at the beginning of the trajectory';
   ELSE RAISE INFO 'FAILED - Stop at the beginning of the trajectory';
END IF;
END; $BODY$;

Basically, each test is yet another PL/pgSQL function that doesn’t return anything (i.e. returns void) but outputs messages about the status of the test. Here I made heavy use of the PERFORM statement which executes the provided function but discards the results:


Update: The source code for this function is now available on https://github.com/anitagraser/postgis-spatiotemporal

  • Page 1 of 99 ( 1967 posts )
  • >>

Back to Top

Sponsors