Page 1 of 118 (2350 posts)

  • talks about »


Last update:
Tue Feb 18 08:30:26 2020

A Django site.

QGIS Planet

Collecting data using QGIS and Input app

In this post, we will walk you through basic steps to set up a survey project in QGIS desktop and using it in Input app to collect data in field using your Android of iPhone/iPad deveice.

Software needed

To start with, you will need to install the following software applications:

In addition, you will need to register with the Mergin service. The Mergin service allows you to transfer data between your PC/laptop and mobile/table via the cloud. Note that after signing up to the Mergin service, you have to activate the account by clicking on the link sent to your email.

Configuring QGIS project

To be able to survey data, we need to set up a project in QGIS. Usually, you will need some data for your background layer (so that you can locate yourself!). In addition, you need to set up a table (or layer), to store your survey information.

For background data, we are going to use Open Street Map. For survey table, we need to decide on a form structure and the type of feature you want to survey (e.g. point of interest, tracks or parcel of land). In this case, we want to survey potholes. Also, it would be good to attach some notes for each pothole, take a photo of it and add a date for survey. The GIS format best suited to store spatial information, is Geopackage.

Let’s start by opening QGIS and add the above layers to our project. To simplify things, we can create a folder on Desktop (referred to in this tutorial as data collection folder) and store everything there.

Open QGIS from your PC/laptop. From the Browser panel (usually located on the top left side), expand XYZ Tiles and double-click on OpenStreetMap to add it to QGIS: QGIS

Browser panel in QGIS

You should see the OSM layer:

Adding OSM XYZ layer

Save your project as pothole survey in the data collection folder.

To create a survey layer, in QGIS, from the main menu select Layer > Create Layer > New Geopackage Layer …. Note that Geopackage is a file based database where you can store multiple tables (spatial or non-spatial). A new window will appear:

Creating a geodatabase

For Database click on and select the data collection folder on your Desktop and then type survey-db.gpkg for the name of your database.

For Table name, type Potholes.

For Geometry type, select Point.

For Coordinate Reference System (CRS), click on the icon to the right of EPSG:4326 - WGS84. A new window will appear. Under Filter section on the top of the window, type: 3857 and under Predefined Coordinate Reference Systems, select WGS 84 / Pseudo-Mercator EPSG:3857. Then click OK.

Assigning CRS

We can now create the column headers for our table under New Field section. For this form, we want to create the following columns to store data: Date, Notes, Photo

For Name, type Date

For Type, select Date

Click on Add to Field lists to add your column.

Repeat the same process for Notes and Photos columns, but make sure to change the Type for those columns to Text. At this stage, you should see an image similar to the one below:

Sharing projects through Mergin

Go ahead and click OK to create the layer and add it to QGIS.

Styling layers and setting up forms

The default style applied to Potholes layer is not very visible probably. To change it:

In the Layer Panels right-click on Potholes layer and select Properties. A new window will appear. From the left panel, select Symbology. Try to change the style to something shown in the image below:

Sharing projects through Mergin

Click on Apply.

We can also change the way user fills in the form. By default, you have to type in the values. But by using different widgets, we can simplify filling the form in the field.

In the Properties window, from the left panel, select Attribute forms.

Sharing projects through Mergin

We are going to change the Widget Type for each of the Fields.

fid is an auto-increment field and we can keep it hidden from users. So, highlight the fid field under Field section and then from the Widget Type select Hidden

For Data, it should have automatically selected the correct widget type:

Sharing projects through Mergin

For Notes, you can also leave the Widget Type as Text Edit.

For Photos, we need to change the Widget Type to Attachment. Also make sure to select the option for Relative paths. This will allow us to attach photos using mobile camera or gallery folder to the pothole point.

Tip: You can scroll further down and under Integrated Document Viewer and select Type as Image. This will show the image in QGIS forms too.

Sharing projects through Mergin

Project set up is completed and we can save the project.

Transferring data to mobile devices

You have 2 options to transfer your data to the mobile through the Mergin service: through website or through Mergin plugin in QGIS. In this tutorial we are going to use the plugin from within QGIS.

In QGIS, from the main menu, select Plugins > Manage and Install Plugins …. A new window will appear. From the left panel, select All and then in the search section (on the top) search for Mergin. Select the plugin from the list and click on Install plugin. After installation, you need to restart your QGIS.

After the restart, you should be able to see the Mergin icon in your Browser Panel:

Sharing projects through Mergin

In the Browser Panel, right click on the Mergin and select Configure. Type in your username (or email address) and password that you have registered with the Mergin service.

Sharing projects through Mergin

Click on Test Connection and you should see a green OK.

If you have selected to Save credentials (so you do not need to type in the username and password again) and you have not configured QGIS password manager, you will be prompted to set a password for your QGIS password manager.

After clicking OK, you should see a list of folders on your Mergin connection in your browser panel:

Sharing projects through Mergin

We can know upload the data:

Right click on the Mergin and select Create new project. A new window will appear:

For Project name type Potholes survey

Select Initialize from local drive

Click on … and and select data collection folder

Sharing projects through Mergin

Once click OK, the project will be created and content of the data collection folder will be uploaded there.

The project is now ready to be downloaded on your mobile device.

Collecting data using Input apps

The project can now be accessed from Input app. Open your Input app and for the first time you should see a screen similar to the image below:

Sharing projects through Mergin

To log in to the Mergin service, you can select My projects or the green and white icon on the top right.

Sharing projects through Mergin

Type your Mergin username (or email address) and password and then select Sign in.

Once signed in, select My projects and you will see Potholes survey project in the lists

Sharing projects through Mergin

Select the download icon on the right side of Potholes survey to download your project on the phone and make it ready for survey.

After downloading is completed, select Home and you should be able to see Potholes survey.

Sharing projects through Mergin

Select Potholes survey and it will open the map:

Sharing projects through Mergin

To record a feature, select Record button and the pointer changes to a cross-hair.

Sharing projects through Mergin

By default, the cross-hair centres to your location (the orange point) on the map. You can move the map and adjust the location. To recentre the map to your location, you can select GPS button. Once you are happy with the location, you can select Add point and the form for your point will appear:

Sharing projects through Mergin

Fill in the form and press Save. You should see the map with the newly captured pothole:

Sharing projects through Mergin

Synchronising data

The data you have captured on your phone can be synchronised through the Mergin service.

In Input app, select Projects and then My projects. You should see a double arrow on the right side of the Potholes survey.

Sharing projects through Mergin

Select the double arrow to sync your project. You can also open QGIS from your PC/laptop and synchronise changes back to your desktop:

In QGIS, from the Browser Panel under Mergin > My projects right-click on Potholes survey and select Synchronize

Sharing projects through Mergin

After synchronising is completed, you should be able to see the point and its associated form on your QGIS.

Sharing projects through Mergin

Further reading

Input app’s manual can be found here.

With the Mergin service, multiple users can collect data on the same project. For more information, see this the blog post.

For more information on how to set up complex forms and map themes see QGIS documentation.

Who is behind QGIS at Oslandia ?

You are using QGIS and look for support services to improve your experience and solve problems ? Oslandia is here to help you with our full QGIS editor service range ! Discover our team members below.

You will probably interact first with our pre-sales engineer Bertrand Parpoil. He leads Geographical Information System projects for 15 years for large corporations, public administrations or hi-tech SME. Bertrand will listen to your needs and explore your use cases, to offer you the best set of services.

Régis Haubourg also takes part in the first steps of projects to analyze your usages and improve them. GIS Expert, he knows QGIS by heart and will make the most of its capabilities. As QGIS Community Manager at Oslandia, he is very active in the QGIS community of developers and contributors. He is president of the Francophone OSGeo local chapter ( OSGeo-fr ), QGIS voting member, organizes the French QGIS day conference in Montpellier, and participates to QGIS community meetings. Before joining Oslandia, he led the migration to QGIS and PostGIS at the Adour-Garonne Water Agency, and now guides our clients with their GIS migrations to OpenSource solutions. Régis is also a great asset when working on water, hydrology and other specific thematic subjects.

Loïc Bartoletti develops QGIS, specializing in features corresponding to his fields of interests : network management, topography, urbanism, architecture… We find him contributing to advanced vector editing in QGIS, writing Python plugins, namely for DICT management. Pushing CAD and migrations from CAD tools to GIS and QGIS is one of his major goals. He will develop your custom applications, combining technical expertise and functional competences. When bored, Loïc packages software on FreeBSD.

Vincent Mora is senior developer in Python and C++, as well as PostGIS expert. He has a strong experience in numerical simulation. He likes coupling GIS (PostGIS, QGIS) with 3D numerical computing for risk management or production optimization. Vincent is an official QGIS committer and can directly integrate your needs into the core of the software. He is also GDAL committer and optimizes low-level layers of your applications. Among numerous activities, Vincent serves as lead developer of the development team for Hydra Software, a tool dedicated to unified hydraulics and hydrology modelling and simulation based on QGIS.

Hugo Mercier is an officiel QGIS committer too for several years. He regularly talks in international conferences on PostGIS, QGIS and other OpenSource GIS softwares. He will implement your needs with new QGIS features, develop innovative plugins ( like QGeoloGIS ) and design and build your new custom applications, solving all kind of technological challenges.

Paul Blottière completes our QGIS committers : very active on core development, Paul has refactored the QGIS server component to bring it to an industry-grade quality level. He also designed and implemented the infrastructure allowing to guarantee QGIS server performances. He dedicated himself to QGIS server OGC certification, especially for WMS (1.3). Thanks to this work QGIS is now a reference OGC implementation.

Julien Cabièces recently joined Oslandia, and quickly dived into QGIS : he contributes to the core of this Desktop GIS, on the server component, as well as applications linked to numerical simulation. Coming from a satellite imagery company with industrial applications, he uses his flexibility to answer all your needs. He brings quality and professionalism to your projects, minimizing risks for large production deployments.

You may also meet Vincent Picavet. Oslandia’s founder is a voting member, and is involved in the project’s evolution and the organization of the community.

Aside from these core contributors, all other Oslandia members also master QGIS integrate this tool into their day-to-day projects.

Bertrand, Régis, Loïc, Vincent (x2), Hugo, Paul et Julien are in tune with you and will be happy to work together for your migrations, application development, and all your desires to contribute to the QGIS ecosystem. Do not hesitate to contact us !

First working MovingPandas setup on Databricks

In December, I wrote about GeoPandas on Databricks. Back then, I also tried to get MovingPandas working but without luck. (While GeoPandas can be installed using Databricks’ dbutils.library.installPyPI("geopandas") this PyPI install just didn’t want to work for MovingPandas.)

Now that MovingPandas is available from conda-forge, I gave it another try and … *spoiler alert* … it works!

First of all, conda support on Databricks is in beta. It’s not included in the default runtimes. At the time of writing this post, “6.0 Conda Beta” is the latest runtime with conda:

Once the cluster is up and connected to the notebook, a quick conda list shows the installed packages:

Time to install MovingPandas! I went with a 100% conda-forge installation. This takes a looong time (almost half an hour)!

When the installs are finally done, it get’s serious: time to test the imports!


Now we can put the MovingPandas data structures to good use. But first we need to load some movement data:

Or course, the points in this GeoDataFrame can be plotted. However, the plot isn’t automatically displayed once plot() is called on the GeoDataFrame. Instead, Databricks provides a display() function to display Matplotlib figures:

MovingPandas also uses Matplotlib. Therefore we can use the same approach to plot the TrajectoryCollection that can be created from the GeoDataFrame:

These Matplotlib plots are nice and quick but they lack interactivity and therefore are of limited use for data exploration.

MovingPandas provides interactive plotting (including base maps) using hvplot. hvplot is based on Bokeh and, luckily, the Databricks documentation tells us that bokeh plots can be exported to html and then displayed using  displayHTML():

Of course, we could achieve all this on MyBinder as well (and much more quickly). However, Databricks gets interesting once we can add (Py)Spark and distributed processing to the mix. For example, “Getting started with PySpark & GeoPandas on Databricks” shows a spatial join function that adds polygon information to a point GeoDataFrame.

A potential use case for MovingPandas would be to speed up flow map computations. The recently added aggregator functionality (currently in master only) first computes clusters of significant trajectory points and then aggregates the trajectories into flows between these clusters. Matching trajectory points to the closest cluster could be a potential use case for distributed computing. Each trajectory (or each point) can be handled independently, only the cluster locations have to be broadcast to all workers.

Flow map (screenshot from MovingPandas tutorial 4_generalization_and_aggregation.ipynb)


Public Service Announcement: Update to the latest point release now

QGIS users who have adopted the 3.10 version when initially released at the end of October 2019 have likely noticed a sharp drop in reliability. The underlying issues have now been addressed in 3.10.2, all users are advised to update *now*.

When QGIS 3.10 was first released in the end of October 2019, a pair of libraries – namely GDAL and PROJ – were updated to their next-generation versions. The advantages are plenty: GeoPDF export[1] support, more accurate coordinate transformation, etc. For those interested, more technical information on this is available here[2].

The update of these crucial libraries led to a number of regressions. While we expected some issues to arise, the seriousness of the disruption caught us off guard. Yet, it was also somewhat inevitable: QGIS is the first large GIS project to expose these next-generation libraries to the masses. The large number of QGIS users across the globe were essentially stress testing both new code within QGIS as well as the libraries themselves.

Thanks to dedicated users taking time to file in report and the community helping out as well as our project sponsors for allowing us to fund development time, developers have been able to fix all known regressions in both in QGIS as well as underlying GDAL and PROJ libraries, benefiting a large number of open source projects.

As a result of this collective effort by the community, QGIS 3.10.2 is now back to being the reliable and stable GIS software we all love. As such, we cannot stress enough the importance of updating now.

Once again, thanks to our community of testers, sponsors, and developers for their countless hours and efforts in making QGIS better.

Happy mapping!


(Fr) Oslandia recrute : Ingénieur(e) développement d’applications SIG ( Python / SQL / QGIS )

Sorry, this entry is only available in French.

QGIS Processing, Model Designer and ETL Campaign crowdfund launched!

QGIS Processing offers a rich and expandable set of algorithms which can operate on spatial data, along with a powerful Model Designer which allows users to string together these algorithms to create custom workflows.

Since its introduction in QGIS 2, the Processing framework has seen an intensive amount of development and optimisation efforts. In recent QGIS releases it offers a very user-friendly way of performing complex spatial data processing tasks, all without requiring ANY expensive third-party tools or software licenses!

At North Road we are passionate about the QGIS Processing framework, and have invested considerable effort in this framework over the past 5 years. We’re proud to announce that our latest crowd-funding campaign is focused on further expanding the capabilities and flexibility of Processing and the Processing Model Designer!

Unlike a typical crowdfunding campaign, where a specific funding target and deadline is set, we’re running this campaign a little differently. Instead, this campaign is taking the form of a “à la carte” menu of Processing enhancements. These range from small “paper-cut” style fixes, through to larger architectural improvements, and are each individually priced accordingly. We are asking backers to pick individual enhancements from this “menu of enhancements” and fund that enhancement’s development in full. In order to make this campaign affordable for a wide range of backers, we’ve included a huge range of enhancements which vary in price from smaller amounts to larger amounts.

You can read the full details of the campaign and browse the list of proposed enhancements at the campaign page.

QGIS Abstract Connections API


The goal of the new API is twofold:

  1. provide a unified way to store and retrieve data provider connections in the QGIS settings
  2. provide an abstract set of methods to perform most common operation on DB data sources (e.g. browse tables, drop/create a table/schema, run arbitrary SQL commands etc.)


The new API is documented in and it provides a few specializations for DB connections ( and an initial PR implementation for web service-based connections (


While the whole of the desired refactoring work was too big for a single grant request, the first work package has been completed and the following data providers have been partially or totally refactored to make use of the new connections API:

  • postgres
  • geopackage (OGR)
  • spatialite


The new API was also used to implement the automatic loading of layer dependencies (not part of the grant program).


For developers interested in working with the new API, a set Python tests are available to show how to use the methods: (see also the postgres and spatialite companion tests).


There is still a large amount of work to be done in order to complete all the desired refactoring and to remove all the Python and C++ code that will be ultimately be made redundant. In particular, future work should be undertaken to:

  • port all remaining data providers to the new API
  • refactor and eliminate the remaining DB-manager connectors to make use of the abstract API
  • eliminate duplicate and untested code inside the Processing framework for working with Postgres databases and port the code to the new, stable, well-tested API
  • refactor and eliminate the remaining QGIS browser data items to make use of the abstract API 


For further information, the following paragraphs (taken from the original grant proposal) will provide full details about the background of this work.


  • DB-Manager is an important part of the QGIS interface, which allows browsing/previews of different DB-based data sources, complex queries, management of layers, import-export etc., DB creation and deletion etc.
  • After the QGIS 3.0 release, improvements within the core browser widgets implemented in C++ have resulted in a (constantly growing) degree of overlapping functionality between the browser and db manager.
  • After QGIS 3 API improvements concerning layer import and export functionality, there are many duplicated implementations between browser and db manager – some functionality is better in browser, some functionality is better in db manager. Users are forced to choose between two competing semi-complete alternatives, instead of having one, complete, well integrated solution.
  • There are no unit tests for DB-Manager and this leads to frequent regressions, which (aside from being frustrating for users) consume a substantial part of our development time and budget during the bugfixing programs. Furthermore the nature of large Python codebases like db manager makes it very easy to accidentally break functionality with no warning or errors during development.


Proposed solution

We propose to start refactoring the DB-manager plugin functionality into core C++ implementation, reusing existing core API and replacing redundant duplicate functionality.

The clear advantages are:

  • no duplicate functionality, so it’s easier for users to understand and use
  • more usage of well-tested and well-maintained core C++ API
  • testability and immediate feedback on API breaks (an advantage of C++ is that the application won’t even build if an API is changed or accidentally misused)
  • better performance
  • the ability to expose database management functionality via stable PyQGIS API, allowing other plugins and scripts to utilise this functionality. In future, Processing algorithms may also be developed which would take advantage of these functions (e.g. “create schema”, “drop table”, “vacuum table” algorithms)
  • DB management functionality would be available within the main QGIS window (from the Browser panel), instead of as a separate dialog.


Grant proposal package

The above mentioned work is too large to be completed within a single grant, so what we propose here is to start the refactoring needed in order to have a core stable C++ API that can be used by the application and the plugins and that will be available to fully move DB manager to C++ API in the future avoiding duplication of code and functionality.

  • create an interface for databases that expose the required functions to a coherent API
  • add missing tests and documentation for the a.m. API
  • porting some basic functions from db manager to the new api:
    • create table (with native field types support)
    • create schema
    • delete table
    • Rename table

The API will be exposed through the browser and it will be used by the DB manager instead of the Python implementation that is currently used.

Overview of QGIS 3.12 Mesh Features

Ready for 3D meshes, vector streamlines or contour export?

The releases of QGIS 3.12, MDAL 0.5.0 and Crayfish 3.2.1 are planned for end of February 2020. We are proud to present you few of upcoming features we implemented for this release:

  • vector trace animation
  • 3D stacked meshes
  • mesh calculator enhancements
  • export contours
  • various smaller enhancements (reference time support, resampling, export plot data, mdal_translate utility)

If you are hesitant to wait till end of February, feel free to get nightly build and test it out!

Support for vector trace animation and streamlines (QGIS)

Last feature from QGIS 2.x/Crayfish 2.x series that was not ported to QGIS 3 is finally available. You would be able to visualize streamlines and particles for vector datasets in mesh layers. In QGIS main menu, under Mesh>Crayfish>Export Trace you are also able to export animation with the particle traces to various video formats

Trace animation

This feature was funded by TUFLOW

Support for 3d Stacked Meshes (e.g. TUFLOW FV format)

MDAL and QGIS now supports 3D Stacked Meshes, particularly for TUFLOW-FV format. For this release, you need to choose appropriate averaging method in the QGIS interface and you are able to browse the data similarly to any other 2D dataset. 3d stacked

In Crayfish 3.2.1, you can create plots of the profile showing the variation along Z-axis.

3d stacked plot

The technical description can be found in the following QEP

This feature was funded by TUFLOW

On the fly resampling of data defined on faces to vertices

For datasets defined on faces, one can choose to interpolate data to vertices with neighbour average method. When no data interpolation method is chosen, each pixel on a single face has a single value/color. With data on vertices, the rendering for each pixel is interpolated from the values on the vertices, making smoother figures.

Use mesh contours styling panel to switch between the data interpolation methods.

No Mesh Data Resampling Dialog Mesh Data Resampling Mesh Data Resampling Dialog

This feature was funded by Austrian Ministry of Agriculture, Forestry, Environment and Water Management

Smooth export of the contours (Crayfish processing algorithm)

We have implemented a new algorithm in QGIS’s analysis library to export directly contour lines and polygons. The method is not based on GDAL as it was in the Crayfish 2.x releases. It is both faster and with smoother shapes, matching rendered images from QGIS. You can find the new processing algorithm in Crayfish processing toolbox.

Mesh Contours

This feature was funded by Austrian Ministry of Agriculture, Forestry, Environment and Water Management

Support of datasets defined on faces in QGIS Mesh Calculator

From QGIS 3.12 you can use mesh calculator for all datasets, both defined on faces and vertices. Additionally, it allows users to store the result of mesh calculator under different name or format. This allows for example to work with FLO-2D or HEC-RAS data in the QGIS mesh calculator

This feature was funded by Austrian Ministry of Agriculture, Forestry, Environment and Water Management

Support for reference time (QGIS)

For various dataset type, for example GRIB and NetCDF, the reference time in QGIS time settings dialog is prepopulated from the raw data and does not need to be set manually. Also we fixed various bugs related to time parsing, so in QGIS 3.12 it should be possible to format and show your time in plots/animations in proper way.

Reference Time

This feature was funded by TUFLOW

Support for conversion of 2dm to UGRID mesh (mdal_translate utility)

MDAL library now has a new utility: mdal_translate. For now, use can use the utility to convert text-based 2dm mesh definition files to UGRID NetCDF/HDF5 binary-based format and save up to 80% disk and speed up loading of your mesh by similar amount.

This feature was funded by TUFLOW

Support for export of 2D plot data (processing)

With Crayfish 3.2.1 you can export your time series or cross section raw dat to CSV format for further processing.

This feature was funded by Lutra Consulting

Movement data in GIS #27: extracting trip origin clusters from MovingPandas trajectories

This post is a follow-up to the draft template for exploring movement data I wrote about in my previous post. Specifically, I want to address step 4: Exploring patterns in trajectory and event data.

The patterns I want to explore in this post are clusters of trip origins. The case study presented here is an extension of the MovingPandas ship data analysis notebook.

The analysis consists of 4 steps:

  1. Splitting continuous GPS tracks into individual trips
  2. Extracting trip origins (start locations)
  3. Clustering trip origins
  4. Exploring clusters

Since I have already removed AIS records with a speed over ground (SOG) value of zero from the dataset, we can use the split_by_observation_gap() function to split the continuous observations into individual trips. Trips that are shorter than 100 meters are automatically discarded as irrelevant clutter:

traj_collection.min_length = 100
trips = traj_collection.split_by_observation_gap(timedelta(minutes=5))

The split operation results in 302 individual trips:

Passenger vessel trajectories are blue, high-speed craft green, tankers red, and cargo vessels orange. Other vessel trajectories are gray.

To extract trip origins, we can use the get_start_locations() function. The list of column names defines which columns are carried over from the trajectory’s GeoDataFrame to the origins GeoDataFrame:

origins = trips.get_start_locations(['SOG', 'ShipType']) 

The following density-based clustering step is based on a blog post by Geoff Boeing and uses scikit-learn’s DBSCAN implementation:

from sklearn.cluster import DBSCAN
from geopy.distance import great_circle
from shapely.geometry import MultiPoint

origins['lat'] = origins.geometry.y
origins['lon'] = origins.geometry.x
matrix = origins.as_matrix(columns=['lat', 'lon'])

kms_per_radian = 6371.0088
epsilon = 0.1 / kms_per_radian

db = DBSCAN(eps=epsilon, min_samples=1, algorithm='ball_tree', metric='haversine').fit(np.radians(matrix))
cluster_labels = db.labels_
num_clusters = len(set(cluster_labels))
clusters = pd.Series([matrix[cluster_labels == n] for n in range(num_clusters)])
print('Number of clusters: {}'.format(num_clusters))

Resulting in 69 clusters.

Finally, we can add the cluster labels to the origins GeoDataFrame and plot the result:

origins['cluster'] = cluster_labels

To analyze the clusters, we can compute summary statistics of the trip origins assigned to each cluster. For example, we compute a representative (center-most) point, count the number of trips, and compute the mean speed (SOG) value:

def get_centermost_point(cluster):
    centroid = (MultiPoint(cluster).centroid.x, MultiPoint(cluster).centroid.y)
    centermost_point = min(cluster, key=lambda point: great_circle(point, centroid).m)
    return Point(tuple(centermost_point)[1], tuple(centermost_point)[0])
centermost_points = 

The largest cluster with a low mean speed (indicating a docking or anchoring location) is cluster 29 which contains 43 trips from passenger vessels, high-speed craft, an an undefined vessel:

To explore the overall cluster pattern, we can plot the clusters colored by speed and scaled by the number of trips:

Besides cluster 29, this visualization reveals multiple smaller origin clusters with low speeds that indicate different docking locations in the analysis area.

Cluster locations with high speeds on the other hand indicate locations where vessels enter the analysis area. In a next step, it might be interesting to compute flows between clusters to gain insights about connections and travel times.

It’s worth noting that AIS data contains additional information, such as vessel status, that could be used to extract docking or anchoring locations. However, the workflow presented here is more generally applicable to any movement data tracks that can be split into meaningful trips.

For the full interactive ship data analysis tutorial visit

This post is part of a series. Read more about movement data in GIS.

QGIS Snapping improvements

A few months ago, we proposed to the QGIS grant program to make improvements to the snap cache in QGIS. The community vote selected our project which was funded by Developments are now mostly finished.

In short, snapping is crucial for editing geospatial features. It is the only way to ensuring they are topologically related, ie, connected vertices have exactly the same coordinates even if manual digitizing on screen is imprecise by nature.  Snapping correctly supposes QGIS have in memory an indexed cache of the geometries to snap to. And maintainting this cache when data is modified, sometimes by another user or database logic, can be a real challenge. This it exactly what this work adresses.

The proposal was divided into two different tasks:

  • Manage circular dependencies
  • Relax the snap cache index build

Manage cicular data dependencies

Data dependencies

Data dependency is an existing feature that allows you to configure QGIS to reload layers (and their snapping cache) when a layer is modified.

It is useful when you store your data in a database and you set up triggers to maintain consistency between the different tables of your data model.

For instance, say you have topological informations containing lines and nodes. Nodes are part of lines and lines go through nodes. Then, you move a node in QGIS, and save your modifications to the database. In order to keep the data consistent, a trigger updates the geometry of the line going through the modified node.

Node 2 is modified, Line 1 is updated accordingly

QGIS, as a database client, has no information that the line layer currently displayed in the canvas needs to be refreshed after the trigger. Although the map canvas will be up to date, because QGIS fetches data for display without any caching system, the snapping cache is not and you’ll end up with ghost snapping highlights issues.

Snapping highlights (light red) differ from real line (orange)

Defining a dependency between nodes and lines layers tells QGIS that it has to refresh the line layer when a node is modified.

Dependencies configuration: Lines layer will be refreshed whenever Nodes layer is modified

It also have to work the other way, modifying a line should update the nodes to ensure they still are on the line.

Circular data dependencies

So here we are, lines depend on nodes which depend on lines which depend on nodes which…

That’s what circular dependencies is about. This specific behavior was previously forbidden and needed a special way to deal with it. Thanks to this recent development, it is now possible.

It’s also possible to add the layer itself as one of its own dependencies. It helps dealing with specific cases where one feature modification could lead to a modification of another feature in the same layer (to keep consistency on road networks for instance).

Road 2 is modified, Road 1 is updated accordingly

This feature is available in the next QGIS LTR version 3.10.

Relax the snapping cache index build

If you work in QGIS with huge projects displaying a lot of vector data, and you enable snapping while editing these data, you probably already met this dialog:

Snap indexing dialog

This dialog informs you that data are currently being indexed so you can snap on them while you will edit feature geometry. And for big projects, this dialog can last for a really long time. Let’s work on speeding it up!

What’s a snap index?

Let’s say you want to move a line and snap it onto another one. While you drag your line with the mouse, QGIS will look for an existing geometry beneath the mouse cursor (with a certain pixel tolerance) every time you move your mouse. Without spatial index, QGIS will have to go through every geometry in your layer to check if the given geometry is beneath the cursor position. This would be very ineffective.

In order to prevent this, QGIS keeps an index where vector data are stored in a way that it can quickly find out what geometry is beneath the mouse cursor. The building of this data structure takes time and that is what the progress dialog is about.

Firstly: Parallelize snap index build

If you want to be able to snap on all layers in your project, then QGIS will have to build one snap index for each layer. This operation was made sequentially meaning that if you have for instance 20 layers and the index building last approximatively 3 seconds for each, then the whole index building will last 1 minute. We made modifications to QGIS so that index building could be done in parallel. As a result, the total index building time could theoretically be 3 seconds!

4 layers snap index being built in parallel

However, parallel operations are limited by the number of CPU cores of your machine, meaning that if you have 4 cores (core i7 for instance) then the total time will be up to 4 times faster than when the building is sequential (and last 15 seconds in our example).

Secondly: relax the snap build

For big projects, parallelizing index building is not enough and still takes too much time. Futhermore, to reduce snap index building, an existing optimisation was to build the spatial index for a specific area of interest (determined according to the displayed area and layer size). As a consequence, when you’ve done waiting for an index currently building and you move the map or zoom in/out, you could possibly trigger another snap index building and wait again.

So, the idea was to avoid waiting at all. Snap index is now built whenever it needs to (when you first enable snapping, when you move or zoom) but the user doesn’t have to wait for the build to be over and can continue what it was doing (creating feature, moving…). Snapping highlights will be missing when the index is currently being built and will appear gradually as soon as they finished. That’s what we call the relaxing mode.

No waiting dialog, snapping highlights appears as soon as snap index is ready

This feature has been merged into current QGIS master and will be present in future QGIS 3.12 release. We keep working on this feature in order to make it more stable and efficient.

What’s next

We’ll continue to improve this feature in the coming days, if you have the chance to test it and encounter issues please let us know on the QGIS tracker. If you think about a missing feature or just want to know more about QGIS, feel free to contact us at [email protected]. And please have a look at our support offering for QGIS.

Many thanks to QGIS grant program for funding these new features. Thanks also to all the people involved in reviewing the code and helping to better understand the existing mechanism.


Reports from the winning grant proposals 2018

With the QGIS Grant Programme 2018, we were able to support seven proposals that were aimed to improve the QGIS project, including software, infrastructure, and documentation. These are the reports on the work that has been done within the individual projects:

  1. Increased stability for Processing GUI and External Providers (Nyall Dawson)
    Many bugs in 3rd party providers have been fixed and lots of new unit tests added. The GUI includes new C++ classes and a  new framework that landed in QGIS 3.4. For more details see Nyall’s report on the mailing list.
  2. OSGeo4W updates (Jürgen Fischer)
    The updates performed in this project were essential to bring QGIS 3.x to Windows.
  3. Resurrect Processing “R” Provider (Nyall Dawson)
    The R provider has been implemented as a provider plugin. The plugin’s beta phase was first announced in Nov 2018 and the plugin is now available for general use.
  4. OpenCL support for processing core algs (Alessandro Pasotti)
    The following processing algorithms have been ported: slope, aspect, hillshade, and ruggedness. Even if was not in scope for this QEP, the hillshade renderer has also been optimized. For more details see qgis/QGIS#7451.
  5. QGIS server OGC compliant and certified for WFS (Régis Haubourg)
    This project fixed numerous issues to get closer to the goal of getting QGIS Server WFS certified. However, the project ran out of resources before the goal could be achieved. For details see the current WFS tests status page.
  6. Charts and drawings on attribute forms (Matthias Kuhn)
    For details read “The new QML widgets in QGIS” and see qgis/QGIS#7801.
  7. Update of QGIS Training Manual (Matteo Ghetta)
    This project hasn’t been completed yet.

Thank you to everyone who participated and made this round of grants a great success and thank you to all our sponsor and donors who make this initiative possible!

Movement data in GIS #26: towards a template for exploring movement data

Exploring new datasets can be challenging. Addressing this challenge, there is a whole field called exploratory data analysis that focuses on exploring datasets, often with visual methods.

Concerning movement data in particular, there’s a comprehensive book on the visual analysis of movement by Andrienko et al. (2013) and a host of papers, such as the recent state of the art summary by Andrienko et al. (2017).

However, while the literature does provide concepts, methods, and example applications, these have not yet translated into readily available tools for analysts to use in their daily work. To fill this gap, I’m working on a template for movement data exploration implemented in Python using MovingPandas. The proposed workflow consists of five main steps:

  1. Establishing an overview by visualizing raw input data records
  2. Putting records in context by exploring information from consecutive movement data records (such as: time between records, speed, and direction)
  3. Extracting trajectories & events by dividing the raw continuous tracks into individual trajectories and/or events
  4. Exploring patterns in trajectory and event data by looking at groups of the trajectories or events
  5. Analyzing outliers by looking at potential outliers and how they may challenge preconceived assumptions about the dataset characteristics

To ensure a reproducible workflow, I’m designing the template as a a Jupyter notebook. It combines spatial and non-spatial plots using the awesome hvPlot library:

This notebook is a work-in-progress and you can follow its development at Your feedback is most welcome!



  • Andrienko G, Andrienko N, Bak P, Keim D, Wrobel S (2013) Visual analytics of movement. Springer Science & Business Media.
  • Andrienko G, Andrienko N, Chen W, Maciejewski R, Zhao Y (2017) Visual Analytics of Mobility and Transportation: State of the Art and Further Research Directions. IEEE Transactions on Intelligent Transportation Systems 18(8):2232–2249, DOI 10.1109/TITS.2017.2683539

GRASS GIS 7.8.2 released

GRASS GIS 7.8.2 released with updated PROJ 6 and GDAL 3 support

What’s new in a nutshell

As a follow-up to the recent GRASS GIS 7.8.1 we have pusblished the new stable release GRASS GIS 7.8.2.
Besides other improvements, the release contains important PROJ 4/5/6 related datum handling fixes, wxGUI fixes and a fix for the vector import from PostGIS databases.

An overview of the new features in the 7.8 release series is available at new features in GRASS GIS 7.8.

Binaries/Installer download:

Source code download:

See also our detailed announcement:

First time users may explore the first steps tutorial after installation.


The Geographic Resources Analysis Support System (, commonly referred to as GRASS GIS, is an Open Source Geographic Information System providing powerful raster, vector and geospatial processing capabilities in a single integrated software suite. GRASS GIS includes tools for spatial modeling, visualization of raster and vector data, management and analysis of geospatial data, and the processing of satellite and aerial imagery. It also provides the capability to produce sophisticated presentation graphics and hardcopy maps. GRASS GIS has been translated into about twenty languages and supports a huge array of data formats. It can be used either as a stand-alone application or as backend for other software packages such as QGIS and R geostatistics. It is distributed freely under the terms of the GNU General Public License (GPL). GRASS GIS is a founding member of the Open Source Geospatial Foundation (OSGeo).

The GRASS Development Team, December 2019

The post GRASS GIS 7.8.2 released appeared first on GFOSS Blog | GRASS GIS and OSGeo News.

Getting started with PySpark & GeoPandas on Databricks

Over the last years, many data analysis platforms have added spatial support to their portfolio. Just two days ago, Databricks have published an extensive post on spatial analysis. I took their post as a sign that it is time to look into how PySpark and GeoPandas can work together to achieve scalable spatial analysis workflows.

If you sign up for Databricks Community Edition, you get access to a toy cluster for experimenting with (Py)Spark. This considerably lowers the entry barrier to Spark since you don’t need to bother with installing anything yourself. They also provide a notebook environment:

I’ve followed the official Databricks GeoPandas example notebook but expanded it to read from a real geodata format (GeoPackage) rather than from CSV.

I’m using test data from the MovingPandas repository: demodata_geolife.gpkg contains a hand full of trajectories from the Geolife dataset. Demodata_grid.gpkg contains a simple 3×4 grid that covers the same geographic extent as the geolife sample:

Once the files are downloaded, we can use GeoPandas to read the GeoPackages:

Note that the display() function is used to show the plot.

The same applies to the grid data:

When the GeoDataFrames are ready, we can start using them in PySpark. To do so, it is necessary to convert from GeoDataFrame to PySpark DataFrame. Therefore, I’ve implemented a simple function that performs the conversion and turn the Point geometries into lon and lat columns:

To compute new values for our DataFrame, we can use existing or user-defined functions (UDF). Here’s a simple hello world function and associated UDF:

A spatial UDF is a little more involved. For example, here’s an UDF that finds the first polygon that intersects the specified lat/lon and returns that polygon’s ID. Note how we first broadcast the grid DataFrame to ensure that it is available on all computation nodes:

It’s worth noting that PySpark has its peculiarities. Since it’s a Python wrapper of a strongly typed language, we need to pay close attention to types in our Python code. For example, when defining UDFs, if the specified return type (Integertype in the above example) does not match the actual value returned by the find_intersection() function, this will cause rather cryptic errors.

To plot the results, I’m converting the joined PySpark DataFrame back to GeoDataFrame:

I’ve published this notebook so you can give it a try. (Any notebook published on Databricks is supposed to stay online for six months, so if you’re trying to access it after June 2020, this link may be broken.)

The plugin for Dutch Spatial Zoning plans shows the effectiveness of an Open Source approach

Rececently the ruimtelijkeplannen plugin for using dutch spatial zoning plans in QGIS was renewed. A lot of extra functionality was added, sponsored by LBP|SIGHT, a company which uses this plugin frequently. As it happened, we now have a small list list of organizations who contributed to the development of this plugin, illustrating the power of … Continue reading The plugin for Dutch Spatial Zoning plans shows the effectiveness of an Open Source approach

Remarks on SVN-trac to GitHub migration

GRASS GIS is an open source geoinformation system which is developed by a globally distributed team of developers. Besides the source code developers also message translators, people who write documentation, those who report bugs and wishes and more are involved.

1. Early days… from pre-Internet to CVS and SVN

While GRASS GIS is under development since 1982 (no typo!) it has been put into a centralized source code management system in December 1999. Why so late? Because the World Wide Web (WWW) became available in the 1990s along with tools like browsers and such, followed by the development of distributed source code management tools. We moved on 29th Dec 1999 (think Y2K bug) the entire code into our instance of CVS (Concurrent Versioning System). With OSGeo being founded in 2006, we migrated the CVS repository to SVN (Subversion for the source code management) and trac (bug and wish tracker) on 8 Dec 2007.

2. Time to move on: git

Now, after more than 10 years using SVN/trac time had come to move on and join the large group of projects managing their source code in git (see also our related Wiki page on migration). Git comes with numerous advantages, yet we needed to decide which hosting platform to use. Options where,, gitlab or gitea on OSGeo infrastructure, or other platforms. Through a survey we found out that the preference among contributors is GitHub. While not being open source itself it offers several advantages: it is widely known (good to get new developers interested and involved), numerous OSGeo projects are hosted there under the GitHub “OSGeo organization“.

If all fails (say, one day GitHub no longer being a reasonable choice) the import of our project from GitHub to GitLab is always possible. Indeed, we meanwhile mirror our code on OSGeo’s gitea server.

Relevant script code and migration ticket:

Relevant steps:

  • migrated SVN trunk -> git master
  • migrated and tagged release branches (milestones)
  • deleted “develbranch6” (we compared it to “releasebranch_6_4” and didn’t discover relevant differences)
  • Fix commit messages (yes, we really wanted to be brave, updating decades of commit messages!):
    • references to old RT tracker tickets (used Dec 2000 – Dec 2006)
    • references to old GForge tracker tickets (used Jan 2007 – Dec 2008)
    • references to other trac tickets (#x ->…)

3. Source code migration: the new git repositories

  • github repository “grass” (repo)

    • Source code from 1999 to present day (SVN-trunk -> git-master)
    • all 7.x release branches
  • github repository “grass-legacy” (repo)

    • separate repository for older GRASS GIS releases (3.2, 4.x, 5.x, 6.x), hence source code now available in git since 1987!
  • github repository “grass-addons” (repo)

    • repository for addons
  • github repository “grass-promo” (repo)
    • repository for promotional material
  • github repository “grass-website” (repo)
    • repository for upcoming new Website

4. Remarks on the “grass-legacy” repository

What special about it:

  • the source code goes back to 1987!
  • file timestamps (which I tried to preserve for decades :-) have been used to reconstruct the source code history (e.g., releasebranch_3_2)
  • junk files removed (plenty of leftover old binary files, files consisting of a special char only etc)
  • having this grass-legacy repo available in parallel to the main grass repo which contains the  recent source code we have a continuous source code coverage from 1987 to today in git.
  • size is about 250MB

What’s missing

  • the 4.3 source code doesn’t have distinct timestamps. Someone must have once packaged without mtime preservation… a pity. Perhaps a volunteer may fix that by carrying over the timestamps from GRASS GIS 4.2 in case the md5sum of a file is identical (or so).

5. Trac issue migration

A series of links had to be updated. Martin Landa invested days and days on that (thanks!!). He used the related GDAL efforts as a basis (Even Rouault: thanks!). As the date for the trac migration we selected 2007-12-09 (r25479) as it was the first SVN commit (after the years in CVS). The migration of trac bugs to github (i.e. transfer of trac ticket content) required several steps:

Link updates in the ticket texts:

  • links to other tickets (now to be pointed to full trac URL). Note that there were many styles of referring in the commit log message which had to be parsed accordingly
  • links to trac wiki (now to be pointed to full trac URL)
  • links source code in SVN (now to be pointed to full trac URL)
  • images and attachments (now to be pointed to full trac URL)


  • “operating system” trac label into the github issue text itself (following the new issue reporting template)
  • converting milestones/tickets/comments/labels
  • converting trac usernames to Github usernames
  • setting assignees if possible, set new “grass-svn2git” an assignee otherwise
  • slowing down transfer to match the 60 requests per second API limit rate at github

6. Fun with user name mapping

Given GRASS GIS’ history of 35+ years we had to invest major effort in identifying and mapping user names throughout the decades (see also bug tracker history). The following circumstances could be identified:

  • user present in CVS but not in SVN
  • user present in SVN but not in CVS
  • user present in both with identical name
  • user present in both with different name (well, in our initial CVS days in 1999 we often naivly picked our surnames like “martin”, “helena”, “markus”, “michael” … cute yet no scaling very much over the years!) as some were changed in the CVS to SVN migration in 2007, leading to
    • colliding user names
  • some users already having a github account (with mostly different name again)

We came up with several lookup tables, aiming at catching all variants. Just a “few” hours to dig in old source code files and in emails for finding all the missing email addresses…

7. Labels for issues

We cleaned up the trac component of the bug reports, coming up with the following categories which have to be visually grouped by color since the label list is just sorted alphabetically in github/gitlab:

  • Issue category:
    • bug
    • enhancement
  • Issue solution (other than fixing and closing it normally):
    • duplicate
    • invalid
    • wontfix
    • worksforme
  • Priority:
    • blocker
    • critical
    • feedback needed
  • Components:
    • docs
    • GUI
    • libs
    • modules
    • packaging
    • python
    • translations
    • unittests
    • Windows specific

Note that the complete issue migration is still to be done (as of Nov. 2019). Hopefully addressed at the GRASS GIS Community Sprint Prague 2019.

8. Setting up the github repository

In order to avoid users being flooded by emails due to the parsing of user contributions which normally triggers an email from github) we reached out to GitHub support in order to temporarily disable these notifications until all source code and selected issues were migrated.

The issue conversion rate was 4 min per trac bug to be converted and uploaded to github. Fairly slow but likely due to the API rate limit imposed and the fact that the migration script above generates a lot of API requests rather than combined ones..
Note to future projects to be migrated: use the new gihub import API (unfortunately we got to know about its existence too late in our migration process).

Here out timings which occurred during the GRASS GIS project migration from SVN to github:

  • grass repo: XX hours (all GRASS GIS 7.x code)
  • grass-legacy repo: XX hours (all GRASS GIS 3.x-6.x code)
  • NNN issues: XX hours – forthcoming.

9. New issue reporting template

In order to guide the user when reporting new issues, we will develop a small template – forthcoming.

10. Email notifications: issues to grass-dev and commits to grass-commit

We changed the settings from SVN post-hook to Github commit notifications and they flow in smoothly into the grass-commit mailing list. Join it to follow the development.

Overall, after now several months of using our new workflow we can state that things work fine.

The post Remarks on SVN-trac to GitHub migration appeared first on GFOSS Blog | GRASS GIS and OSGeo News.

QGIS Server is ready for the new OGC API for Features protocol.

The new OGC API for Features (OAPIF) (also formerly known as WFS3) is one of the first protocols of the new generation of OGC web services and we are happy to announce that QGIS Server is ready to serve data following the specifications of this new protocol.

A lot of work has been going on during last summer to make sure QGIS Server was ready to support the new family of REST APIs, the underlying architecture allows in fact to expand QGIS Server API capabilities with any kind of new API that will be available in the future.

The new API is very similar to the well known WFS, but it also comes with a distinct set of features like content negotiations, REST actions, HTML templates, JSON as a first class citizen, self-documentation of the API (following OpenAPI specifications) and a preliminary implementation (the specifications are not yet finalized) of the simple transactions.

The new API is already in the QGIS Server documentation, it only misses the transaction part because the specifications are not yet final and we don’t want people start relying on an API that is probably going to change quite soon.

The vast majority of this new development has been possible thanks to the volunteer work of our core developers but we also wish to thank OSGeo and QGIS sustaining members and donors for funding a substantial part of the following activities:

OPENAPI validation (completed)
Online demo (TODO)
CI validation/ OGC CITE (started)
Expose Schema (completed)
Simple Transactions (completed)
Returned fields filter (completed)
Documentation (completed except for transactions)
JSON performance comparison with WFS (TODO)
Time filter support (completed)

Enjoy the new API and beware that this is only the first of a brand new series of OGC APIs that will make much easier for users to interact with data and for developers to create applications that consume those data.

Text provided by Alessandro Pasotti (QGIS core developer)

Tracking, calculating and merging vector changes with Input and QGIS

The latest beta release of Input (v.0.4.90) comes with smart diff support for vectors. This will allow you to use the app (current beta version) in a collaborative environment, where several users can make changes to a single survey layer (geopackage).

What does it mean?

It is often the case, where multiple users need to make changes to a single vector layer. If you work in office, this issue is usually addressed by having a central geodatabase (e.g. Postgres/PostGIS). If you want extra information (e.g. audit trails, versioning, latest changes, etc), you can modify your database, to keep track of it.

The problem arises when you want to collect data, without having access to the central geodatabase. You can do some manual work to handle this scenario, but it can easily lead into data management nightmare.

To simplify the workflow, we have developed Geodiff, a multi-platform library to keep track of changes, calculate the differences, merge and consolidate the differences.

How can I use it?

Geodiff has been integrated into the beta version of Input. This will allow you to share a project with your team and edit a single layer (geopackage format) all at the same time, even when you are offline.

To start with, you need to create a project and upload it to Mergin. You can then share the project (with write access) with your colleagues:

Sharing projects through Mergin

The project contains a survey layer (trees.gpkg). There is only one feature present within the layer:

Sharing projects through Mergin

The project is shared with two users. Both users download the project and take their devices to the field:

User 1, carried out a survey (using iPhone!), by adding a tree and editing the attribute table of the existing one. The changes were synchronised back through the Mergin:

Meanwhile, User 2 added a new feature to the survey layer. Once User 2 tries to synchronise the changes, Input automatically detects the changes not only made through User 2, but also the ones uploaded to the Mergin. The layer will be patched both locally and on the server to take all the changes into account:

The data administrator can now pull all the changes from both users in QGIS:

You can also see the history of changes to the project and the survey layer on the Mergin website. Below shows the changes from different users:

Sharing projects through Mergin

To see changes from each user, you can click on the version and it lists the changes. In this example, User 1 (jack) added a new feature and modified an existing feature:

Sharing projects through Mergin

You can also see the extended history and see where the changes have been made:

Sharing projects through Mergin

How can I test this new feature?

You can use Beta version of Input app in Android or TestFlight in iOS:

Get it on Google Play Get it on App Store

For any issues or feedback, please file a ticket on Input repository

(Nederlands) Kort verslag oprichtingsvergadering QGIS gebruikersgroep

Sorry, this entry is only available in the Dutch language

(Nederlands) Lid worden?

Sorry, this entry is only available in the Dutch language

  • Page 1 of 118 ( 2350 posts )
  • >>

Back to Top