Page 1 of 11 (212 posts)

  • talks about »
  • gis

Tags

Last update:
Wed May 22 19:05:19 2019

A Django site.

QGIS Planet

QField 1.0 is here

Let’s get straight to the point

It’s official, QField for QGIS 1.0 is out!

Get it while it’s hot on the Playstore (qfield.org/get) or on GitHub

We are incredibly pleased and proud of just having released such a jewel and are convinced that, thanks to all its features and conscious design choices, QField will make your field digitizing work much more efficient and pleasant.

Packed with loads of useful features like online and offline features digitizing, geometry and attributes editing, attribute search, powerful forms, theme switching, GPS support, camera integration and much more, QField is the powerful tool for those who need to edit on the go and would like to avoid standing in the swamp with a laptop or paper charts.

Let’s see what makes QField probably* the best mobile GIS in the world.

Work efficiently

QField focuses on efficiently getting GIS field work done and combines a minimal design with sophisticated technology to get data from the field to the office in a comfortable and easy way.

Fast and reactive

Thanks to the underlying QGIS engine and a lot of optimizations, QField is powerful and snappy. Even with complex projects, QField is a joy to work with.

Easy handling

Conscious design choices and a continuous focus on a minimal user interface drive QField’s development. This allows us to deliver a product wich is uncluttered and extremly user-friendly

Quickly digitise

Allowing a seamless digitizing experience is a paramount goal of QField. Thanks to a cleverly designed adaptive user interface and specific features like real-time attribute checks and snapping support, QField allows its users to be extremely time efficient.

Unmatched feature set

To be the best, you need to be clever but also skillful.

QField’s efficiency is matched only by its featureset that allows its users to make the most out of their fieldwork time.

Powerful cartography combined with full text search

The beauty of GIS is that maps are dynamic. Layers can individually be shown and hidden and information can be presented more or less prominently based on the task at hand. QField supports the endless styling possibilities offered by QGIS and thanks to a well placed theme switcher you can change the looks of the entire project with a single click. For even more customizability, QField allows hiding and showing layers by simply long-pressing on the layer name.

Furthermore, QField boasts a fully configurable attribute text search that will allow you to geolocate and edit that exact object you were looking for.

Geometry editing

Editing Geometries on the field is probably the most complex task an operator has to deal with. QField simplifies this process through an adaptive toolbar that appears only when necessary, snapping support and a crosshair digitizer.

Thanks to these enhancements, QField allows reducing the error rate significantly.

Support for high precision GNSS

Simple internal GPS accuracy might be enough for basic projects but cadastral surveying and other high accuracy digitizations have much higher requirements. QFields natively listens to the Android location services so it can take advantage of the best location provided by external devices.

Generate PDF

Thanks to QField’s native support for generating PDFs based on QGIS’s print layouts, your on the fly daily report map is just one click away.

Intuitive project chooser

When dealing with multiple projects, quickly being able to switch between them is key. QField comes with a beautiful file selector with favorite directories (long press on a folder to add it to the favorites and long press on the favorites list to remove it) and an automatic list of the last three opened projects that will save you heaps of time while looking for your projects.

Your data – Your decisions

QField does not impose any constraint on the data model, it is your data and you decide what they should look like and what values are acceptable. QField can enforce constraints for you and you can choose among various type of widgets to represent your data. QGIS will preconfigure some field types automatically, all you’ll have then to do is tweak the settings if you want and your project is ready for mobile prime time. Our documentation has all the information you need.

Extends your Geo Data Infrastructure seamlessly

QField uses QGIS to set up maps and forms so it automatically supports a wide variety of data formats. Thanks to this, you can comfortably prepare your project once and then deploy it everywhere. And since QGIS also has a server component, your project can be served on a WebGIS with the very same beautiful looks.

In fact you can see this exact infrastructure up and running under demo.qfield.org and with the “online_survey.qgs” project included in the QField demo projects.

Synchronize with WiFi, Cable or Network

You can synchronize your project and data (in case you are not using a centralized online database) using various methods thanks to our QFieldSync plugin.

Future cloud integration

In the near future we will add a cloud synchronization functionality, so that you will be able to seamlessly manage your project online and have them automatically deployed to your devices.

Installing and contributing

You can easily install QField using the Playstore (qfield.org/get), find out more on the documentation site (qfield.org), watch some demo videos on our channel (qfield.org/demo) and report problems to our issues tracking system (qfield.org/issues). Please note that the Playstore update can take some hours to roll out and if you had installed a version directly from GitHub, you might have to uninstall it to get the latest Playstore update.

QField, like QGIS, is an open source project. Everyone is welcome to contribute to making the product even better – whether it is with financial support, translation, documentation work, enthusiastic programming or visionary ideas.

We would like to thank our fantastic community for all the great translations, documentations, bug reports and general feedback they gave us. Thanks to all this, we were able to fix plenty of bugs, address performance issues and even add some super cool new features.

Development and deployment services

As masterminds behind QField and core contributor to QGIS, we are the perfect partner for your project. If you want to help us build a better QField or QGIS, or if you need any services related to the whole QGIS stack, don’t hesitate to contact us.

OPENGIS.ch

OPENGIS.ch helps you setting up your spatial data infrastructure based on seamlessly integrated desktop, web, and mobile components.
We support your team in planning, developing, deploying and running your infrastructure. Thanks to several senior geodata infrastructure experts, QGIS core developers and the makers of the mobile data acquisition solution QField, OPENGIS.ch has all it takes to make your project a success. OPENGIS.ch is known for its commitment to high-quality products and its continuous efforts to improve the open source ecosystem.

* We might be biased, but we do believe it

QField RC5 – Last call for testing

We are really happy to announce the fifth and (hopefully) last 1.0 release candidate in QField’s history! This means that QField 1.0 is closer than ever.

Get it while it’s hot on the Playstore (https://qfield.org/get) or on GitHub

Thanks to all the feedback by the fantastic community we were able to fix plenty of bugs, address performance issues and even add some super cool new features.

New file selector

Among the new features, the most important is the flashy new file selector with favorite directories (long press on a folder to add it to the favorites and longpress on the favorites list to remove it) and an automatic list of the last three opened projects that will save you heaps of time while looking for your projects.

Another lifesaver is the newly added support for pasting text from the clipboard in the search bar. Finally, we added a smart and unobtrusive “rate this app” dialog to make it easier for you to give QField the ★★★★★ you always wanted to give it 🙂

Search functionality

List of improvements since RC3

  • New Custom file selector (#476)
  • Favorite directories in file selector (#507)
  • Recent projects in file selector (#499)
  • Ripple effect in file selector (#505)
  • Smart unobtrusive “rate this app” dialog (#510)
  • clear value in date/time if invalid when losing focus (#464)
  • fix crash when switching layer (#498)
  • Respect DPI in multiline fontsize
  • Value Map compatibility with QGIS 2 and lazy loading for performance improvements
  • Use external valuemap model
  • allow to copy text from clipboard in search bar
  • respect keep scale option in locator
  • optimize scale when searching points (#472)
  • add frame to search results
  • Update to Qt 5.12.1 (for android 6+)

You can easily install QField using the Playstore (https://qfield.org/get), find out more on the documentation site (https://qfield.org), watch some demo videos on our channel (https://qfield.org/demo) and report problems to our issues tracking system (https://qfield.org/issues). Please note that the Playstore update can take some hours to roll out and if you had installed a version directly from GitHub, you might have to uninstall it to get the latest playstore update.

QField, like QGIS, is an open source project. Everyone is welcome to contribute making the product even better – whether it is with financial support, enthusiastic programming, translation and documentation work or visionary ideas.

If you want to help us build a better QField or QGIS, or need any services related to the whole QGIS stack don’t hesitate to contact us.

Stand-alone PyQGIS scripts with OSGeo4W

PyQGIS scripts are great to automate spatial processing workflows. It’s easy to run these scripts inside QGIS but it can be even more convenient to run PyQGIS scripts without even having to launch QGIS. To create a so-called “stand-alone” PyQGIS script, there are a few things that need to be taken care of. The following steps show how to set up PyCharm for stand-alone PyQGIS development on Windows10 with OSGeo4W.

An essential first step is to ensure that all environment variables are set correctly. The most reliable approach is to go to C:\OSGeo4W64\bin (or wherever OSGeo4W is installed on your machine), make a copy of qgis-dev-g7.bat (or any other QGIS version that you have installed) and rename it to pycharm.bat:

Instead of launching QGIS, we want that pycharm.bat launches PyCharm. Therefore, we edit the final line in the .bat file to start pycharm64.exe:

In PyCharm itself, the main task to finish our setup is configuring the project interpreter:

First, we add a new “system interpreter” for Python 3.7 using the corresponding OSGeo4W Python installation.

To finish the interpreter config, we need to add two additional paths pointing to QGIS\python and QGIS\python\plugins:

That’s it! Now we can start developing our stand-alone PyQGIS script.

The following example shows the necessary steps, particularly:

  1. Initializing QGIS
  2. Initializing Processing
  3. Running a Processing algorithm
import sys

from qgis.core import QgsApplication, QgsProcessingFeedback
from qgis.analysis import QgsNativeAlgorithms

QgsApplication.setPrefixPath(r'C:\OSGeo4W64\apps\qgis-dev', True)
qgs = QgsApplication([], False)
qgs.initQgis()

# Add the path to processing so we can import it next
sys.path.append(r'C:\OSGeo4W64\apps\qgis-dev\python\plugins')
# Imports usually should be at the top of a script but this unconventional 
# order is necessary here because QGIS has to be initialized first
import processing
from processing.core.Processing import Processing

Processing.initialize()
QgsApplication.processingRegistry().addProvider(QgsNativeAlgorithms())
feedback = QgsProcessingFeedback()

rivers = r'D:\Documents\Geodata\NaturalEarthData\Natural_Earth_quick_start\10m_physical\ne_10m_rivers_lake_centerlines.shp'
output = r'D:\Documents\Geodata\temp\danube3.shp'
expression = "name LIKE '%Danube%'"

danube = processing.run(
    'native:extractbyexpression',
    {'INPUT': rivers, 'EXPRESSION': expression, 'OUTPUT': output},
    feedback=feedback
    )['OUTPUT']

print(danube)

Call for testing: GRASS GIS with Python 3

Please help us testing the Python3 support in the yet unreleased GRASS GIS trunk (i.e., version “grass77” which will be released as “grass78” in the near future).

1. Why Python 3?

Python 2 is end-of-life (EOL); the current Python 2.7 will retire in 11 months from today (see https://pythonclock.org). We want to follow the “Moving to require Python 3” and complete the change to Python 3. And we need a broader community testing.

2. Download and test!

Packages are available at time:

3. Instructions for testing

4. Problems found? Please report them to us

Problems and bugs can be reported in the GRASS GIS trac. Code changes are welcome!

Thanks for testing grass77!

The post Call for testing: GRASS GIS with Python 3 appeared first on GFOSS Blog | GRASS GIS and OSGeo News.

Dealing with delayed measurements in (Geo)Pandas

Yesterday, I learned about a cool use case in data-driven agriculture that requires dealing with delayed measurements. As Bert mentions, for example, potatoes end up in the machines and are counted a few seconds after they’re actually taken out of the ground:

Therefore, in order to accurately map yield, we need to take this temporal offset into account.

We need to make sure that time and location stay untouched, but need to shift the potato count value. To support this use case, I’ve implemented apply_offset_seconds() for trajectories in movingpandas:

    def apply_offset_seconds(self, column, offset):
        self.df[column] = self.df[column].shift(offset, freq='1s')

The following test illustrates its use: you can see how the value column is shifted by 120 second. Geometry and time remain unchanged but the value column is shifted accordingly. In this test, we look at the row with index 2 which we access using iloc[2]:

    def test_offset_seconds(self):
        df = pd.DataFrame([
            {'geometry': Point(0, 0), 't': datetime(2018, 1, 1, 12, 0, 0), 'value': 1},
            {'geometry': Point(-6, 10), 't': datetime(2018, 1, 1, 12, 1, 0), 'value': 2},
            {'geometry': Point(6, 6), 't': datetime(2018, 1, 1, 12, 2, 0), 'value': 3},
            {'geometry': Point(6, 12), 't': datetime(2018, 1, 1, 12, 3, 0), 'value':4},
            {'geometry': Point(6, 18), 't': datetime(2018, 1, 1, 12, 4, 0), 'value':5}
        ]).set_index('t')
        geo_df = GeoDataFrame(df, crs={'init': '31256'})
        traj = Trajectory(1, geo_df)
        traj.apply_offset_seconds('value', -120)
        self.assertEqual(traj.df.iloc[2].value, 5)
        self.assertEqual(traj.df.iloc[2].geometry, Point(6, 6))

From CSV to GeoDataFrame in two lines

Pandas is great for data munging and with the help of GeoPandas, these capabilities expand into the spatial realm.

With just two lines, it’s quick and easy to transform a plain headerless CSV file into a GeoDataFrame. (If your CSV is nice and already contains a header, you can skip the header=None and names=FILE_HEADER parameters.)

usecols=USE_COLS is also optional and allows us to specify that we only want to use a subset of the columns available in the CSV.

After the obligatory imports and setting of variables, all we need to do is read the CSV into a regular DataFrame and then construct a GeoDataFrame.

import pandas as pd
from geopandas import GeoDataFrame
from shapely.geometry import Point

FILE_NAME = "/temp/your.csv"
FILE_HEADER = ['a', 'b', 'c', 'd', 'e', 'x', 'y']
USE_COLS = ['a', 'x', 'y']

df = pd.read_csv(
    FILE_NAME, delimiter=";", header=None,
    names=FILE_HEADER, usecols=USE_COLS)
gdf = GeoDataFrame(
    df.drop(['x', 'y'], axis=1),
    crs={'init': 'epsg:4326'},
    geometry=[Point(xy) for xy in zip(df.x, df.y)])

It’s also possible to create the point objects using a lambda function as shown by weiji14 on GIS.SE.

Visualization of borehole logs with QGIS

At Oslandia, we have been working on a component based on the QGIS API for the visualization of well and borehole logs.

This component is aiming at displaying data collected vertically along wells dug underground. It mainly focuses on data organized in series of contiguous samples, and is generic enough to be used for both vertical (where the Y axis corresponds to a depth underground) and horizontal data (where the X axis corresponds to time for example). One of the main concerns was to ensure good display performances with an important volume of sample points (usually hundred of thousands sample points).

We have already been working on plot components in the past, but for specific QGIS plugins (both for timeseries and stratigraphic logs) and thought we will put these past experiences to good use by creating a more generic library.

We decided to represent sample points as geometry features in order to be able to reuse the rich symbology engine of QGIS. This allows users to represent their data whatever they like without having to rewrite an entire symbology engine. We also benefit from all the performance optimizations that have been added and polished over the years (on-the-fly geometry simplification for example).

Albeit written in Python, we achieved good display performances. The key trick was to avoid copy of data between the sample points read by QGIS and the Python graphing component. It is achieved thanks to the fact that the PyQGIS API has some functions that respect the buffer protocol.

You can have a look at the following video to see this component integrated in an existing plugin.

Visualization of borehole logs withing QGIS

It is distributed as usual under an open source license and the code repository can be found on GitHub.

Do not hesitate to contact us ( [email protected] ) if you are interested in any enhancements around this component.

PyQGIS101 part 10 published!

PyQGIS 101: Introduction to QGIS Python programming for non-programmers has now reached the part 10 milestone!

Beyond the obligatory Hello world! example, the contents so far include:

If you’ve been thinking about learning Python programming, but never got around to actually start doing it, give PyQGIS101 a try.

I’d like to thank everyone who has already provided feedback to the exercises. Every comment is important to help me understand the pain points of learning Python for QGIS.

I recently read an article – unfortunately I forgot to bookmark it and cannot locate it anymore – that described the problems with learning to program very well: in the beginning, it’s rather slow going, you don’t know the right terminology and therefore don’t know what to google for when you run into issues. But there comes this point, when you finally get it, when the terminology becomes clearer, when you start thinking “that might work” and it actually does! I hope that PyQGIS101 will be a help along the way.

Visualize Postgres JSON data in QML widgets

As promised some time ago in “The new QML widgets in QGIS – When widgets get unbridled” we still owe you some fancy unicorns, but first let’s have a look at another nice feature that has been introduced in QGIS 3.4 LTR,  the reading of PostgreSQL JSON and JSONB types.
With JSON you have a lot of possibilities for storing unstructured data. In our case, it’s mainly interesting when the data are stored as an array or a JSON object. Let’s have a look at two examples.

Visualize Postgres JSON data with common widgets

With the usual QGIS widgets “List” and “Key/Value” you are able to display JSON arrays and simple JSON objects.

JSON array as List

[
    "European dark bee",
    "Carniolan honey bee",
    "Buckfast bee"
]

Simple JSON object as Key/Value

{
    "nomenclatura":"Apis mellifera mellifera",
    "name":"European dark bee",
    "link":"https://en.wikipedia.org/wiki/European_dark_bee"
}

Or of course both as plain text in the “Text Edit” widget:

Say hi to Postgres JSON in QML widget

Probably, your JSON data does not look really nice with the aforementioned widgets, luckily since QGIS 3.4, you are free to create your own QML widget. Since QGIS already loads the JSON data into structures that are supported by QML, we can use all the JSON data within the QML code.
Let’s assume you have the JSON array from above and you like the elegance of the blue of Jacques Majorelle. You create your personal list widget by adding the JSON field as an expression:

import QtQuick 2.0
Rectangle {
    width: 310; height: 250; color: "grey"
    Column {
        anchors.horizontalCenter: parent.horizontalCenter
        anchors.verticalCenter: parent.verticalCenter
        spacing: 5
        Repeater {
            model:expression.evaluate("\"jvalue\"")
            Rectangle {
                color: "#6050dc"
                width: 300; height: 50; radius: 10.0
                Text {
                    anchors.centerIn: parent
                    font.pointSize: 24
                    text: modelData
                }
            }
        }
    }
}

You will have your very personal list:

JSON also allows storing more complex data, like for example a list of objects. In that case, you will reach the limits of the common QGIS widgets.
Let’s assume you have a table looking like this:

nomenclatura name link
Apis mellifera mellifera European dark bee https://en.wikipedia.org/wiki/European_dark_bee
Apis mellifera carnica Carniolan honey bee https://en.wikipedia.org/wiki/Carniolan_honey_bee
Apis mellifera Buckfast bee https://en.wikipedia.org/wiki/Buckfast_bee

In JSON it would be stored like this:

[
    {"nomenclatura":"Apis mellifera mellifera","name":"European dark bee","link":"https://en.wikipedia.org/wiki/European_dark_bee"},
    {"nomenclatura":"Apis mellifera carnica","name":"Carniolan honey bee","link":"https://en.wikipedia.org/wiki/Carniolan_honey_bee"},
    {"nomenclatura":"Apis mellifera","name":"Buckfast bee","link":"https://en.wikipedia.org/wiki/Buckfast_bee"}
]

With the QML Widget you can use the QML TableView to visualize:

import QtQuick 2.0
import QtQuick.Controls 1.4
TableView {
    width: 600
    model: expression.evaluate("\"jvalue\"")
    TableViewColumn {
        role: "nomenclatura"
        title: "Nomenclature"
        width: 200
    }
    TableViewColumn {
        role: "name"
        title: "Name"
        width: 200
    }
    TableViewColumn {
        role: "link"
        title: "Wikipedia"
        width: 200
    }
}


Or, even more powerful, you can create your super individual table using the model and create each row by using a QML Repeater.
Additionally, you can use a lot of fancy stuff like:

  • mouse interaction
  • animation
  • opening an external link
  • … and so on


The QML code for that looks like this.

import QtQuick 2.0
Rectangle {
    width: 610; height: 500
    Column {
        anchors.horizontalCenter: parent.horizontalCenter
        anchors.verticalCenter: parent.verticalCenter
        Repeater {
            model: expression.evaluate("\"jvalue\"")
            Row {
                id: theRow
                height: mouseArea1.containsMouse || mouseArea2.containsMouse || mouseArea3.containsMouse ? 70 : 50;
                Rectangle { color: "lightblue";
                            border.width: 1
                            width: 150; height: parent.height
                            Text { anchors.centerIn: parent
                                   font.pointSize: 10; text: modelData.nomenclatura }
                            MouseArea {
                                id: mouseArea1
                                anchors.fill: parent
                                hoverEnabled: true
                            }
                }
                Rectangle { color: "lightgreen";
                            border.width: 1
                            width: 150; height: parent.height
                            Text { anchors.centerIn: parent
                                   font.pointSize: 10; text: modelData.name }
                            MouseArea {
                                id: mouseArea2
                                anchors.fill: parent
                                hoverEnabled: true
                            }
                }
                Rectangle {
                            id: linkField
                            color: "lightyellow";
                            border.width: 1
                            width: 300; height: parent.height
                            Text { anchors.centerIn: parent
                                   font.pointSize: 10; text: modelData.link }
                            MouseArea {
                                id: mouseArea3
                                anchors.fill: parent
                                hoverEnabled: true
                                onPressed: linkField.state = "PRESSED"
                                onReleased: linkField.state = "RELEASED"
                                onClicked: Qt.openUrlExternally(modelData.link)
                            }
                            states: [
                                State {
                                    name: "PRESSED"
                                    PropertyChanges { target: linkField; color: "green"}
                                },
                                State {
                                    name: "RELEASED"
                                    PropertyChanges { target: linkField; color: "lightyellow"}
                                }
                            ]
                            transitions: [
                                Transition {
                                    from: "PRESSED"
                                    to: "RELEASED"
                                    ColorAnimation { target: linkField; duration: 1000}
                                },
                                Transition {
                                    from: "RELEASED"
                                    to: "PRESSED"
                                    ColorAnimation { target: linkField; duration: 1000}
                                }
                            ]
                }
            }
        }
    }
}

And that’s it

I hope you liked reading and you will enjoy using it to make beautiful widgets and forms. If you have questions or inputs, feel free to add a comment.
… and in case you still asking where the promised unicorns are. Here’s is a super-fancy implementation 😉

QGIS Server 3 : OGC Certification work for WFS 1.1.0

QGIS Server is an open source OGC data server which uses QGIS engine as backend. It becomes really awesome because a simple desktop qgis project file can be rendered as web services with exactly the same rendering, and without any mapfile or xml coding by hand.

QGIS Server provides a way to serve OGC web services like WMS, WCS, WFS and WMTS resources from a QGIS project, but can also extend services like GetPrint which takes advantage of QGIS’s map composer power to generate high quality PDF outputs.

Since the 3.0.2 version, QGIS Server is certified as an official OGC reference implementation for WMS 1.3.0 and reports are generated in a daily continuous integration to avoid regressions.

 

Thus, the next step was naturally to take a look at the WFS 1.1.0 thanks to the support of the QGIS Grant Program

Side note, this Grant program is made possible thanks to your direct sponsoring and micro-donations to QGIS.org.

TEAM Engine test suite for WFS 1.1.0

We use a tool provided by the OGC Compliance Program to run dedicated tests on the server : Teamengine (Test, Evaluation, And Measurement Engine).

Test suites are available through a web interface. However, for the needs of continuous integration, these tests have to be run without user interaction. In the case of WMS 1.3.0, nothing more easy than using the REST API:

[pastacode lang=”bash” manual=”%24%20curl%20%22http%3A%2F%2Flocalhost%3A8081%2Fteamengine%2Frest%2Fsuites%2Fwms%2F1.20%2Frun%3Fqueryable%3Dqueryable%26basic%3Dbasic%26capabilities-url%3Dhttp%3A%2F%2F172.17.0.2%2Fqgisserver%3FREQUEST%3DGetCapabilities%2526SERVICE%3DWMS%2526VERSION%3D1.3.0%2526MAP%3D%2Fdata%2Fteamengine_wms_130.qgs” message=”” highlight=”” provider=”manual”/]

 

However, the WFS 1.1.0 test suite does not provide a REST API and makes the situation less straightforward. We switched to using TEAM Engine directly from command line:

[pastacode lang=”bash” manual=”%24%20cd%20te_base%0A%24%20.%2Fbin%2Funix%2Ftest.sh%20-source%3Dwfs%2F1.1.0%2Fctl%2Fmain.ctl%20-form%3Dparams.xml” message=”” highlight=”” provider=”manual”/]

The params.xml file allows to configure underlying tests. In this particular case, the GetCapabilities URL of the QGIS Server to test is given.  Results are available thanks to the viewlog.sh shell script:

[pastacode lang=”bash” manual=”%24%20.%2Fbin%2Funix%2Fviewlog.sh%20-logdir%3Dte_base%2Fusers%2Froot%2F%20-session%3Ds0001%0ATest%20wfs%3Awfs-main%20type%20Mandatory%20(s0001)%20Failed%20(InheritedFailure)%0A%20%20%20Test%20wfs%3Areadiness-tests%20type%20Mandatory%20(s0001%2Fd68e38807_1)%20Failed%20(InheritedFailure)%0A%20%20%20%20%20%20Test%20ctl%3ASchematronValidatingParser%20type%20Mandatory%20(s0001%2Fd68e38807_1%2Fd68e588_1)%20Failed%0A%20%20%20%20%20%20Test%20wfs%3Abasic-main%20type%20Mandatory%20(s0001%2Fd68e38807_1%2Fd68e636_1)%20Failed%20(InheritedFailure)%0A%20%20%20%20%20%20%20%20%20Test%20wfs%3Arun-GetCapabilities-basic-cc-GET%20type%20Mandatory%20(s0001%2Fd68e38807_1%2Fd68e636_1%2Fd68e28810_1)%20Failed%20(InheritedFailure)%0A%20%20%20%20%20%20%20%20%20%20%20%20Test%20wfs%3Awfs-1.1.0-Basic-GetCapabilities-tc1%20type%20Mandatory%20(s0001%2Fd68e38807_1%2Fd68e636_1%2Fd68e28810_1%2Fd68e1095_1)%20Passed%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20Test%20ctl%3Aassert-xpath%20type%20Mandatory%20(s0001%2Fd68e38807_1%2Fd68e636_1%2Fd68e28810_1%2Fd68e1095_1%2Fd68e1234_1)%20Passed%0A%20%20%20%20%20%20%20%20%20%20%20%20Test%20wfs%3Awfs-1.1.0-Basic-GetCapabilities-tc2%20type%20Mandatory%20(s0001%2Fd68e38807_1%2Fd68e636_1%2Fd68e28810_1%2Fd68e1100_1)%20Passed%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20Test%20ctl%3Aassert-xpath%20type%20Mandatory%20(s0001%2Fd68e38807_1%2Fd68e636_1%2Fd68e28810_1%2Fd68e1100_1%2Fd68e1305_1)%20Passed%0A%20%20%20%20%20%20%20%20%20%20%20%20Test%20wfs%3Awfs-1.1.0-Basic-GetCapabilities-tc3%20type%20Mandatory%20(s0001%2Fd68e38807_1%2Fd68e636_1%2Fd68e28810_1%2Fd68e1105_1)%20Passed%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20Test%20ctl%3Aassert-xpath%20type%20Mandatory%20(s0001%2Fd68e38807_1%2Fd68e636_1%2Fd68e28810_1%2Fd68e1105_1%2Fd68e1558_1)%20Passed%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20Test%20ctl%3Aassert-xpath%20type%20Mandatory%20(s0001%2Fd68e38807_1%2Fd68e636_1%2Fd68e28810_1%2Fd68e1105_1%2Fd68e1582_1)%20Passed%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20Test%20ctl%3Aassert-xpath%20type%20Mandatory%20(s0001%2Fd68e38807_1%2Fd68e636_1%2Fd68e28810_1%2Fd68e1105_1%2Fd68e1606_1)%20Passed%0A…%0A…” message=”” highlight=”” provider=”manual”/]

Finally, a Python script has been written to read these logs and generate HTML report easily readable. Thanks to our QGIS-Server-CertifSuite, providing the continuous integration infrastructure with Docker images, these reports are also generated daily.

Bugfix and Conclusion

First results were clear: a lot of work is necessary to have a QGIS Server certified for WFS 1.1.0!

We started fixing the issues one by one:

And now we have a much better support than 6 months ago

However, some work still need to be done to finally obtain the OGC certification for WFS 1.1.0. To be continued!

Please contact us if you want QGIS server to become a reference implementation for all OGC service !

The new QML widgets in QGIS – When widgets get unbridled

Individuality is the definition of freedom. And freedom is the fundamental requirement of man’s mind. QGIS possibly cannot give you all the freedom you require in life. But at least a lot of freedom in how you manage your work. QGIS 3.4.0 LTR was released last week and it comes loaded with features supporting big freedom in the configuration of your projects.  Let’s focus on the QML Widget. QML is the smart casual look of widgets. With the help of some simple code, you will be able to visualize your data in the attribute form like never before. You can display beautiful charts, complex JSON data, and fancy colored unicorns. 

How it’s done

Let’s start with an example. In the Attribute Form configuration in the Layer Properties you have first to activate the drag and drop designer. Not only can you drag the field- and relation-items from the available widget list to the form layout list, but also a QML Widget. When you drop this item, it creates an “instance” of a QML Widget. This means, you can drag and drop as many QML Widgets as you like to have on your form and configure each of them individually.

QML (Qt Modelling Language) is a user interface specification and programming language. It allows developers and designers alike to create highly performant, fluidly animated and visually appealing applications. QML offers a highly readable, declarative, JSON-like syntax with support for imperative JavaScript expressions combined with dynamic property bindings. Source: Qt documentation

It’s a bit like an HTML page on the attribute form but very well integrated with Qt.
On dropping the item, the configuration dialog pops up. After closing you can come back to it by double-clicking the item in the form layout list, like you do it with containers and tabs. If your don’t know QML that much yet, the default snippets from the drop-down can create an example of a rectangle, a pie chart or a bar chart. This could help you to create your own widget. On the right you can see a preview of your widget in real time. There are powerful layout possibilities to design it according to your ideas. For more information about it see the QML layout part of the Qt documentation.

But a chart makes no sense if there is no data. Of course, you can enter the data directly into your QML code, but most likely you need the data of the features to be visualized. This brings us to expressions. You can use them like you are used to in Default Values, Constraints and Display Messages. You’ll find the well-known expression builder widget in this configuration as well.

Using expressions

So let’s assume we want to visualize who holds what share of a forest. These forests are owned by the country (national), the canton (cantonal) or private. To keep it simple we have three attributes for that: national_share,cantonal_share and private_share.
After creating the default pie chart you will find this snippet in the QML code text area:

PieSeries {
    id: pieSeries
    PieSlice { label: "First slice"; value: 25 }
    PieSlice { label: "Second slice"; value: 45 }
    PieSlice { label: "Third slice"; value: 30 }
}

Let’s set the field expressions into the PieSlice-values. Just select them in the expression widget and add them with the + into the chart.

As you can see the expressions in the code are wrapped inside expression.evaluate("<expression>"). This means there are no limits in using expressions.
You are open to use more complex expressions like e.g. for the title property of the pie chart:

title: expression.evaluate("CASE WHEN @layer_name LIKE \"forest\" THEN \"Forest\" ELSE @layer_name END")

Or in case the task with forest shares would be solved with relations to other layers by filling up a model with the children and the children’s share. This is possible by using expressions with the help of the expression functionrelation_aggregate.
More information about expressions can be found in the QGIS Documentation.
Back to our example. The result will look like this on the attribute form. It visualizes the share values in the pie chart.

The visualization is not (yet) updated in real time when the values change. But this would be a nice thing to have in the future… If you would like to support this, please contact us.

And that’s it

I hope you liked reading and you will enjoy using it to make beautiful widgets and forms. If you have questions or inputs, feel free to add a comment.
… and in case you still asking where the promised unicorns are. Well you have to wait for the part 2 of this article 😉
 

Geocoding with Geopy

Need to geocode some addresses? Here’s a five-lines-of-code solution based on “An A-Z of useful Python tricks” by Peter Gleeson:

from geopy import GoogleV3
place = "Krems an der Donau"
location = GoogleV3().geocode(place)
print(location.address)
print("POINT({},{})".format(location.latitude,location.longitude))

For more info, check out geopy:

geopy is a Python 2 and 3 client for several popular geocoding web services.
geopy includes geocoder classes for the OpenStreetMap Nominatim, ESRI ArcGIS, Google Geocoding API (V3), Baidu Maps, Bing Maps API, Yandex, IGN France, GeoNames, Pelias, geocode.earth, OpenMapQuest, PickPoint, What3Words, OpenCage, SmartyStreets, GeocodeFarm, and Here geocoder services.

QGIS speaks a lot of languages

QGIS is a real cosmopolitan. Born in Alaska sixteen years ago, it has spread all over the world since. Thanks to its open source mentality, it finds not only in economically strong countries big usergroups. No question, that beside all the developers, there is a bunch of brave translators giving everything to make and keep QGIS multilingual. It’s translated in over forty languages – even to Mandarin Chinese and Esperanto. Not only the application, but also its plugins.
And since the feature-loaded long term release 3.4.0 even the QGS-Projects themselves.  Thanks to the friendly support of QGIS Usergroup Switzerland and the QGEP Project.

How it comes

Plugins are often shipped with pre-configured project files. To provide them in the users individual language, you’ve been required to translate the project manually in the properties and store it separately. When you needed to change something, you have been coerced to update every single file. This is a big effort and fault-prone. So there appeared the idea to have translation files for each required language, and when the user opens the only one project, it will be translated to his specific language. And that’s, what this new functionality does.

How it’s done

Like QGIS and the plugins, the projects are translated with the Qt translation process. Means, it makes the translation according to a Qt Compiled Translation Source File (.qm file). When the user opens a project, QGIS checks for a .qm file laying in the same folder like the .qgs file, having the same name like the .qgs file and having the language-code as postfix of the users language (the language configured in the QGIS settings).
So when the user opens a project named “citybees.qgs” that is originally in English, but his QGIS language is German, it checks for a file named “citybees_de.qm”. The project’s layer names, field-aliases, container names and much more will be translated to German and the project will be automatically stored as “citybees_de.qgs”. So the user has his German project version and can use and edit it like he wishes. Super easy.

Start from the beginning

Swiss people love honey and so they are diligent beekeepers. So, let’s assume you want to provide a project about beekeeping in cities to your Swiss customers. Because in Switzerland people talk four different languages, you need to have the project multilingual in German, French and Italian. We skip Romansh, not because it’s less important or in the Romansh speaking parts are no cities, but because QGIS does not support Romansh (if you’ll propose this one day, you will have my vote).
Anyway. Let’s see what you have to do, to deliver the projects with the .qm file for German, French or Italian.

1. Create the project

You create your project about beekeeping and store it as “citybees.qgs”. You don’t have to care about languages at the moment. You name everything in your language. Assumed it’s English, you name the layers “apiary” and “area” and the fields “fid”, “bee-species”, “beekeeper” and so on.
On changes you will edit always this project and no translated projects.

2. Generate Ts File

In the project properties in the section General there is the part to generate a translation source file. First, you select the source language, to have this information in programs you’ll edit the file (like Qt Linguist or Transifex) afterwards. Per default the language of your QGIS is selected here.

When pressing Generate TS File you will find in your projects folder the new file “citybees.ts”.
But wait a minute, why are we generating a .ts file when we need a .qm file?
The .ts file is the translation source file and it’s the uncompiled .qm file. The .qm file contains compact binary format code Qt can make the translation of programs with, but you and your brave translator cannot read it. So you create a .ts file looking like this:

<!DOCTYPE TS>
<TS sourcelanguage="en_US">
 <context>
  <name>project:layers:apiary__offline__1f89f4fd_49da_4eb9_90b3_1f7b5e82840c</name>
  <message>
   <source>apiary</source>
   <translation type="unfinished"/>
  </message>
 </context>
 <context>
  <name>project:layers:apiary__offline__1f89f4fd_49da_4eb9_90b3_1f7b5e82840c:fieldaliases</name>
  <message>
   <source>fid</source>
   <translation type="unfinished"/>
  </message>
 </context>
 <context>
  <name>project:layers:apiary__offline__1f89f4fd_49da_4eb9_90b3_1f7b5e82840c:fieldaliases</name>
  <message>
   <source>bee_species</source>
   <translation type="unfinished"/>
  </message>
 </context>
 <context>
  <name>project:layers:apiary__offline__1f89f4fd_49da_4eb9_90b3_1f7b5e82840c:fieldaliases</name>
  <message>
   <source>beekeeper</source>
   <translation type="unfinished"/>
  </message>
 </context>
[...]

You see it’s simple XML code that contains mainly untranslated text in the <source> element and empty space for the translated text in the <translation> element.
You could enter your translations directly in this file using the text editor and then build the .qm file with the command lrelease, but it’s preferable to use tools like Qt Linguist orweb-based services like Transifex or Weblate.

3. Translate your File in Qt Linguist

You open Qt Linguist an you select the target language of the .qm file you want to build in the end. Let’s choose German.
The file opens and you see a list of entries described by the Context. The context is, where the strings are located in the QGIS project.

The string “beekeeper” stays in the context project:layers:apiary_1f7b5e82839c:fieldaliases and this means it’s a field or alias of the layer apiary_1f7b5e82839c in the project.
The translation is done simply over the graphical interface. To confirm your translation you can set the check mark.

4. Finally build your .qm file

You compile the translation – means build a .qm file – by simply select Release as… in the Qt Linguist and store it as “citybees_de.qm”.

And now your customer will be able to open your project in German 🙂

What’s translated

Most of the needed parameters like layer names and fields are translated. There could be still some strings in your use case that are not like for example action titles or labels. But the solution is designed, that it’s extendable for more project parameters (see the next chapter). So don’t worry if you will find parameters that cannot be translated yet. They possibly will be in the future.

Translatable strings:

  • layer names
  • layer group names
  • form attributes like tab titles and group box titles
  • relation names
  • field names and aliases
  • value relations

With field names and aliases it has a special behavior: while we should not translate the field names itself because they can be used as identification, the aliases are translated only. In case there is no alias in the original project, the translation of the field name would be written as an alias in the translated project. The field name stays the same. So you can just generate and translate without the fear of overwriting field names.

Content translation

There is a possibility to have a translation of the content as well. In particular using the value relation widget.
Let’s assume, you want to have the values in the field bee_species translated as well. Means the following bee species should be German:
– European dark bee
– Carniolan honey bee
– Buckfast bee

You can solve this by using the value relation widget. Means you create a non geometrical layer “beespecies” and enter the following values:

nomenclatura name_en name_de name_fr name_it
Apis mellifera mellifera European dark bee Dunkle Europäische Biene abeille noire ape nera
Apis mellifera carnica Carniolan honey bee Kärntner Biene abeille carniolienne ape carnica
Apis mellifera Buckfast bee Buckfastbiene abeille Buckfast ape Buckfast

And configure the field bee_species of apiary as value relation widget:

And now comes the magic:

When you create the .ts file it includes not only the field name bee_species but also the referenced value name_en:

 <context>
  <name>project:layers:apiary__offline__1f89f4fd_49da_4eb9_90b3_1f7b5e82840c:fields:bee_species:valuerelationvalue</name>
  <message>
   <source>name_en</source>
   <translation type="unfinished"/>
  </message>
 </context>

When you now “translate” the name_en to name_de the field is referenced to the German values of the entry:

Getting technical

Let’s have a quick look into the source code, shall we?

Generate Ts File

When you press the Generate Ts File button in the project properties, in the background happens the following:
First QGIS scans for the translatable strings in the currently loaded project and collects everything in a QgsTranslationContext object. This object contains the filename of the .ts file and all the collected translatable strings.
The strings are collected by firing a signal called requestForTranslatableObjects delivering the QgsTranslationContext. Every object that contains translatable strings connects a slot registering the strings in the received QgsTranslationContext:

void registerTranslatableObjects( QgsTranslationContext *translationContext )
{
  const QList<QgsLayerTreeLayer *> layers = mRootGroup->findLayers();
  for ( const QgsLayerTreeLayer *layer : layers )
  {
    translationContext->registerTranslation( QStringLiteral( "project:layers:%1" ).arg( layer->layerId() ), layer->name() );
[...]

This will be a growing list of strings as new features are added and missing bits are discovered.

Translate by QTranslator

The translation is made using the QTranslator. It’s loaded with the .qm file on reading the project and on loading of every single translatable string, it’s called to translate:

QString layername = mTranslator->translate( QStringLiteral( "project:layers:%1" ).arg( node.namedItem( QStringLiteral( "id" ) ).toElement().text() ), node.namedItem( QStringLiteral( "layername" ) ).toElement().text(), disambiguation, n );

That’s it

I hope you liked reading and if you have questions or inputs, feel free to add a comment.
Enjoy this cosmopolitious feature! 🙂

Plotting GPS Trajectories with error ellipses using Time Manager

This is a guest post by Time Manager collaborator and Python expert, Ariadni-Karolina Alexiou.

Today we’re going to look at how to visualize the error bounds of a GPS trace in time. The goal is to do an in-depth visual exploration using QGIS and Time Manager in order to learn more about the data we have.

The Data

We have a file that contains GPS locations of an object in time, which has been created by a GPS tracker. The tracker also keeps track of the error covariance matrix for each point in time, that is, what confidence it has in the measurements it gives. Here is what the file looks like:

data.png

Error Covariance Matrix

What are those sd* fields? According to the manual: The estimated standard deviations of the solution assuming a priori error model and error parameters by the positioning options. What it basically means is that the real GPS location will be located no further than three standard deviations across north and east from the measured location, most of (99.7%) the time. A way to represent this visually is to create an ellipse that maps this area of where the real location can be.ellipse_ab

An ellipse can be uniquely defined from the lengths of the segments a and b and its rotation angle. For more details on how to get those ellipse parameters from the covariance matrix, please see the footnote.

Ground truth data

We also happen to have a file with the actual locations (also in longitudes and latitudes) of the object for the same time frame as the GPS (also in seconds), provided through another tracking method which is more accurate in this case.

actual_data

This is because, the object was me running on a rooftop in Zürich wearing several tracking devices (not just GPS), and I knew exactly which floor tiles I was hitting.

The goal is to explore, visually, the relationship between the GPS data and the actual locations in time. I hope to get an idea of the accuracy, and what can influence it.

First look

Loading the GPS data into QGIS and Time Manager, we can indeed see the GPS locations vis-a-vis the actual locations in time.

actual_vs_gps

Let’s see if the actual locations that were measured independently fall inside the ellipse coverage area. To do this, we need to use the covariance data to render ellipses.

Creating the ellipses

I considered using the ellipses marker from QGIS.

ellipse_marker.png

It is possible to switch from Millimeter to Map Unit and edit a data defined override for symbol width, height and rotation. Symbol width would be the a parameter of the ellipse, symbol height the b parameter and rotation simply the angle. The thing is, we haven’t computed any of these values yet, we just have the error covariance values in our dataset.

Because of the re-projections and matrix calculations inherent into extracting the a, b and angle of the error ellipse at each point in time, I decided to do this calculation offline using Python and relevant libraries, and then simply add a WKT text field with a polygon representation of the ellipse to the file I had. That way, the augmented data could be re-used outside QGIS, for example, to visualize using Leaflet or similar. I could have done a hybrid solution, where I calculated a, b and the angle offline, and then used the dynamic rendering capabilities of QGIS, as well.

I also decided to dump the csv into an sqlite database with an index on the time column, to make time range queries (which Time Manager does) run faster.

Putting it all together

The code for transforming the initial GPS data csv file into an sqlite database can be found in my github along with a small sample of the file containing the GPS data.

I created three ellipses per timestamp, to represent the three standard deviations. Opening QGIS (I used version: 2.12, Las Palmas) and going to Layer>Add Layer>Add SpatialLite Layer, we see the following dialog:

add_spatialite2.png

After adding the layer (say, for the second standard deviation ellipse), we can add it to Time Manager like so:

add_to_tm

We do the process three times to add the three types of ellipses, taking care to style each ellipse differently. I used transparent fill for the second and third standard deviation ellipses.

I also added the data of my  actual positions.

Here is an exported video of the trace (at a place in time where I go forward, backwards and forward again and then stay still).

gps

Conclusions

Looking at the relationship between the actual data and the GPS data, we can see the following:

  • Although the actual position differs from the measured one, the actual position always lies within one or two standard deviations of the measured position (so, inside the purple and golden ellipses).
  • The direction of movement has greater uncertainty (the ellipse is elongated across the line I am running on).
  • When I am standing still, the GPS position is still moving, and unfortunately does not converge to my actual stationary position, but drifts. More research is needed regarding what happens with the GPS data when the tracker is actually still.
  • The GPS position doesn’t jump erratically, which can be good, however, it seems to have trouble ‘catching up’ with the actual position. This means if we’re looking to measure velocity in particular, the GPS tracker might underestimate that.

These findings are empirical, since they are extracted from a single visualization, but we have already learned some new things. We have some new ideas for what questions to ask on a large scale in the data, what additional experiments to run in the future and what limitations we may need to be aware of.

Thanks for reading!

Footnote: Error Covariance Matrix calculations

The error covariance matrix is (according to the definitions of the sd* columns in the manual):

sde * sde sign(sdne) * sdne * sdne
sign(sdne) * sdne * sdne sdn * sdn

It is not a diagonal matrix, which means that the errors across the ‘north’ dimension and the ‘east’ dimension, are not exactly independent.

An important detail is that, while the position is given in longitudes and latitudes, the sdn, sde and sdne fields are in meters. To address this in the code, we convert the longitude and latitudes using UTM projection, so that they are also in meters (northings and eastings).

For more details on the mathematics used to plot the ellipses check out this article by Robert Eisele and the implementation of the ellipse calculations on my github.

Celebrating 35 years of GRASS GIS!

Today marks 35 years of GRASS GIS development – with frequent releases the project keeps pushing the limits in terms of geospatial data processing quality and performance.

GRASS (Geographic Resources Analysis Support System) is a free and open source Geographic Information System (GIS) software suite used for geospatial data management and analysis, image processing, graphics and map production, spatial modeling, and 3D visualization. Since the major GRASS GIS 7 version, it also comes with a feature rich engine for space-time cubes useful for time series processing of Landsat and Copernicus Sentinel satellite data and more. GRASS GIS can be either used as a desktop application or as a backend for other software packages such as QGIS and R. Furthermore, it is frequently used on HPC and cloud infrastructures for massive parallelized data processing.

Brief history
In 1982, under the direction of Bill Goran at the U.S. Army Corps of Engineers Construction Engineering Research Laboratory (CERL), two GIS development efforts were undertaken. First, Lloyd Van Warren, a University of Illinois engineering student, began development on a new computer program that allowed analysis of mapped data.  Second, Jim Westervelt (CERL) developed a GIS package called “LAGRID – the Landscape Architecture Gridcell analysis system” as his master’s thesis. Thirty five years ago, on 29 July 1983, the user manual for this new system titled “GIS Version 1 Reference Manual” was first published by J. Westervelt and M. O’Shea. With the technical guidance of Michael Shapiro (CERL), the software continued its development at the U.S. Army Corps of Engineers Construction Engineering Research Laboratory (USA/CERL) in Champaign, Illinois; and after further expansion version 1.0 was released in 1985 under the name Geographic Resources Analysis Support System (GRASS). The GRASS GIS community was established the same year with the first annual user meeting and the launch of GRASSnet, one of the internet’s early mailing lists. The user community expanded to a larger audience in 1991 with the “Grasshopper” mailing list and the introduction of the World Wide Web. The users’ and programmers’ mailing lists archives for these early years are still available online.
In the mid 1990s the development transferred from USA/CERL to The Open GRASS Consortium (a group who would later generalize to become today’s Open Geospatial Consortium — the OGC). The project coordination eventually shifted to the international development team made up of governmental and academic researchers and university scientists. Reflecting this shift to a project run by the users, for the users, in 1999 GRASS GIS was released under the terms of the GNU General Public License (GPL). A detailed history of GRASS GIS can be found at https://grass.osgeo.org/history/.

Where to next?
The development on GRASS GIS continues with more energy and interest than ever. Parallel to the long-term maintenance of the GRASS 7.4 stable series, effort is well underway on the new upcoming cutting-edge 7.6 release, which will bring many new features, enhancements, and cleanups. As in the past, the GRASS GIS community is open to any contribution, be it in the form of programming, documentation, testing, and financial sponsorship. Please contact us!

About GRASS GIS

The Geographic Resources Analysis Support System (https://grass.osgeo.org/), commonly referred to as GRASS GIS, is an Open Source Geographic Information System providing powerful raster, vector and geospatial processing capabilities in a single integrated software suite. GRASS GIS includes tools for spatial modeling, visualization of raster and vector data, management and analysis of geospatial data, and the processing of satellite and aerial imagery. It also provides the capability to produce sophisticated presentation graphics and hardcopy maps. GRASS GIS has been translated into about twenty languages and supports a huge array of data formats. It can be used either as a stand-alone application or as backend for other software packages such as QGIS and R geostatistics. It is distributed freely under the terms of the GNU General Public License (GPL). GRASS GIS is a founding member of the Open Source Geospatial Foundation (OSGeo).

The GRASS Development Team, July 2018

The post Celebrating 35 years of GRASS GIS! appeared first on GFOSS Blog | GRASS GIS and OSGeo News.

Create a QGIS vector data provider in Python is now possible

 

Why python data providers?

My main reasons for having Python data provider were:

  • quick prototyping
  • web services
  • why not?

 

This topic has been floating in my head for a while since I decided to give it a second look and I finally implemented it and merged for the next 3.2 release.

 

How it’s been done

To make this possible I had to:

  • create a public API for registering the providers
  • create the Python bindings (the hard part)
  • create a sample Python vector data provider (the boring part)
  • make all the tests pass

 

First, let me say that it wasn’t like a walk in the park: the Python bindings part is always like diving into woodoo and black magic recipes before I can get it to work properly.

For the Python provider sample implementation I decided to re-implement the memory (aka: scratch layers) provider because that’s one of the simplest providers and it does not depend on any external storage or backend.

 

How to and examples

For now, the main source of information is the API and the tests:

To register your own provider (PyProvider in the snippet below) these are the basic steps:

metadata = QgsProviderMetadata(PyProvider.providerKey(), PyProvider.description(), PyProvider.createProvider)
QgsProviderRegistry.instance().registerProvider(metadata)

To create your own provider you will need at least the following components:

  • the provider class itself (subclass of QgsVectorDataProvider)
  • a feature source (subclass of QgsAbstractFeatureSource)
  • a feature iterator (subclass of QgsAbstractFeatureIterator)

Be aware that the implementation of a data provider is not easy and you will need to write a lot of code, but at least you could get some inspiration from the existing example.

 

Enjoy wirting data providers in Python and please let me know if you’ve fond this implementation useful!

&#8220;QGZ&#8221; &#8211; A new default project file format for QGIS

Last year we had the opportunity to implement ‘.QGZ’ as  anew variant of  the QGIS 3 project file format.

This is simply a zipped container for the QGS xml file. We took benefit of that container to store the auxiliary storage database into it – only if users choose that optional format though.

In classical mode, the auxiliary database is saved as a .qgd file  along the xml file.

qgd auxiliary storage file

 

qgz file format saving

When choosing the zipped container, the qgd file falls into the qgz, and this becomes totally transparent for end users. No more issues, you can’t delete it or forget to copy it when sharing project files!

The great news today is that for qgis 3.2, qgz will be the default format!

This will offers many possibilities like embbeding:

  • Resources like fonts, SVG, color ramps, and all styling informations
  • A unified container for off-line editing embedding data into a gpkg format
  • plugins, scripts, processing algorithms and modelers

If you are interested in going further this, please contact us!

Docker images for QGIS

With support from Orange we’ve created Docker images for QGIS. All the material for building and running these images is open-source and freely available on GitHub: https://github.com/Oslandia/docker-qgis.

The docker-qgis repository includes two Docker images: qgis-build and qgis-exec.

The qgis-build image includes all the libraries and tools necessary for building QGIS from source. Building QGIS requires installing a lot of dependencies, so it might not be easy depending on the Operating System you use. The qgis-build image makes it easy and repeatable. The result of a QGIS build is Debian Stretch packages that can be used for building the qgis-exec image, as explained below.

The qgis-exec image includes a runtime environment for QGIS Server. The image makes it easy to deploy and run QGIS Server. It is meant to be be generic and usable in various contexts, and in production. There are two ways to build the qgis-exec image. It can be built from the official QGIS 3 Debian packages (http://qgis.org/debian/), or from local QGIS Debian Stretch packages that were built using the qgis-build image.

We encourage you to go test and use our images! We haven’t pushed the images to the Docker Hub yet, but this is in the plans.

Feel free to contact us on GitHub or by email for any question or request!

Movement data in GIS #12: why you should be using PostGIS trajectories

In short: both writing trajectory queries as well as executing them is considerably faster using PostGIS trajectories (as LinestringM) rather than the commonly used point-based approach.

Here are a couple of examples to give you an impression of the differences.

Spoiler alert! Trajectory queries are up to 500 times faster than comparable point-based queries.

A quick look at indexing

In both cases, we have indexed the tracker id, geometry, and time columns to speed up query processing.

The trajectory table has 3 indexes

  • gist (time_range)
  • gist (track gist_geometry_ops_nd)
  • btree (tracker)

The point-based table has 4 indexes

  • gist (pt)
  • btree (trajectory_id)
  • btree (tracker)
  • btree (t)

Length

First, let’s see how to determine trajectory length for all observed moving objects (identified by a tracker id).

Using the point-based approach, we first need to ensure that the points are in the correct temporal order, create the lines, and finally sum up their length:

WITH ordered AS (
 SELECT trajectory_id, tracker, t, pt
 FROM geolife.trajectory_pt
 ORDER BY t
), tmp AS (
 SELECT trajectory_id, tracker, st_makeline(pt) traj
 FROM ordered 
 GROUP BY trajectory_id, tracker
)
SELECT tracker, round(sum(ST_Length(traj::geography)))
FROM tmp
GROUP BY tracker 
ORDER BY tracker

With trajectories, we can go right to computing lengths:

SELECT tracker, round(sum(ST_Length(track::geography)))
FROM geolife.trajectory_ext
GROUP BY tracker
ORDER BY tracker

On my test system, the trajectory query run time is 22.7 sec instead of 43.0 sec for the point-based approach:

Duration

Compared to trajectory length, duration is less complicated in the point-based approach:

WITH tmp AS (
 SELECT trajectory_id, tracker, min(t) start_time, max(t) end_time
 FROM geolife.trajectory_pt
 GROUP BY trajectory_id, tracker
)
SELECT tracker, sum(end_time - start_time)
FROM tmp
GROUP BY tracker
ORDER BY tracker

Still, the trajectory query is less complex and much faster at 31 ms instead of 6.0 sec:

SELECT tracker, sum(upper(time_range) - lower(time_range))
FROM geolife.trajectory_ext
GROUP BY tracker
ORDER BY tracker

Temporal filter

Extracting trajectories that occurred during a certain time frame is another common use case:

WITH tmp AS (
 SELECT trajectory_id, tracker, min(t) start_time, max(t) end_time
 FROM geolife.trajectory_pt
 GROUP BY trajectory_id, tracker
)
SELECT trajectory_id, tracker, start_time, end_time
FROM tmp
WHERE end_time > '2008-11-26 11:00'
AND start_time < '2008-11-26 15:00'
ORDER BY tracker

This point-based query takes 6.0 sec while the shorter trajectory query finishes in 12 ms:

SELECT id, tracker, time_range
FROM geolife.trajectory_ext
WHERE time_range && '[2008-11-26 11:00+1,2008-11-26 15:00+01]'::tstzrange

or equally fast (12 ms) by making use of the n-dimensional index:

WHERE track &&&	ST_Collect(
 ST_MakePointM(-180, -90, extract(epoch from '2008-11-26 11:00'::timestamptz)),
 ST_MakePointM(180, 90, extract(epoch from '2008-11-26 15:00'::timestamptz))
)

Spatial filter

Finally, of course, let’s have a look at spatial filters, for example, trajectories that start in a certain area:

WITH my AS ( 
 SELECT ST_Buffer(ST_SetSRID(ST_MakePoint(116.31894,39.97472),4326),0.0005) areaA
), tmp AS (
 SELECT trajectory_id, tracker, min(t) t 
 FROM geolife.trajectory_pt
 GROUP BY trajectory_id, tracker
)
SELECT distinct traj.tracker, traj.trajectory_id 
FROM tmp
JOIN geolife.trajectory_pt traj
ON tmp.trajectory_id = traj.trajectory_id AND traj.t = tmp.t
JOIN my
ON ST_Within(traj.pt, my.areaA)

This point-based query takes 6.0 sec while the shorter trajectory query finishes in 488 ms:

WITH my AS ( 
 SELECT ST_Buffer(ST_SetSRID(ST_MakePoint(116.31894, 39.97472),4326),0.0005) areaA
)
SELECT id, tracker, ST_AsText(track)
FROM geolife.trajectory_ext
JOIN my
ON areaA && track
AND ST_Within(ST_StartPoint(track), areaA)

For more generic “does this trajectory intersect another geometry”, the points can also be aggregated to a linestring on the fly but that takes 21.9 sec:

I’ll be presenting more work on PostGIS trajectories at GI_Forum in Salzburg in July. In the talk, I’ll also have a look at the custom PG-Trajectory datatype. Here’s the full open-access paper:

Graser, A. (2018) Evaluating Spatio-temporal Data Models for Trajectories in PostGIS Databases. GI_Forum ‒ Journal of Geographic Information Science, 1-2018, 16-33. DOI: 10.1553/giscience2018_01_s16.

You can find my fork of the PG-Trajectory project – including all necessary fixes – on Bitbucket.


This post is part of a series. Read more about movement data in GIS.

Welcome QGIS 3 and bye bye Madeira

Last week I’ve been in Madeira at the hackfest, like all the past events this has been an amazing happening, for those of you who have never been there, a QGIS hackfest is typically an event where QGIS developers and other pasionate contributors like documentation writers, translators etc. gather together to discuss the future of their beloved QGIS software. QGIS hackfest are informal events where meetings are scheduled freely and any topic relevant to the project can be discussed. This time we have brought to the table some interesting topics like:

  • the future of processing providers: should they be part of QGIS code or handled independently as plugins?
  • the road forward to a better bug reporting system and CI platform: move to gitlab?
  • the certification program for QGIS training courses: how (and how much) training companies should give back to the project?
  • SWOT analysis of current QGIS project: very interesting discussion about the status of the project.
  • QGIS Qt Quick modules for mobile QGIS app
Tehre were also some mentoring sessions where I presented:
  • How to set up a development environment and make your first pull request
  • How to write tests for QGIS (in both python and C++)
  At this link you can find all the video recordings of the sessions: https://github.com/qgis/QGIS/wiki/DeveloperMeetingMadeira2018   Here is a link to the Vagrant QGIS developer VM I’ve prepared for the session: https://github.com/elpaso/qgis-dev-vagrant/   I’ve got a good feedback from other devs about my sessions and I’m really happy that somebody found them useful, one of the main goals of a QGIS hackfest should really be to help other developers to ramp up quicly into the project. Other than that, I’ve also find the time to update to QGIS 3.0 some of my old plugins like GeoCoding and QuickWKT.   Thanks to Giovanni Manghi and to Madeira Government for the organizazion and thanks to all QGIS sponsors and donors!   About me: I started as a QGIS plugin author, continued as the developer of the plugin official repository at https://plugins.qgis.org and now I’m one of the top 5 QGIS core contributors. After almost 10 years that I’m in the QGIS project I’m now not only a proud member of the QGIS community but also an advocate for the open source GIS software movement.

  • Page 1 of 11 ( 212 posts )
  • >>
  • gis

Back to Top

Sponsors