Tag: qgis

The goal of the new API is twofold:

1. provide a unified way to store and retrieve data provider connections in the QGIS settings
2. provide an abstract set of methods to perform most common operation on DB data sources (e.g. browse tables, drop/create a table/schema, run arbitrary SQL commands etc.)

The new API is documented in https://qgis.org/api/classQgsAbstractProviderConnection.html and it provides a few specializations for DB connections (https://qgis.org/api/classQgsAbstractDatabaseProviderConnection.html) and an initial PR implementation for web service-based connections (https://github.com/qgis/QGIS/pull/33045).

While the whole of the desired refactoring work was too big for a single grant request, the first work package has been completed and the following data providers have been partially or totally refactored to make use of the new connections API:

• postgres
• geopackage (OGR)
• spatialite

The new API was also used to implement the automatic loading of layer dependencies (not part of the grant program).

For developers interested in working with the new API, a set Python tests are available to show how to use the methods:  https://github.com/qgis/QGIS/blob/master/tests/src/python/test_qgsproviderconnection_ogr_gpkg.py (see also the postgres and spatialite companion tests).

There is still a large amount of work to be done in order to complete all the desired refactoring and to remove all the Python and C++ code that will be ultimately be made redundant. In particular, future work should be undertaken to:

• port all remaining data providers to the new API
• refactor and eliminate the remaining DB-manager connectors to make use of the abstract API
• eliminate duplicate and untested code inside the Processing framework for working with Postgres databases and port the code to the new, stable, well-tested API
• refactor and eliminate the remaining QGIS browser data items to make use of the abstract API 

For further information, the following paragraphs (taken from the original grant proposal) will provide full details about the background of this work.

Background/motivation

• DB-Manager is an important part of the QGIS interface, which allows browsing/previews of different DB-based data sources, complex queries, management of layers, import-export etc., DB creation and deletion etc.
• After the QGIS 3.0 release, improvements within the core browser widgets implemented in C++ have resulted in a (constantly growing) degree of overlapping functionality between the browser and db manager.
• After QGIS 3 API improvements concerning layer import and export functionality, there are many duplicated implementations between browser and db manager – some functionality is better in browser, some functionality is better in db manager. Users are forced to choose between two competing semi-complete alternatives, instead of having one, complete, well integrated solution.
• There are no unit tests for DB-Manager and this leads to frequent regressions, which (aside from being frustrating for users) consume a substantial part of our development time and budget during the bugfixing programs. Furthermore the nature of large Python codebases like db manager makes it very easy to accidentally break functionality with no warning or errors during development.

Proposed solution

We propose to start refactoring the DB-manager plugin functionality into core C++ implementation, reusing existing core API and replacing redundant duplicate functionality.

The clear advantages are:

• no duplicate functionality, so it’s easier for users to understand and use
• more usage of well-tested and well-maintained core C++ API
• testability and immediate feedback on API breaks (an advantage of C++ is that the application wonâ€t even build if an API is changed or accidentally misused)
• better performance
• the ability to expose database management functionality via stable PyQGIS API, allowing other plugins and scripts to utilise this functionality. In future, Processing algorithms may also be developed which would take advantage of these functions (e.g. “create schema”, “drop table”, “vacuum table” algorithms)
• DB management functionality would be available within the main QGIS window (from the Browser panel), instead of as a separate dialog.

Grant proposal package

The above mentioned work is too large to be completed within a single grant, so what we propose here is to start the refactoring needed in order to have a core stable C++ API that can be used by the application and the plugins and that will be available to fully move DB manager to C++ API in the future avoiding duplication of code and functionality.

• create an interface for databases that expose the required functions to a coherent API
• add missing tests and documentation for the a.m. API
• porting some basic functions from db manager to the new api:
  • create table (with native field types support)
  • create schema
  • delete table
  • Rename table

The API will be exposed through the browser and it will be used by the DB manager instead of the Python implementation that is currently used.

PT | EN

As I was preparing a QGIS Project to read a database structured according to the new rules and technical specifications for the Portuguese Cartography, I started to configure the editing forms for several layers, so that:

1. Make some fields read-only, like for example an identifier field.
2. Configure widgets better suited for each field, to help the user and avoid errors. For example, date-time files with a pop-up calendar, and value lists with dropdown selectors.

Basically, I wanted something like this:

Let me say that, in PostGIS layers, QGIS does a great job in figuring out the best widget to use for each field, as well as the constraints to apply. Which is a great help. Nevertheless, some need some extra configuration.

If I had only a few layers and fields, I would have done them all by hand, but after the 5th layer my personal mantra started to chime in:

“If you are using a computer to perform a repetitive manual task, you are doing it wrong!”

So, I began to think how could I configure the layers and fields more systematically. After some research and trial and error, I came up with the following PyQGIS functions.

Make a field Read-only

The identifier field (“identificador”) is automatically generated by the database. Therefore, the user shouldn’t edit it. So I had better make it read only

To make all the identifier fields read-only, I used the following code.

def field_readonly(layer, fieldname, option = True):
    fields = layer.fields()
    field_idx = fields.indexOf(fieldname)
    if field_idx >= 0:
        form_config = layer.editFormConfig()
        form_config.setReadOnly(field_idx, option)
        layer.setEditFormConfig(form_config)

# Example for the field "identificador"

project = QgsProject.instance()
layers = project.mapLayers() 

for layer in layers.values():
    field_readonly(layer,'identificador')

Set fields with DateTime widget

The date fields are configured automatically, but the default widget setting only outputs the date, and not date-time, as the rules required.

I started by setting a field in a layer exactly how I wanted, then I tried to figure out how those setting were saved in PyQGIS using the Python console:

>>>layer = iface.mapCanvas().currentLayer()
>>>layer.fields().indexOf('inicio_objeto')
1
>>>field = layer.fields()[1]
>>>field.editorWidgetSetup().type()
'DateTime'
>>>field.editorWidgetSetup().config()
{'allow_null': True, 'calendar_popup': True, 'display_format': 'yyyy-MM-dd HH:mm:ss', 'field_format': 'yyyy-MM-dd HH:mm:ss', 'field_iso_format': False}

Knowing this, I was able to create a function that allows configuring a field in a layer using the exact same settings, and apply it to all layers.

def field_to_datetime(layer, fieldname):
    config = {'allow_null': True,
              'calendar_popup': True,
              'display_format': 'yyyy-MM-dd HH:mm:ss',
              'field_format': 'yyyy-MM-dd HH:mm:ss',
              'field_iso_format': False}
    type = 'Datetime'
    fields = layer.fields()
    field_idx = fields.indexOf(fieldname)
    if field_idx >= 0:
        widget_setup = QgsEditorWidgetSetup(type,config)
        layer.setEditorWidgetSetup(field_idx, widget_setup)

# Example applied to "inicio_objeto" e "fim_objeto"

for layer in layers.values():
    field_to_datetime(layer,'inicio_objeto')
    field_to_datetime(layer,'fim_objeto')

Setting a field with the Value Relation widget

In the data model, many tables have fields that only allow a limited number of values. Those values are referenced to other tables, the Foreign keys.

In these cases, it’s quite helpful to use a Value Relation widget. To configure fields with it in a programmatic way, it’s quite similar to the earlier example, where we first neet to set an example and see how it’s stored, but in this case, each field has a slightly different settings

Luckily, whoever designed the data model, did a favor to us all by giving the same name to the fields and the related tables, making it possible to automatically adapt the settings for each case.

The function stars by gathering all fields in which the name starts with ‘valor_’ (value). Then, iterating over those fields, adapts the configuration to use the reference layer that as the same name as the field.

def field_to_value_relation(layer):
    fields = layer.fields()
    pattern = re.compile(r'^valor_')
    fields_valor = [field for field in fields if pattern.match(field.name())]
    if len(fields_valor) > 0:
        config = {'AllowMulti': False,
                  'AllowNull': True,
                  'FilterExpression': '',
                  'Key': 'identificador',
                  'Layer': '',
                  'NofColumns': 1,
                  'OrderByValue': False,
                  'UseCompleter': False,
                   'Value': 'descricao'}
        for field in fields_valor:
            field_idx = fields.indexOf(field.name())
            if field_idx >= 0:
                print(field)
                try:
                    target_layer = QgsProject.instance().mapLayersByName(field.name())[0]
                    config['Layer'] = target_layer.id()
                    widget_setup = QgsEditorWidgetSetup('ValueRelation',config)
                    layer.setEditorWidgetSetup(field_idx, widget_setup)
                except:
                    pass
            else:
                return False
    else:
        return False
    return True
    
# Correr função em todas as camadas
for layer in layers.values():
    field_to_value_relation(layer)

Conclusion

In a relatively quick way, I was able to set all the project’s layers with the widgets I needed.

This seems to me like the tip of the iceberg. If one has the need, with some search and patience, other configurations can be changed using PyQGIS. Therefore, think twice before embarking in configuring a big project, layer by layer, field by fields.

QGIS recipes have been available on Conda for a while, but now, that they work for the three main operating systems, getting QGIS from Conda is s starting to become a reliable alternative to other QGIS distributions. Anyway, let’s rewind a bit…

What is Conda?

Conda is an open source package management system and environment management system that runs on Windows, macOS and Linux. Conda quickly installs, runs and updates packages and their dependencies. Conda easily creates, saves, loads and switches between environments on your local computer. It was created for Python programs, but it can package and distribute software for any language.

Why is that of any relevance?

Conda provides a similar way to build, package and install QGIS (or any other software) in Linux, Windows, and Mac.

As a user, it’s the installation part that I enjoy the most. I am a Linux user, and one of the significant limitations is not having an easy way to install more than one version of QGIS on my machine (for example the latest stable version and the Long Term Release). I was able to work around that limitation by compiling QGIS myself, but with Conda, I can install as many versions as I want in a very convenient way.

The following paragraphs explain how to install QGIS using Conda. The instructions and Conda commands should be quite similar for all the operating systems.

Anaconda or miniconda?

First thing you need to do is to install the Conda packaging system. Two distributions install Conda: Anaconda and Miniconda.

TL;DR Anaconda is big (3Gb?) and installs the packaging system and a lot of useful tools, python packages, libraries, etc… . Miniconda is much smaller and installs just the packaging system, which is the bare minimum that you need to work with Conda and will allow you to selectively install the tools and packages you need. I prefer the later.

For more information, check this stack exchange answer on anaconda vs miniconda.

Download anaconda or miniconda installers for your system and follow the instructions to install it.

Windows installer is an executable, you should run it as administrator. The OSX and Linux installers are bash scripts, which means that, once downloaded, you need to run something like this to install:

bash Miniconda3-latest-Linux-x86_64.sh

Installing QGIS

Notice that the Conda tools are used in a command line terminal. Besides, on Windows, you need to use the command prompt that is installed with miniconda.

Using environments

Conda works with environments, which are similar to Python virtual environments but not limited only to python. Basically, it allows isolating different installations or setups without interfering with the rest of the system. I recommend that you always use environments. If, like me, you want to have more that one version of QGIS installed, then the use of environments is mandatory.

Creating an environment is as easy as entering the following command on the terminal:

conda create --name <name_of_the_environment>

For example,

conda create --name qgis_stable

You can choose the version of python to use in your environment by adding the option python=<version>. Currently versions of QGIS run on python 3.6, 3.7, 3.8 and 3.9.

conda create –name qgis_stable python=3.7

To use an environment, you need to activate it.

conda activate qgis_stable

Your terminal prompt will show you the active environment.

(qgis_stable) aneto@oryx:~/miniconda3$

To deactivate the current environment, you run

conda deactivate

Installing packages

Installing packages using Conda is as simples as:

conda install <package_name>

Because conda packages can be stored in different channels, and because the default channels (from the anaconda service) do not contain QGIS, we need to specify the channel we want to get the package from. conda-forge is a community-driven repository of conda recipes and includes updated QGIS packages.

conda install qgis --channel conda-forge

Conda will download the latest available version of QGIS and all its dependencies installing it on the active environment.

Note: Because conda always try to install the latest version, if you want to use the QGIS LTR version, you must specify the QGIS version.

conda install qgis=3.10.12 --channel conda-forge

Uninstalling packages

Uninstalling QGIS is also easy. The quickest option is to delete the entire environment where QGIS was installed. Make sure you deactivate it first.

conda deactivate
conda env remove --name qgis_stable

Another option is to remove QGIS package manually. This is useful if you have other packages installed that you want to keep.

conda activate qgis_stable
conda remove qgis -c conda-forge

This only removes the QGIS package and will leave all other packages that were installed with it. Note that you need to specify the conda-forge channel. Otherwise, Conda will try to update some packages from the default channels during the removal process, and things may get messy.

Running QGIS

To run QGIS, in the terminal, activate the environment (if not activated already) and run the qgis command

conda activate qgis_stable
qgis

Updating QGIS

To update QGIS to the most recent version, you need to run the following command with the respective environment active

conda update qgis -c conda-forge

To update a patch release for the QGIS LTR version you run the install command again with the new version:

conda install qgis=3.10.13 -c conda-forge

Some notes and caveats

Please be aware that QGIS packages on Conda do not provide the same level of user experience as the official Linux, Windows, and Mac installer from the QGIS.org distribution. For example, there are no desktop icons or file association, it does not include GRASS and SAGA, etc …

On the other hand, QGIS installations on Conda it will share user configurations, installed plugins, with any other QGIS installations on your system.

Let’s get straight to the point

It’s official, QField for QGIS 1.0 is out!

Get it while it’s hot on the Playstore ( qfield.org/get) or on GitHub

We are incredibly pleased and proud of just having released such a jewel and are convinced that, thanks to all its features and conscious design choices, QField will make your field digitizing work much more efficient and pleasant.

Packed with loads of useful features like online and offline features digitizing, geometry and attributes editing, attribute search, powerful forms, theme switching, GPS support, camera integration and much more, QField is the powerful tool for those who need to edit on the go and would like to avoid standing in the swamp with a laptop or paper charts.

Let’s see what makes QField probably[su_tooltip style=“bootstrap” rounded=“yes” position=“top” content=“We might be biased, but we do believe it”]*[/su_tooltip] the best mobile GIS in the world.

Work efficiently

QField focuses on efficiently getting GIS field work done and combines a minimal design with sophisticated technology to get data from the field to the office in a comfortable and easy way.

Fast and reactive

Thanks to the underlying QGIS engine and a lot of optimizations, QField is powerful and snappy. Even with complex projects, QField is a joy to work with.

Easy handling

Conscious design choices and a continuous focus on a minimal user interface drive QField’s development. This allows us to deliver a product wich is uncluttered and extremly user-friendly

Quickly digitise

Allowing a seamless digitizing experience is a paramount goal of QField. Thanks to a cleverly designed adaptive user interface and specific features like real-time attribute checks and snapping support, QField allows its users to be extremely time efficient.

Unmatched feature set

To be the best, you need to be clever but also skillful.

QField’s efficiency is matched only by its featureset that allows its users to make the most out of their fieldwork time.

Powerful cartography combined with full text search

The beauty of GIS is that maps are dynamic. Layers can individually be shown and hidden and information can be presented more or less prominently based on the task at hand. QField supports the endless styling possibilities offered by QGIS and thanks to a well placed theme switcher you can change the looks of the entire project with a single click. For even more customizability, QField allows hiding and showing layers by simply long-pressing on the layer name.

Furthermore, QField boasts a fully configurable attribute text search that will allow you to geolocate and edit that exact object you were looking for.

Geometry editing

Editing Geometries on the field is probably the most complex task an operator has to deal with. QField simplifies this process through an adaptive toolbar that appears only when necessary, snapping support and a crosshair digitizer.

Thanks to these enhancements, QField allows reducing the error rate significantly.

Support for high precision GNSS

Simple internal GPS accuracy might be enough for basic projects but cadastral surveying and other high accuracy digitizations have much higher requirements. QFields natively listens to the Android location services so it can take advantage of the best location provided by external devices.

Generate PDF

Thanks to QField’s native support for generating PDFs based on QGIS’s print layouts, your on the fly daily report map is just one click away.

Intuitive project chooser

When dealing with multiple projects, quickly being able to switch between them is key. QField comes with a beautiful file selector with favorite directories (long press on a folder to add it to the favorites and long press on the favorites list to remove it) and an automatic list of the last three opened projects that will save you heaps of time while looking for your projects.

Your data - Your decisions

QField does not impose any constraint on the data model, it is your data and you decide what they should look like and what values are acceptable. QField can enforce constraints for you and you can choose among various type of widgets to represent your data. QGIS will preconfigure some field types automatically, all you’ll have then to do is tweak the settings if you want and your project is ready for mobile prime time. Our documentation has all the information you need.

Extends your Geo Data Infrastructure seamlessly

QField uses QGIS to set up maps and forms so it automatically supports a wide variety of data formats. Thanks to this, you can comfortably prepare your project once and then deploy it everywhere. And since QGIS also has a server component, your project can be served on a WebGIS with the very same beautiful looks.

In fact you can see this exact infrastructure up and running under demo.qfield.org and with the “online_survey.qgs” project included in the QField demo projects.

Synchronize with WiFi, Cable or Network

You can synchronize your project and data (in case you are not using a centralized online database) using various methods thanks to our QFieldSync plugin.

Future cloud integration

In the near future we will add a cloud synchronization functionality, so that you will be able to seamlessly manage your project online and have them automatically deployed to your devices.

Installing and contributing

You can easily install QField using the Playstore ( qfield.org/get), find out more on the documentation site ( qfield.org), watch some demo videos on our channel ( qfield.org/demo) and report problems to our issues tracking system ( qfield.org/issues). Please note that the Playstore update can take some hours to roll out and if you had installed a version directly from GitHub, you might have to uninstall it to get the latest Playstore update.

QField, like QGIS, is an open source project. Everyone is welcome to contribute to making the product even better – whether it is with financial support, translation, documentation work, enthusiastic programming or visionary ideas.

We would like to thank our fantastic community for all the great translations, documentations, bug reports and general feedback they gave us. Thanks to all this, we were able to fix plenty of bugs, address performance issues and even add some super cool new features.

Development and deployment services

As masterminds behind QField and core contributor to QGIS, we are the perfect partner for your project. If you want to help us build a better QField or QGIS, or if you need any services related to the whole QGIS stack, don’t hesitate to contact us.

OPENGIS.ch

OPENGIS.ch helps you setting up your spatial data infrastructure based on seamlessly integrated desktop, web, and mobile components.
We support your team in planning, developing, deploying and running your infrastructure. Thanks to several senior geodata infrastructure experts, QGIS core developers and the makers of the mobile data acquisition solution QField, OPENGIS.ch has all it takes to make your project a success. OPENGIS.ch is known for its commitment to high-quality products and its continuous efforts to improve the open source ecosystem.

\* We might be biased, but we do believe it

We are really happy to announce the release a new great milestone in QField’s history, QField 1.0 Release Candidate 3! (Yes, you might have got a glimpse of the broken RC2 if you where very attentive)

Thanks to the great feedback we received since releasing RC1 we were able to fix plenty of issues and add some more goodies.

We would like to invite everybody to install this Release Candidate and help us test it as much as possible so that we can iron out as many bugs as possible before the final release of QField 1.0.

List of fixes since RC1:
• fixed bad synchronization / geopackage files not written) (PR #455)
• fix glitches in portrait mode (PR #423 and #439)
• fix highlighting of points (search and feature selection) (PR #443)
• fix GPS info window overlapping search icon (PR #438)
• redesign of scale bar (PR #438)
• fix crash in feature form (with invalid relations) (PR #440)
• fix date/time field editing (PR #421 and #458)
• fix project not loading the correct map theme (fix #459)
• fix QGS or QGZ does not exist (PR #453)

Unfortunately, due to necessary updates in the SDK we target, we had to drop support for Android 4.4. The minimum Android requirement as of this RC is Android 5.0 (SDK version 21).

In case playstore does not suggest an update to QField Lucendro 0.11.90, the last working version for Android 4.4, we suggest all Android 4.4 users to uninstall QField 1.0 RC 1 (which was broken on android 4.4) and reinstall QField from the store. This way you should get If you don’t use play store, you can find all QField releases under https://qfield.org/releases

You can easily install QField using the playstore ( https://qfield.org/get), find out more on the documentation site ( https://qfield.org) and report problems to our issues tracking system ( https://qfield.org/issues)

QField, like QGIS, is an open source project. Everyone is welcome to contribute to make the product even better – whether it is with financial support, enthusiastic programming, translation and documentation work or visionary ideas.

If you want to help us build a better QField or QGIS, or need any services related to the whole QGIS stack don’t hesitate to contact us.

Last week the first official “FOSS4G/SOTM Oceania” conference was held at Melbourne University. This was a fantastic event, and there’s simply no way I can extend sufficient thanks to all the organisers and volunteers who put this event together. They did a brilliant job, and their efforts are even more impressive considering it was the inaugural event!

Upfront — this is not a recap of the conference (I’m sure someone else is working on a much more detailed write up of the event!), just some musings I’ve had following my experiences assisting Nathan Woodrow deliver an introductory Python for QGIS workshop he put together for the conference. In short, we both found that delivering this workshop to a group of PyQGIS newcomers was a great way for us to identify “pain points” in the PyQGIS API and areas where we need to improve. The good news is that as a direct result of the experiences during this workshop the API has been improved and streamlined! Let’s explore how:

Part of Nathan’s workshop (notes are available here) focused on a hands-on example of creating a custom QGIS “Processing” script. I’ve found that preparing workshops is guaranteed to expose a bunch of rare and tricky software bugs, and this was no exception! Unfortunately the workshop was scheduled just before the QGIS 3.4.2 patch release which fixed these bugs, but at least they’re fixed now and we can move on…

The bulk of Nathan’s example algorithm is contained within the following block (where “distance” is the length of line segments we want to chop our features up into):

for input_feature in enumerate(features):
    geom = feature.geometry().constGet()
    if isinstance(geom, QgsLineString):
        continue
    first_part = geom.geometryN(0)
    start = 0
    end = distance
    length = first_part.length()

    while start < length:
        new_geom = first_part.curveSubstring(start,end)

        output_feature = input_feature
        output_feature.setGeometry(QgsGeometry(new_geom))
        sink.addFeature(output_feature)

        start += distance
        end += distance

There’s a lot here, but really the guts of this algorithm breaks down to one line:

new_geom = first_part.curveSubstring(start,end)

Basically, a new geometry is created for each trimmed section in the output layer by calling the “curveSubstring” method on the input geometry and passing it a start and end distance along the input line. This returns the portion of that input LineString (or CircularString, or CompoundCurve) between those distances. The PyQGIS API nicely hides the details here – you can safely call this one method and be confident that regardless of the input geometry type the result will be correct.

Unfortunately, while calling the “curveSubstring” method is elegant, all the code surrounding this call is not so elegant. As a (mostly) full-time QGIS developer myself, I tend to look over oddities in the API. It’s easy to justify ugly API as just “how it’s always been”, and over time it’s natural to develop a type of blind spot to these issues.

Let’s start with the first ugly part of this code:

geom = input_feature.geometry().constGet()
if isinstance(geom, QgsLineString):
    continue
first_part = geom.geometryN(0)
# chop first_part into sections of desired length
...

This is rather… confusing… logic to follow. Here the script is fetching the geometry of the input feature, checking if it’s a LineString, and if it IS, then it skips that feature and continues to the next. Wait… what? It’s skipping features with LineString geometries?

Well, yes. The algorithm was written specifically for one workshop, which was using a MultiLineString layer as the demo layer. The script takes a huge shortcut here and says “if the input feature isn’t a MultiLineString, ignore it — we only know how to deal with multi-part geometries”. Immediately following this logic there’s a call to geometryN( 0 ), which returns just the first part of the MultiLineString geometry.

There’s two issues here — one is that the script just plain won’t work for LineString inputs, and the second is that it ignores everything BUT the first part in the geometry. While it would be possible to fix the script and add a check for the input geometry type, put in logic to loop over all the parts of a multi-part input, etc, that’s instantly going to add a LOT of complexity or duplicate code here.

Fortunately, this was the perfect excuse to improve the PyQGIS API itself so that this kind of operation is simpler in future! Nathan and I had a debrief/brainstorm after the workshop, and as a result a new “parts iterator” has been implemented and merged to QGIS master. It’ll be available from version 3.6 on. Using the new iterator, we can simplify the script:

geom = input_feature.geometry()
for part in geom.parts():
    # chop part into sections of desired length
    ...

Win! This is simultaneously more readable, more Pythonic, and automatically works for both LineString and MultiLineString inputs (and in the case of MultiLineStrings, we now correctly handle all parts).

Here’s another pain-point. Looking at the block:

new_geom = part.curveSubstring(start,end)
output_feature = input_feature
output_feature.setGeometry(QgsGeometry(new_geom))

At first glance this looks reasonable – we use curveSubstring to get the portion of the curve, then make a copy of the input_feature as output_feature (this ensures that the features output by the algorithm maintain all the attributes from the input features), and finally set the geometry of the output_feature to be the newly calculated curve portion. The ugliness here comes in this line:

output_feature.setGeometry(QgsGeometry(new_geom))

What’s that extra QgsGeometry(…) call doing here? Without getting too sidetracked into the QGIS geometry API internals, QgsFeature.setGeometry requires a QgsGeometry argument, not the QgsAbstractGeometry subclass which is returned by curveSubstring.

This is a prime example of a “paper-cut” style issue in the PyQGIS API. Experienced developers know and understand the reasons behind this, but for newcomers to PyQGIS, it’s an obscure complexity. Fortunately the solution here was simple — and after the workshop Nathan and I added a new overload to QgsFeature.setGeometry which accepts a QgsAbstractGeometry argument. So in QGIS 3.6 this line can be simplified to:

output_feature.setGeometry(new_geom)

Or, if you wanted to make things more concise, you could put the curveSubstring call directly in here:

output_feature = input_feature
output_feature.setGeometry(part.curveSubstring(start,end))

Let’s have a look at the simplified script for QGIS 3.6:

for input_feature in enumerate(features):
    geom = feature.geometry()
    for part in geom.parts():
        start = 0
        end = distance
        length = part.length()

        while start < length:
            output_feature = input_feature
            output_feature.setGeometry(part.curveSubstring(start,end))
            sink.addFeature(output_feature)

            start += distance
            end += distance

This is MUCH nicer, and will be much easier to explain in the next workshop! The good news is that Nathan has more niceness on the way which will further improve the process of writing QGIS Processing script algorithms. You can see some early prototypes of this work here:

I couldn’t be at the community day but managed to knock out some of the new API on the plane on the way home. **API subject to change #FOSS4G_SotM_Oceania

cc @nyalldawson @underdarkGIS this will create a alg under the hood and check for double inputs, etc pic.twitter.com/49VpyuukGU

— Nathan Woodrow (@madmanwoo) November 23, 2018

So there we go. The process of writing and delivering a workshop helps to look past “API blind spots” and identify the ugly points and traps for those new to the API. As a direct result of this FOSS4G/SOTM Oceania 2018 Workshop, the QGIS 3.6 PyQGIS API will be easier to use, more readable, and less buggy! That’s a win all round!

The heart of the effect involves the very clever idea of using the topographic layer as a subtle opacity mask to simulate mist, fog and atmospheric haze. Have a look at the image below taken on March 17th, 2005 by NASA’s Terra satellite. This is the industrialised Po valley of northern Italy, surrounded by the Alps and Apennine Mountains that rise above the valley’s hazy pollution. The haze adds a sense of depth to the surrounding hills and mountains. It’s not uncommon to see fog and pollution in satellite imagery that gives way to the clear air in high mountains e.g. northern India and Nepal, China, Pakistan and India. Creating a similar mist effect in QGIS is actually quite simple.

First download topography for the Alps and Po region (a 68.55 Mb GeoTiff file derived from freely available EU-DEM data I resampled from 25 to 100m resolution). Next, make sure you have the plugin QuickMapServics (QMS) installed (menu Plugins – Manage and Install Plugins). This great plugin provides access to over 1000 basemaps.

Load the GeoTiff file into QGIS (Raster – Load) and rename the layer Hillshade. Right click the layer to open the Layer Properties window. In the Symbology panel, next to Render Type, choose Hillshade. Change the altitude to 35 degrees, Azimuth to 300 degrees and Z Factor of 1.5 (illuminating the landscape from the top left). Finally, change the Blending mode to Multiply. Click OK to close the dialogue.

To add the basemap layer, Esri World Imagery (Clarity), type “ESRI clarity” in the QMS search bar to find and add the basemap; Go to View – Panels and activate the QMS search bar if it isn’t initially visible. Make sure it’s the bottommost layer.

Oh, that’s a bit disappointing, we only increased the relief little a bit. It’s missing the vitally important mist layer.

To create mist, right click the Hillshade layer and choose Duplicate. Rename the new layer Mist and make sure it’s above the Hillshade layer. Now open the Layer Properties window of the layer, we’re going edit it’s attributes to make it look like mist.

Change the Render type to Singleband Pseudocolor and use 0 and 3000 for the min and max values (limiting maximum latitude of the mist to 3000 meters). Then open the colour ramp window by clicking on the Color ramp and enter these values:

• Left Gradient – HSV 215 15 50 and 75% transparency
• Right Gradient – HSV 215 15 50 and 0% transparency

Close the Color Ramp dialogue. In the Layer Properties window, and this is very important, change the Blending mode to Lighten. Click OK to close the Layer Properties window.

Wow, we have mist!

(View Full Size)

The mist effect looks great. It certainly adds a lot of realism to the topographic map, it now looks quite like NASA’s images. This is just a quick and basic map so there’s lots of scope to improve the effect. Play around with the colour of the mist layer and its opacity, or even brighten the Hillshade layer underneath. See what effects these changes have.

Here’s another example below. In this example I duplicated the hillshade layer and set the second hillshade layer to Multidirectional Hillshading (yes, QGIS 3.2 has Multidirectional Hillshading). I then adjusted the transparency of both hillshade layers so they blended together nicely. I then replaced the basemap with another duplicated topography layer that I coloured using the gradient sd-a (by Jim Mossman, 2005) using the cpt-city plugin. And lastly, I doubled the opacity of the mist layer turning it into a milky fog. I think it looks great!

(View Full Size)

What next? Well, there’s lots of possibilities. Perhaps download Martian topography and add mist to the bottom of Valles Marineris?

References:

Eduard Imhof – Biography

TV documentary about Eduard Imhof

The Map as an Artistic Territory: Relief Shading Works and Studies by Eduard Imhof

Haze in northern Italy – NASA Terra Satellite

What is OpenCL?

From https://en.wikipedia.org/wiki/OpenCL:

OpenCL (Open Computing Language) is a framework for writing programs that execute across heterogeneous platforms consisting of central processing units (CPUs), graphics processing units (GPUs), digital signal processors (DSPs), field-programmable gate arrays (FPGAs) and other processors or hardware accelerators. OpenCL specifies programming languages (based on C99 and C++11) for programming these devices and application programming interfaces (APIs) to control the platform and execute programs on the compute devices. OpenCL provides a standard interface for parallel computing using task- and data-based parallelism.

Basically, you write a program and you execute it on a GPU (or, less frequently, on a CPU or on a DSP) taking advantage of the huge parallel programming capabilities of the modern graphic cards.

Depending on many different factors, the speed gain can vary to a great extent, but it is typically around one order of magnitude.

How QGIS benefits from OpenCL?

The work I’ve done consisted in integrating OpenCL support into QGIS and writing all the utilities to load, build and run OpenCL programs.

For now, I’ve ported the following QGIS core algorithms, all of them are availabe in processing:

• slope
• aspect
• hillshade
• ruggedness

Since the framework to support OpenCL is now in place, I think that more algorithms will be ported over the time.

During this development, even if was not in scope, the hillshade renderer has been optimized for speed and it can also benefit of OpenCL acceleration.

How to activate OpenCL support

OpenCL support is optional and opt-in, to use it, you need to activate it into the QGIS options dialog like shown in the screenshot below:

How much performance gain can I expect?

Well, YMMV, but here are some figures for a big DEM raster, low values mean faster execution.

GDAL means CPU execution using the GDAL processing algorithm.

How to install the OpenCL drivers?

Of course it depends on your specific hardware and on your O.S., AMD, NVidia and Intel have different distributions channels, in general the driver for your graphic card will also provide the OpenCL driver, if your GPU is compatible, if OpenCL is not available on your current machine, try to Google for OpenCL, your O.S. and graphic card.

If there is no OpenCL support for your graphic card, you might try to install a driver for your GPU (Intel for example provides them) and you will probably have a decent acceleration even if not as much as you can get on a real graphic card.

This fact worths some more explanation: you might ask your self why running and algorithm directly on the CPU and running it on the same CPU but using OpenCL would make any difference and the reason why it is generally faster by using OpenCL is that OpenCL will run the algorithm in parallel on all cores of your CPU, while a normal application (and QGIS does not make an exception here) will use a single core.

How to build QGIS with OpenCL support on Ubuntu

Just a quick note: you’ll need to install the OpenCL headers and the ICD library:

Credits

I started this work as a proof of concept in my spare time (that it is not much, lately) and when I realized that it was promising, I submitted a QGIS grant proposal in order to allocate some working time to port more algorithms, write tests and polish the implementation.

This work would not be possible without all the generous sponsors and donors that feed the QGIS grant program year after year, many thanks to the QGIS community for this amazing support!

Jürgen Fischer was as usual very helpful and took care of the windows builds, now available in OSGeo4W packages.

Nyall Dawson helped with the code review and with testing the implementation on different cards and machines.

Matthias Kuhn reviewed the code.

Even Rouault pointed me to some highly efficient GDAL algorithm optimizations that I’ve been able to integrate in QGIS.

Why python data providers?

My main reasons for having Python data provider were:

• quick prototyping
• web services
• why not?

This topic has been floating in my head for a while since I decided to give it a second look and I finally implemented it and merged for the next 3.2 release.

How it’s been done

To make this possible I had to:

• create a public API for registering the providers
• create the Python bindings (the hard part)
• create a sample Python vector data provider (the boring part)
• make all the tests pass

First, let me say that it wasn’t like a walk in the park: the Python bindings part is always like diving into woodoo and black magic recipes before I can get it to work properly.

For the Python provider sample implementation I decided to re-implement the memory (aka: scratch layers) provider because that’s one of the simplest providers and it does not depend on any external storage or backend.

How to and examples

For now, the main source of information is the API and the tests:

• • https://github.com/qgis/QGIS/blob/master/src/core/qgsproviderregistry.h#L188
  • https://github.com/qgis/QGIS/blob/master/tests/src/python/provider_python.py
  • https://github.com/qgis/QGIS/blob/master/tests/src/python/test_provider_python.py

To register your own provider (PyProvider in the snippet below) these are the basic steps:

metadata = QgsProviderMetadata(PyProvider.providerKey(), PyProvider.description(), PyProvider.createProvider)
QgsProviderRegistry.instance().registerProvider(metadata)

To create your own provider you will need at least the following components:

• the provider class itself (subclass of QgsVectorDataProvider)
• a feature source (subclass of QgsAbstractFeatureSource)
• a feature iterator (subclass of QgsAbstractFeatureIterator)

Be aware that the implementation of a data provider is not easy and you will need to write a lot of code, but at least you could get some inspiration from the existing example.

Enjoy wirting data providers in Python and please let me know if you’ve fond this implementation useful!

This article describes a new algorithm for the processing modeler called feature filter algorithm. If you are already familiar with ETL concepts and the graphical modeler, you can directly jump to the section the feature filter algorithm.

Building workflows for repetitive tasks

When building workflows for simple or complex geodata infrastructures, one of the most common tasks one encounters is to extract some of the features and copy them to another destination. Sometimes they need to be modified and a few attributes calculated or deleted, maybe even the geometry needs to be adjusted or in some fancy situations one even wants to generate a couple of objects from one input object. This process is often called ETL (Extract, Transform, Load) and it is something that is worth mastering as a GIS expert. Let’s imagine a situation where we sent a field worker out to collect information about public infrastructure, equipped with a brand-new tablet and the latest and greatest version of QField. To make his task super easy, we prepare one single layer for him with an attribute type which can be set to Bus Station, Car Parking or Train Station. Now back in the office we want to integrate this back into our spatially enabled database which has been designed with 3 target tables.

Easy enough to go to QGIS and select those features by type one after the other and use a bit of copy-paste. And maybe fiddling a bit with the attributes. But hey, after all we are a bit lazy and on the one hand like to have an ice cream later on that afternoon and on the other hand like to avoid errors. Those who are lucky enough to know SQL and have full access to the database are well enough equipped to do the job.

Short introduction to the graphical modeler

For those who just want to quickly do this job visually within QGIS, there is a tool called modeler in the processing plugin. With the help of this tool it is straightforward for everyone to automate processes. To get started with the modeler, simply enable the processing plugin and click on Processing > Graphical Modeler. Within the modeler, there are Inputs and Algorithms available. Inputs are there to define variables, algorithms on the other hand transform those variables. In its most simple form, there is one vector feature source (a layer) as input and one algorithm, for example a fixed distance buffer which in turn has one output layer with all buffered features. Such a model can be saved and reused. To run a model directly from the modeler, click the play button on top. Once saved, it appears in the processing toolbox. Every time a model is run, the input layer can be handed to the model. Or it can even run in batch mode on a list of layers or files. With this in place, the job of doing the buffer can now be run on 200 input layers without any manual interaction. Simple as that. Pro tip: processing models do not have to be complex. They can also be used to preconfigure single algorithms so when an algorithm is run, the parameters which you never change are predefined already. For example you can add a Simplify geometries to 1 meter algorithm which only takes a layer as parameter and has the 1 meter tolerance built-in.

The feature filter algorithm

Now back to the job of splitting the infrastructure layer into 3 different layers. To do this job visually and easily within QGIS, there is now a new algorithm available in QGIS 3.2. It is called Feature Filter and available in the processing modeler. To make use of it, we open the processing modeler and first add a new Vector Features input and name it Infrastructure. Since we know in this project we will always deal with points, we can make already specify that in this first dialog.

Let’s now add a Feature Filter algorithm and use the following configuration: The Infrastructure layer is set as input, and we define three outputs for Train Stations, Bus Stations and Car Parking. All layers will be final outputs on which no further transformations will be applied within this model and they will be directly written to a new layer.

Now it’s time to run our new model and check that it does what it promised. We can also uncheck the final output checkbox and send filtered features to further processing algorithms. For example sending them through a buffer based on an attribute size (although as a QGIS professional you know you should rather be using styles than modifying the geometry in most situations in such cases).

Conclusion

With this new algorithm built directly inside the core of QGIS, the processing framework is now able to transform and refine features of a dataset with the same precision as an open heart surgery. Of course you can get more creative in the filter criteria. Apart from the obvious ones to do geometry modifications, there are two particularly interesting ones if you liked this one

• The Refactor Fields algorithm allows calculating new fields or rename fields based on expressions
• The Append plugin allows adding those features to an existing vector layer such as a database table

The data from this walkthrough is available for download as [download id=“3917”]. If you would like to test this new feature but do not yet have a concrete use-case in mind, here is a task for you: get an openstreetmap extract, import it using ogr2ogr and split the lines into different layers roads, rivers and railways, the polygons into lakes, forests and cities, the points according to your own liking. If there is big enough interest for this, we might write another blog post on this topic.

We would like to thank the QGIS user group Switzerland for making this project possible through funding.