Page 1 of 3 (58 posts)

  • talks about »
  • geospatial

Tags

Last update:
Fri Jul 19 16:31:13 2019

A Django site.

QGIS Planet

QGIS Print Layouts Graphs and Charts — target reached!

We’ve just passed the extended deadline for our recent QGIS Print Layouts Graphs and Charts campaign, and the great news is that thanks to a large number of generous backers we’ve successfully hit the target for this campaign! This has only been possible thanks to the tireless work of the QGIS community and user groups in promoting this campaign and spreading the word.

The Print Layouts Graphs and Charts campaign is a joint effort with our friends at Faunalia, so we’ll soon be starting work together on all the wonderful new functionality heading to the QGIS DataPlotly plugin as a result. The work will be commencing late June, just after the QGIS 3.8.0 final release. Keep an eye out for further updates on the development from this time! You can read more about what’s coming in detail at the campaign page.

We’d like to take this opportunity to extend our heartfelt thanks to all the backers who have pledged to support this project:

  • Federico Gianoli
  • Papercraft Mountains
  • Liam McCrae
  • Henry Walshaw
  • Raúl Sangonzalo
  • Ferdinando Urbano
  • pitsch-ing.ch
  • Carbon-X
  • Gabriel Diosan
  • Rene Giovanni Borella
  • Enrico Bertonati
  • Guido Ingwer
  • David Addy
  • Gerd Jünger
  • Andreas Neumann
  • Stefano Campus
  • Michael Jabot
  • Korto
  • Enrico Ferreguti
  • Carlo A. Nicolini
  • Salvatore Fiandaca
  • Alberto Grava
  • Hans van der Kwast
  • Ben Hur Pintor
  • Silvio Grosso
  • Nobusuke Iwasaki
  • Alasdair Rae
  • Manori Senanayake
  • Canton de Neuchâtel
  • Matthias Daues
  • Alteri Seculo
  • SunGIS Ltd.
  • Stu Smith
  • Keolis Rennes
  • Gabriel Diosan
  • Aiden Price
  • Giacomo Ponticelli
  • Diane Fritz
  • Gemio Bissolati
  • Claire Birnie
  • Nicolas Roelandt
  • Rocco Pispico
  • Gabriel Bengtsson
  • Birds Eye View
  • Barend Köbben
  • Roberto Marzocchi (GTER)
  • Yoichi Kayama
  • Alessandro Sarretta
  • Luca Angeli
  • Luca Bellani
  • giswelt
  • Stefan Giese
  • Ben Harding
  • Joao Gaspar
  • Romain Lacroix
  • Ryan Cooper
  • Daniele Bonaposta
  • QGIS Swedish User Group
  • Nino Formica
  • Michael Gieding
  • Amedeo Fadini
  • Andrew Hannell
  • Stefano
  • Phil Wyatt
  • Brett Edmond Carlock
  • Transitec

 

QGIS Print Layouts Graphs and Charts – campaign deadline extended!

If you’re a regular reader of this blog, it won’t surprise you to hear that we’re very excited about adding rich charting and graph functionality to QGIS’ Print Layout designer! Alongside our friends at Faunalia, we’re currently running a crowd funding campaign to make this a reality.

So, while the required funds weren’t raised within our original April 30 deadline, we’ve decided to extend this campaign by an additional 30 days in the hopes that the users and organisations from the wider QGIS community will jump onboard and pledge the remaining required funds.

This missing feature is a large gap in QGIS printing capabilities, so we’re counting on you to show your support and spread the word to your local user groups, QGIS users, and any organisations you know of who rely on QGIS and would love to see its inbuilt reporting capabilities levelled up!

QGIS (and SLYR!), now with Hash Lines support

Thanks to an anonymous corporate sponsor, we’ve recently had the opportunity to add a new Hashed Line symbol type for QGIS 3.8. This allows for a repeating line segment to be drawn over the length of a feature, with a line-sub symbol used to render each individual segment.

There’s tons of options available for customising the appearance and placement of line hashes. We based the feature heavily off QGIS’ existing “Marker Line” support, so you can create hashed lines placed at set intervals, on line vertices, or at the start/end/middle of lines. There’s options to offset the lines, and tweak the rotation angle of individual hashes too. Added to QGIS’ rich support for “data defined” symbol properties, this allows for a huge range of new symbol effects.

E.g. using a data defined hash length which increases over the length of a feature gives us this effect:

Or, when using a complex line sub-symbol for rendering each hash, we can get something like this:

Or even go completely “meta” and use a hashed line sub symbol for the hash line itself!

With the right combination of symbol settings and QGIS draw effects you can even emulate a calligraphic pen effect:

Or a chunky green highlighter!

This same corporate sponsor also funded a change which results in a huge improvement to the appearance of both rotated hashed lines and marker lines. Previously, when marker or hash lines were rendered, the symbol angles were determined by taking the exact line orientation at the position of the symbol. This often leads to undesirable rendering effects, where little “bumps” or corners in lines which occur at the position of the symbol cause the marker or hash line to be oriented at a very different angle to what the eye expects to see.

With this improvement, the angle is instead calculated by averaging the line over a specified distance either side of the symbol. E.g. averaging the line angle over 4mm means we take the points along the line 2mm from either side of the symbol placement, and use these instead to calculate the line angle for that symbol. This has the effect of smoothing (or removing) any tiny local deviations from the overall line direction, resulting in much nicer visual orientation of marker or hash lines.

It’s easiest to show the difference visually. Here’s a before image, showing arrow markers following a line feature. Pay specific attention to the 3rd and last arrow, which seem completely random oriented due to the little shifts in line direction:

With new smoothing logic this is much improved:

The difference is even more noticeable for hashed lines. Here’s the before:

…and the after:

Suffice to say, cartographers will definitely appreciate the result!

Lastly, we’ve taken this new hash line feature as an opportunity to implement automatic conversion of ESRI hash line symbols within our SLYR ESRI to QGIS conversion tool. Read more about SLYR here, and how you can purchase this tool for .style, .lyr and .mxd document conversion.

QGIS Print Layouts Graphs and Charts crowdfund launched!

Ever wished QGIS had a way to insert dynamic, feature rich charts and graphs directly inside print layouts? If so, our latest crowdfunding campaign has you covered! This missing feature is a large gap in QGIS printing capabilities, so we’re planning on filling that gap by exposing the powerful QGIS “Data Plotly” plugin to allow these charts to be embedded inside your layouts, and allow them to be created and modified in a simple, interactive style.

If you’re not aware of the existing capabilities of the DataPlotly plugin, here’s a quick screencast which should get you excited about the possibilities here…

QGIS is already a reporting powerhouse, and we believe that linking DataPlotly with QGIS print layouts will boost the current functionality up an order of magnitude! To make it possible we need 8600€ pledged before 30 April 2019. North Road is collaborating on this campaign with our friends at Faunalia, and development work will be shared between the two consultancy firms.

You can help make this a reality by supporting the campaign or by sharing the page and increasing exposure to the campaign. Full details about the planned functionality and how to contribute are available at the campaign page.

Announcing our SLYR (MXD to QGIS) funding drive!

One product which North Road had the chance to develop last year, and which we are super-proud of, is our SLYR ESRI style to QGIS conversion tool. If you haven’t heard of it before, this tool allows automatic conversion of ESRI .style database files to their equivalent QGIS symbology equivalent. It works well for the most part, and now we’re keen to take this to the next stage.

The good news is that North Road have been conducting extensive research and development over the past 12 months, and we’re pleased to announce our plans for extending SLYR to support ESRI LYR and MXD documents. The LYR and MXD formats are proprietary ESRI-only formats, with no public specifications allowing their use. This is a huge issue for organisations who want to move from an ESRI environment to the open geospatial world, yet are held back by hundreds (or thousands!) of existing ESRI MXD map documents and layer styles which they currently cannot utilise outside of the ESRI software ecosystem. Furthermore, many providers of spatial data only include ESRI specific layer formatting files with their data supplies. This leaves users with no means of utilising these official, pre-defined styles in non-ESRI tools.

In order for us to continue development of the SLYR tool and unlock use of LYR and MXD formats outside of ESRI tools, we are conducting a funding campaign. Sponsors of the campaign will receive access to the tools as they are developed and gain access to official support channels covering their use. At the conclusion of this drive we’ll be releasing all the tools and specifications under a free, open-source license.

You can read the full details of the campaign here, including pricing to become a project sponsor and gain access to the tools as they develop. As a campaign launch promo, we’re offering the first 10 sponsors a super-special discounted rate (as a reward for jumping on the development early).

The mockup below shows what the end goal is: seamless, fully integrated, automatic conversion of MXD and LYR files directly within the QGIS desktop application!

Thoughts on “FOSS4G/SOTM Oceania 2018”, and the PyQGIS API improvements which it caused

Last week the first official “FOSS4G/SOTM Oceania” conference was held at Melbourne University. This was a fantastic event, and there’s simply no way I can extend sufficient thanks to all the organisers and volunteers who put this event together. They did a brilliant job, and their efforts are even more impressive considering it was the inaugural event!

Upfront — this is not a recap of the conference (I’m sure someone else is working on a much more detailed write up of the event!), just some musings I’ve had following my experiences assisting Nathan Woodrow deliver an introductory Python for QGIS workshop he put together for the conference. In short, we both found that delivering this workshop to a group of PyQGIS newcomers was a great way for us to identify “pain points” in the PyQGIS API and areas where we need to improve. The good news is that as a direct result of the experiences during this workshop the API has been improved and streamlined! Let’s explore how:

Part of Nathan’s workshop (notes are available here) focused on a hands-on example of creating a custom QGIS “Processing” script. I’ve found that preparing workshops is guaranteed to expose a bunch of rare and tricky software bugs, and this was no exception! Unfortunately the workshop was scheduled just before the QGIS 3.4.2 patch release which fixed these bugs, but at least they’re fixed now and we can move on…

The bulk of Nathan’s example algorithm is contained within the following block (where “distance” is the length of line segments we want to chop our features up into):

for input_feature in enumerate(features):
    geom = feature.geometry().constGet()
    if isinstance(geom, QgsLineString):
        continue
    first_part = geom.geometryN(0)
    start = 0
    end = distance
    length = first_part.length()

    while start < length:
        new_geom = first_part.curveSubstring(start,end)

        output_feature = input_feature
        output_feature.setGeometry(QgsGeometry(new_geom))
        sink.addFeature(output_feature)

        start += distance
        end += distance

There’s a lot here, but really the guts of this algorithm breaks down to one line:

new_geom = first_part.curveSubstring(start,end)

Basically, a new geometry is created for each trimmed section in the output layer by calling the “curveSubstring” method on the input geometry and passing it a start and end distance along the input line. This returns the portion of that input LineString (or CircularString, or CompoundCurve) between those distances. The PyQGIS API nicely hides the details here – you can safely call this one method and be confident that regardless of the input geometry type the result will be correct.

Unfortunately, while calling the “curveSubstring” method is elegant, all the code surrounding this call is not so elegant. As a (mostly) full-time QGIS developer myself, I tend to look over oddities in the API. It’s easy to justify ugly API as just “how it’s always been”, and over time it’s natural to develop a type of blind spot to these issues.

Let’s start with the first ugly part of this code:

geom = input_feature.geometry().constGet()
if isinstance(geom, QgsLineString):
    continue
first_part = geom.geometryN(0)
# chop first_part into sections of desired length
...

This is rather… confusing… logic to follow. Here the script is fetching the geometry of the input feature, checking if it’s a LineString, and if it IS, then it skips that feature and continues to the next. Wait… what? It’s skipping features with LineString geometries?

Well, yes. The algorithm was written specifically for one workshop, which was using a MultiLineString layer as the demo layer. The script takes a huge shortcut here and says “if the input feature isn’t a MultiLineString, ignore it — we only know how to deal with multi-part geometries”. Immediately following this logic there’s a call to geometryN( 0 ), which returns just the first part of the MultiLineString geometry.

There’s two issues here — one is that the script just plain won’t work for LineString inputs, and the second is that it ignores everything BUT the first part in the geometry. While it would be possible to fix the script and add a check for the input geometry type, put in logic to loop over all the parts of a multi-part input, etc, that’s instantly going to add a LOT of complexity or duplicate code here.

Fortunately, this was the perfect excuse to improve the PyQGIS API itself so that this kind of operation is simpler in future! Nathan and I had a debrief/brainstorm after the workshop, and as a result a new “parts iterator” has been implemented and merged to QGIS master. It’ll be available from version 3.6 on. Using the new iterator, we can simplify the script:

geom = input_feature.geometry()
for part in geom.parts():
    # chop part into sections of desired length
    ...

Win! This is simultaneously more readable, more Pythonic, and automatically works for both LineString and MultiLineString inputs (and in the case of MultiLineStrings, we now correctly handle all parts).

Here’s another pain-point. Looking at the block:

new_geom = part.curveSubstring(start,end)
output_feature = input_feature
output_feature.setGeometry(QgsGeometry(new_geom))

At first glance this looks reasonable – we use curveSubstring to get the portion of the curve, then make a copy of the input_feature as output_feature (this ensures that the features output by the algorithm maintain all the attributes from the input features), and finally set the geometry of the output_feature to be the newly calculated curve portion. The ugliness here comes in this line:

output_feature.setGeometry(QgsGeometry(new_geom))

What’s that extra QgsGeometry(…) call doing here? Without getting too sidetracked into the QGIS geometry API internals, QgsFeature.setGeometry requires a QgsGeometry argument, not the QgsAbstractGeometry subclass which is returned by curveSubstring.

This is a prime example of a “paper-cut” style issue in the PyQGIS API. Experienced developers know and understand the reasons behind this, but for newcomers to PyQGIS, it’s an obscure complexity. Fortunately the solution here was simple — and after the workshop Nathan and I added a new overload to QgsFeature.setGeometry which accepts a QgsAbstractGeometry argument. So in QGIS 3.6 this line can be simplified to:

output_feature.setGeometry(new_geom)

Or, if you wanted to make things more concise, you could put the curveSubstring call directly in here:

output_feature = input_feature
output_feature.setGeometry(part.curveSubstring(start,end))

Let’s have a look at the simplified script for QGIS 3.6:

for input_feature in enumerate(features):
    geom = feature.geometry()
    for part in geom.parts():
        start = 0
        end = distance
        length = part.length()

        while start < length:
            output_feature = input_feature
            output_feature.setGeometry(part.curveSubstring(start,end))
            sink.addFeature(output_feature)

            start += distance
            end += distance

This is MUCH nicer, and will be much easier to explain in the next workshop! The good news is that Nathan has more niceness on the way which will further improve the process of writing QGIS Processing script algorithms. You can see some early prototypes of this work here:

So there we go. The process of writing and delivering a workshop helps to look past “API blind spots” and identify the ugly points and traps for those new to the API. As a direct result of this FOSS4G/SOTM Oceania 2018 Workshop, the QGIS 3.6 PyQGIS API will be easier to use, more readable, and less buggy! That’s a win all round!

Edit Features “In Place” crowdfund — made it to QGIS 3.4!

Well, thanks to the resounding success of our QGIS edit-in-place crowdfunding campaign, we’ve been frantically smashing away at our keyboards in an attempt to reward the QGIS community by sneaking this feature in a whole 4 months earlier than originally promised! And, we’re very proud to announce, that this exciting new feature has been implemented and will be included in the upcoming QGIS 3.4 release (due late October 2018). So go ahead — grab one of the nightly pre-release of QGIS 3.4 and checkout the results.

This wouldn’t have been possible without the rapid response to the campaign and the generosity of our wonderful backers:

(In addition to these backers, we’ve also received numerous anonymous donations to this feature from many other individuals — while we can’t list you all publicly, you’re also in our thanks!)

 

Keep an eye on this blog for other upcoming QGIS crowdfunding campaigns targeted at QGIS 3.6 and beyond… we’ve got lots more exciting work planned for these releases!

 

Edit Features “In Place” crowdfund — target reached!

Well, the final pledges have been tallied and we’re very proud to announce that our latest crowd funding campaign has been a roaring success!

We’ve been completely blown away by the response to this campaign. Thanks to some incredibly generous backers and donors, we’ve been able to hit the campaign target with plenty of time to spare. As a result, we’ll be pushing hard to reward the generosity of the community by trying to sneak this feature in for the upcoming QGIS 3.4 release (instead of the originally promised 3.6 release)! You can read more about what we’re adding at the campaign page.

We’d like to take this opportunity to extend our heartfelt thanks to all the backers who have pledged to support this project:

In addition to these backers, we’ve also received numerous anonymous donations to this feature from many other individuals — while we can’t list you all publicly, you’re also in our thanks!

Stay tuned for more updates to come as work proceeds on this feature…

Edit Features “In Place” Using QGIS Spatial Operations crowdfund launched!

We’ve just launched a new QGIS crowd funding campaign which we’re super-excited about! This time, we’re addressing what we see as the major shortcoming within QGIS vector layer editing tools, and bridging the gap between the vast power of QGIS’ Processing algorithms and easy-to-use operations which modify layer features “in place”. Here’s a quick sneak preview of what we have planned:

 

QGIS is already a vector editing powerhouse, and we believe that this improvement will boost the current functionality up an order of magnitude! To make it possible we need 6500€ pledged before 30 September 2018.

This is also our first crowdfunding campaign in which we’re running a “dual funding” approach, which we think should make things friendly and easy for both corporate backers and end user contributions alike. Read more about this at the full campaign page.

You can help make this a reality by supporting the campaign or by sharing the page and increasing exposure to the campaign. Updates to follow!

 

Drill down (cascading) forms in QGIS crowdfund – final stretch!

Update: donations are now closed, with the outcome of the campaign pending!

We’re nearing the final hours of our crowd funding campaign to implement a drill-down (cascading) field support within QGIS forms, and thanks to numerous generous backers we’re very close to hitting the funding goal! This is a really exciting new feature which would help add greater flexibility and power to QGIS feature forms, but in order to implement it for QGIS 3.2 we need to hit the funding target by 11 May 2018.

As a result, we’re dropping the minimum contribution amount and throwing open the campaign for payments of any amount. These smaller payment will be treated as direct donations to the campaign, so unlike the standard campaign backing these are payable up front. In the case that the campaign IS NOT successful, the donations will not be refunded and will instead be reinvested back into the QGIS (via bug fixing and maintenance efforts). Of course, if you’d prefer to pledge using the standard crowdfunding “no payment if campaign unsuccessful” model you’re more than welcome to! (Full details are available on the campaign page).

Donations closed – outcome pending!

Full details are available on the campaign page.

Drill-down (cascading) forms in QGIS crowdfund launched!

We’ve just launched a new crowd funding campaign to implement a drill-down (cascading) field support within QGIS forms. Full details are available on the campaign page.

This is a really exciting new feature which would help add greater flexibility and power to QGIS feature forms! To make it possible we need 3500€ pledged before 11 May 2018. You can help make this a reality by supporting the campaign or by sharing the page and increasing exposure to the campaign. Updates to follow!

Implementing an in-house “New Project Wizard” for QGIS

Recently, we were required to implement a custom “New Project Wizard” for use in a client’s internal QGIS installation. The goal here was that users would be required to fill out certain metadata fields whenever they created a new QGIS project.

Fortunately, the PyQGIS (and underlying Qt) libraries makes this possibly, and relatively straightforward to do. Qt has a powerful API for creating multi-page “wizard” type dialogs, via the QWizard and QWizardPage classes. Let’s have a quick look at writing a custom wizard using these classes, and finally we’ll hook it into the QGIS interface using some PyQGIS magic.

We’ll start super simple, creating a single page wizard with no settings. To do this we first create a Page1 subclass of QWizardPage, a ProjectWizard subclass of QWizard, and a simple runNewProjectWizard function which launches the wizard. (The code below is designed for QGIS 3.0, but will run with only small modifications on QGIS 2.x):

class Page1(QWizardPage):

    def __init__(self, parent=None):
        super().__init__(parent)
        self.setTitle('General Properties')
        self.setSubTitle('Enter general properties for this project.')


class ProjectWizard(QWizard):
    
    def __init__(self, parent=None):
        super().__init__(parent)
        
        self.addPage(Page1(self))
        self.setWindowTitle("New Project")


def runNewProjectWizard():
    d=ProjectWizard()
    d.exec()

If this code is executed in the QGIS Python console, you’ll see something like this:

Not too fancy (or functional) yet, but still not bad for 20 lines of code! We can instantly make this a bit nicer by inserting a custom logo into the widget. This is done by calling setPixmap inside the ProjectWizard constructor.

class ProjectWizard(QWizard):
    
    def __init__(self, parent=None):
        super().__init__(parent)
        
        self.addPage(Page1(self))
        self.setWindowTitle("New Project")

        logo_image = QImage('path_to_logo.png')
        self.setPixmap(QWizard.LogoPixmap, QPixmap.fromImage(logo_image))

That’s a bit nicer. QWizard has HEAPS of options for tweaking the wizards — best to read about those over at the Qt documentation. Our next step is to start adding some settings to this wizard. We’ll keep things easy for now and just insert a number of text input boxes (QLineEdits) into Page1:

class Page1(QWizardPage):

    def __init__(self, parent=None):
        super().__init__(parent)
        self.setTitle('General Properties')
        self.setSubTitle('Enter general properties for this project.')

        # create some widgets
        self.project_number_line_edit = QLineEdit()
        self.project_title_line_edit = QLineEdit()
        self.author_line_edit = QLineEdit()        
        
        # set the page layout
        layout = QGridLayout()
        layout.addWidget(QLabel('Project Number'),0,0)
        layout.addWidget(self.project_number_line_edit,0,1)
        layout.addWidget(QLabel('Title'),1,0)
        layout.addWidget(self.project_title_line_edit,1,1)
        layout.addWidget(QLabel('Author'),2,0)
        layout.addWidget(self.author_line_edit,2,1)
        self.setLayout(layout)

There’s nothing particularly new here, especially if you’ve used Qt widgets before. We make a number of QLineEdit widgets, and then create a grid layout containing these widgets and accompanying labels (QLabels). Here’s the result if we run our wizard now:

So now there’s the option to enter a project number, title and author. The next step is to force users to populate these fields before they can complete the wizard. Fortunately, QWizardPage has us covered here and we can use the registerField() function to do this. By calling registerField, we make the wizard aware of the settings we’ve added on this page, allowing us to retrieve their values when the wizard completes. We can also use registerField to automatically force their population by appending a * to the end of the field names. Just like this…

class Page1(QWizardPage):
    def __init__(self, parent=None):
        super().__init__(parent)
        ...
        self.registerField('number*',self.project_number_line_edit)
        self.registerField('title*',self.project_title_line_edit)
        self.registerField('author*',self.author_line_edit)

If we ran the wizard now, we’d be forced to enter something for project number, title and author before the Finish button becomes enabled. Neat! By registering the fields, we’ve also allowed their values to be retrieved after the wizard completes. Let’s alter runNewProjectWizard to retrieve these values and do something with them:

def runNewProjectWizard():
   d=ProjectWizard()
   d.exec()

   # Set the project title
   title=d.field('title')
   QgsProject.instance().setTitle(d.field('title'))

   # Create expression variables for the author and project number
   number=d.field('number')
   QgsExpressionContextUtils.setProjectVariable(QgsProject.instance(),'project_number', number)
   author=d.field('author')
   QgsExpressionContextUtils.setProjectVariable(QgsProject.instance(),'project_author', author)
 

Here, we set the project title directly and create expression variables for the project number and author. This allows their use within QGIS expressions via the @project_number and @project_author variables. Accordingly, they can be embedded into print layout templates so that layout elements are automatically populated with the corresponding author and project number. Nifty!

Ok, let’s beef up our wizard by adding a second page, asking the user to select a sensible projection (coordinate reference system) for their project. Thanks to improvements in QGIS 3.0, it’s super-easy to embed a powerful pre-made projection selector widget into your scripts, which even includes a handy preview of the area of the world that the projection is valid for.

class Page2(QWizardPage):

    def __init__(self, parent=None):
        super().__init__(parent)
        self.setTitle('Project Coordinate System')
        self.setSubTitle('Choosing an appropriate projection is important to ensure accurate distance and area measurements.')
        
        self.proj_selector = QgsProjectionSelectionTreeWidget()
        layout = QVBoxLayout()
        layout.addWidget(self.proj_selector)
        self.setLayout(layout)
        
        self.registerField('crs',self.proj_selector)
        self.proj_selector.crsSelected.connect(self.crs_selected)
        
    def crs_selected(self):
        self.setField('crs',self.proj_selector.crs())
        self.completeChanged.emit()
        
    def isComplete(self):
        return self.proj_selector.crs().isValid()

There’s a lot happening here. First, we subclass QWizardPage to create a second page in our widget. Then, just like before, we add some widgets to this page and set the page’s layout. In this case we are using the standard QgsProjectionSelectionTreeWidget to give users a projection choice. Again, we let the wizard know about our new setting by a call to registerField. However, since QWizard has no knowledge about how to handle a QgsProjectionSelectionTreeWidget, there’s a bit more to do here. So we make a connection to the projection selector’s crsSelected signal, hooking it up to a function which sets the wizard’s “crs” field value to the widget’s selected CRS. Here, we also emit the completeChanged signal, which indicates that the wizard page should re-validate the current settings. Lastly, we override QWizardPage’s isComplete method, checking that there’s a valid CRS selection in the selector widget. If we run the wizard now we’ll be forced to choose a valid CRS from the widget before the wizard allows us to proceed:

Lastly, we need to adapt runNewProjectWizard to also handle the projection setting:

def runNewProjectWizard():
    d=ProjectWizard()
    d.exec()

    # Set the project crs
    crs=d.field('crs')
    QgsProject.instance().setCrs(crs)

    # Set the project title
    title=d.field('title')
    ...

Great! A fully functional New Project wizard. The final piece of the puzzle is triggering this wizard when a user creates a new project within QGIS. To do this, we hook into the iface.newProjectCreated signal. By connecting to this signal, our code will be called whenever the user creates a new project (after all the logic for saving and closing the current project has been performed). It’s as simple as this:

iface.newProjectCreated.connect(runNewProjectWizard)

Now, whenever a new project is made, our wizard is triggered – forcing users to populate the required fields and setting up the project accordingly!

There’s one last little bit to do – we also need to prevent users cancelling or closing the wizard before completing it. That’s done by changing a couple of settings in the ProjectWizard constructor, and by overriding the default reject method (which prevents closing the dialog by pressing escape).

class ProjectWizard(QWizard):
    
    def __init__(self, parent=None):
        super().__init__(parent)
        ...
        self.setOption(QWizard.NoCancelButton, True)
        self.setWindowFlags(self.windowFlags() | QtCore.Qt.CustomizeWindowHint)
        self.setWindowFlags(self.windowFlags() & ~QtCore.Qt.WindowCloseButtonHint)

    def reject(self):
        pass

Here’s the full version of our code, ready for copying and pasting into the QGIS Python console:

icon_path = '/home/nyall/nr_logo.png'

class ProjectWizard(QWizard):
    
    def __init__(self, parent=None):
        super().__init__(parent)
        
        self.addPage(Page1(self))
        self.addPage(Page2(self))
        self.setWindowTitle("New Project")
        
        logo_image=QImage('path_to_logo.png')
        self.setPixmap(QWizard.LogoPixmap, QPixmap.fromImage(logo_image))
        
        self.setOption(QWizard.NoCancelButton, True)
        self.setWindowFlags(self.windowFlags() | QtCore.Qt.CustomizeWindowHint)
        self.setWindowFlags(self.windowFlags() & ~QtCore.Qt.WindowCloseButtonHint)
    def reject(self):
        pass
class Page1(QWizardPage):
    
    def __init__(self, parent=None):
        super().__init__(parent)
        self.setTitle('General Properties')
        self.setSubTitle('Enter general properties for this project.')

        # create some widgets
        self.project_number_line_edit = QLineEdit()
        self.project_title_line_edit = QLineEdit()
        self.author_line_edit = QLineEdit()        
        
        # set the page layout
        layout = QGridLayout()
        layout.addWidget(QLabel('Project Number'),0,0)
        layout.addWidget(self.project_number_line_edit,0,1)
        layout.addWidget(QLabel('Title'),1,0)
        layout.addWidget(self.project_title_line_edit,1,1)
        layout.addWidget(QLabel('Author'),2,0)
        layout.addWidget(self.author_line_edit,2,1)
        self.setLayout(layout)
        
        self.registerField('number*',self.project_number_line_edit)
        self.registerField('title*',self.project_title_line_edit)
        self.registerField('author*',self.author_line_edit)
 
 
class Page2(QWizardPage):
    
    def __init__(self, parent=None):
        super().__init__(parent)
        self.setTitle('Project Coordinate System')
        self.setSubTitle('Choosing an appropriate projection is important to ensure accurate distance and area measurements.')
        
        self.proj_selector = QgsProjectionSelectionTreeWidget()
        layout = QVBoxLayout()
        layout.addWidget(self.proj_selector)
        self.setLayout(layout)
        
        self.registerField('crs',self.proj_selector)
        self.proj_selector.crsSelected.connect(self.crs_selected)
        
    def crs_selected(self):
        self.setField('crs',self.proj_selector.crs())
        self.completeChanged.emit()
        
    def isComplete(self):
        return self.proj_selector.crs().isValid()
 
        
def runNewProjectWizard():
    d=ProjectWizard()
    d.exec()
    
    # Set the project crs
    crs=d.field('crs')
    QgsProject.instance().setCrs(crs)
    
    # Set the project title
    title=d.field('title')
    QgsProject.instance().setTitle(d.field('title'))

    # Create expression variables for the author and project number
    number=d.field('number')
    QgsExpressionContextUtils.setProjectVariable(QgsProject.instance(),'project_number', number)
    author=d.field('author')
    QgsExpressionContextUtils.setProjectVariable(QgsProject.instance(),'project_author', author)
    
    
iface.newProjectCreated.connect(runNewProjectWizard)

QGIS layouts rewrite – progress report #1

Following our recent successful QGIS Layout and Reporting Engine crowdfunding campaign, we’ve been hard at working ripping up the internals of the QGIS 2.x print composer and rebuilding a brand new, shiny QGIS layouts engine. This is exciting work – it’s very satisfying to be able to cleanup a lot of the old composer code in QGIS and take opportunities along the way to fix long standing bugs and add new features.

While it’s not ready for daily use yet, there’s already been a lot of interesting changes which have landed in the layouts work as a result of this campaign. Let’s take a look at what’s been implemented so far…

  • We’ve added support for different measurements units all throughout layouts. While this means it’s now possible to set page sizes using centimeters, inches, pixels, points, etc, it goes much deeper than just that. In layouts, everything which has a size or position can take advantage of this unit support. So you can have page sizes in centimeters, but a map item with a size set in points, and positioned in millimeters! Having pixels as a unit type makes creation of screen-based layouts much easier – even right down to pixel perfect positioning and sizing of items…
  • Page handling has been totally reworked. Instead of the single “number of pages” control available in QGIS 2.x, layouts have complete flexibility in page setup. It’s now possible to have a layout with mixed page sizes and orientations (including data defined page size for different pages in the layout!). 
  • A revised status bar, with improved layout interaction widgets. We’ve also taken the opportunity to add some new features like a zoom level slider and option to zoom to layout width:
  • Layout interaction tools (such as pan/zoom/insert item/etc) have been reworked. There’s now a much more flexible framework for creation of layout tools (based off the main QGIS map canvas approach), which even allows for plugins to implement their own layout interaction tools! As part of this we’ve addressed a long standing annoyance which meant that creating new items always drew the “preview” shape of the new item as a rectangle – even for non-rectangular items. Now you get a real shape showing exactly how the created item will be sized and positioned:
  • On the topic of plugins – the layout branch has full support for plugin-provided item types. This means that QGIS plugins can create new classes of items which can be added to a layout. This opens the door for plugins allowing charts and visualisations which take advantage of all the mature Python and JS charting libraries! This is a really exciting change – in 2.x there was no way for plugins to extend or interact with composer, so we’re really keen to see where the community takes this when 3.0 is released.
  • We’ve ported another feature commonly found in illustration/DTP applications. Now, when you’re creating a new item and just click in your layout (instead of click-and-drag), you get a handy dialog allowing you to specify the exact position and dimensions for the created item. You can again see in this dialog how layouts have full support for units for both the position and size:
  • Another oft-requested feature which we’ve finally been able to add (thanks to the refactored and cleaned code) is a context menu for layouts! It’s currently quite empty, but will be expanded as this work progresses…
  • Snapping to guides and grids has been reworked. We’ve added a new snapping marker to show exactly were items will be snapped to:
  • Snapping to guides now occurs when creating new layout items (this didn’t happen in Composer in 2.x – only snapping to grids occurred when drawing new items).
  • The snapped cursor position is shown in status bar whenever a snapped point will be used, instead of the unsnapped position.
  • Unlike in Composers in QGIS 2.x, Layouts in 3.0 adopt the standard UX of dragging out rulers to create guide lines (instead of clicking on a ruler position to create a new guide). Creation of a horizontal guide is now done by grabbing the top ruler and dragging it down, and a vertical guide is created by grabbing the left ruler and dragging it out to the layout.
  • Better feedback is given in the ruler when a guide can be dragged. We now show guide positions in the rulers, and give an indication (via mouse cursor change) when these guides can be repositioned by click-and-drag.
  • Another very exciting change is the addition of a new “Guide Manager”. The guide manager allows numeric modification of existing guides and creation of new guides. Finally it’s possible to position guides at exact locations! Again, you can see the full support for layout units in place here – guides can be positioned using any available unit.
  • There’s also a handy new shortcut in the Guide Manager to allow applying the guides from the current page to all other pages in your layout.
  • We’ve refined the snapping logic. In Composer in QGIS 2.x,  grids would always take precedence whenever both a grid and guide were within tolerance of a point. Now, guides will always take precedence – since they have been manually set by users we make the assumption that they have been explicitly placed at highly desirable snapping locations, and should be selected over the general background grid. Additionally, grid snapping was previously only done if BOTH the x and y of the point could be snapped to the grid. We now snap to the nearest grid line for x/y separately. This means if a point is close to a vertical grid line but not a horizontal one it will still snap to that nearby vertical grid line.
  • Lastly, we’ve added a handy context menu to the rulers:

This is just a taster of the great new functionality coming in QGIS 3.0. This is all a direct result of the forward-thinking investments and generosity of the backers in our QGIS Layout and Reporting Engine crowdfunding campaign. Without their contributions, none of this would be possible – so our thanks go out to those organisations and individuals once again!

Stay tuned for more updates as the work continues…

 

 

QGIS Layout and Reporting Engine Campaign – a success!

Thanks to the tireless efforts and incredible generosity of the QGIS user community, our crowdfunded QGIS Layout and Reporting Engine campaign was a tremendous success! We’ve reached the funding goal for this project, and as a result QGIS 3.0 will include a more powerful print composer with a reworked code base. You can read more about what we have planned at the campaign page.

We’d like to take this opportunity to extend our heartfelt thanks to all the backers who have pledged to support this project:

We’ve also received numerous anonymous contributions in addition to these – please know that the QGIS community extends their gratitude for your contributions too! This campaign was also successful thanks to The Agency for Data Supply and Efficiency, Denmark, who stepped up and have funded an initial component of this project directly.

We’d also like to thank every member of the QGIS community who assisted with promoting this campaign and bringing it to the attention of these backers. Without your efforts we would not have been able to reach these backers and the campaign would not have been successful.

We’ll be posting more updates as this work progresses. Stay tuned…

 

QGIS Composer Rewrite and Layout Engine crowdfund – half way there!

If you’ve been following our recent blog posts, you’ll be aware that we are currently running a crowd funding campaign to extend the capabilities of QGIS’ print composer. You can read full details about this over at the campaign page.

The good news is that we’ve just hit the mid way point of the funds! Many generous backers have stepped up with contributions and we’re well on the way to reaching the funding goal. However, we still need your help make this work a reality.

Right now, what we need most is interested users and community members who will reach out to their local QGIS users and seek more backing for the campaign. We need to publicise the campaign beyond the regular online QGIS community, to the thousands of enterprises and organisations which rely on QGIS for their daily mapping operations. We need community members who can get in contact with these organisations and help convince them that investing back into the open source software they utilise is beneficial (and often will even SAVE them money in the long run, due to the increased productivity that changes like our composer improvements will bring!).

So, while social media reshares have been vital to reaching the current stage, we now need more “hands on” helpers who will take this on. If you know of any organisations which depend on QGIS for their mapping outputs, now’s the time to get in contact with them directly and advise them of this campaign!

 

 

 

The Inaugural QGIS Australia Hackfest – Noosa 2017

Last week we kicked off the first (of hopefully many) Australian QGIS hackfests Developers Meetings. It was attended by 3 of the core QGIS development team: Nathan Woodrow, Martin Dobias and myself (Nyall Dawson), along with various family members. While there’s been QGIS hackfests in Europe for over 10 years, and others scattered throughout various countries (I think there was a Japanese one recently… but Twitter’s translate tool leaves me with little confidence about this!), there’s been no events like this in the Southern hemisphere yet. I’ve been to a couple in Europe and found them to be a great way to build involvement in the project, for both developers and non-developers alike.

In truth the Australian hackfest plans began mostly an excuse for Nathan and I to catch up with Martin Dobias before he heads back out of this hemisphere and returns to Europe. That said, Nathan and I have long spoken about ways we can build up the QGIS community in Australia, so in many ways this was a trial run for future events. It was based it in Noosa, QLD (and yes, we did manage to tear ourselves away from our screens long enough to visit the beach!).

Nathan Woodrow (@NathanW2), myself (@nyalldawson), and Martin Dobias (@wonder-sk)

Here’s a short summary of what we worked on during the hackfest:

  • Martin implemented a new iterator style accessor for vertices within geometries. The current approach to accessing vertices in QGIS is far from optimal. You either have the choice of an inefficient methods (eg QgsGeometry.asPolyline(), asPolygon(), etc) which requires translations of all vertices to a different data structure (losing any z/m dimensional values in the process), or an equally inefficient QgsAbstractGeometry.coordinateSequence() method, which at least keeps z/m values but still requires expensive copies of every vertex in the geometry. For QGIS 3.0 we’ve made a huge focus on optimising geometry operations and vertex access is one of the largest performance killers remaining in the QGIS code. Martin’s work adds a proper iterator for the vertices within a geometry object, both avoiding all these expensive copies and also simplifying the API for plugins. When this work lands traversing the vertices will become as simple as
for v in geom.vertices():
   ... do something with the vertex!
  • Martin is also planning on extending this work to allow simple iteration over the parts and rings within geometries too. When this lands in QGIS we can expect to see much faster geometry operations.
  • Nathan fixed a long standing hassle with running standalone PyQGIS scripts outside of the QGIS application on Windows. In earlier versions there’s a LOT of batch file mangling and environment variable juggling required before you can safely import the qgis libraries within Python. Thanks to Nathan’s work, in QGIS 3.0 this will be as simple as just making sure that the QGIS python libraries are included in your Python path, and then importing qgis.core/gui etc will work without any need to create environment variables for OSGEO/GDAL/PLUGINS/etc. Anyone who has fought with this in the past will definitely appreciate this change, and users of Python IDEs will also appreciate how simple it is now to make the PyQGIS libraries available in these environments.
  • Nathan also worked on “profiles” support for QGIS 3.0. This work will add isolated user profiles within QGIS, similar to how Chrome handles this. Each profile has it’s own separate set of settings, plugins, etc. This work is designed to benefit both plugin developers and QGIS users within enterprise environments. You can read more about what Nathan has planned for this here.
  • I continued the ongoing work of moving long running interface “blocking” operations to background tasks. In QGIS 3.0 many of these tasks churn away in the background, allowing you to continue work while the operation completes. It’s been implemented so far for vector and raster layer saving, map exports to images/PDF (not composers unfortunately), and obtaining feature counts within legends. During the hackfest I moved the layer import which occurs when you drag and drop a layer to a destination in the browser to a background task.
  • On the same topic, I took some inspiration from a commit in Sourcepole’s QGIS fork and reworked how composer maps are cached. One of my biggest gripes with QGIS’ composer is how slow it is to work with when you’ve got a complex map included. This change pushes the map redrawing into a background thread, so that these redraws no longer “lock up” the UI. It makes a HUGE difference in how usable composer is. This improvement also allowed me to remove those confusing map item “modes” (Cache/Render/Rectangle) – now everything is redrawn silently in the background whenever required.
  • Lastly, I spent a lot of time on a fun feature I’ve long wanted in QGIS – a unified search “locator” bar. This feature is heavily inspired by Qt Creator’s locator bar. It sits away down in the status bar, and entering any text here fires up a bunch of background search tasks. Inbuilt searches include searching the layers within the current project (am I the only one who loses layers in the tree in complex projects!?), print layouts in the project, processing algorithms, and menu/toolbar actions. The intention here is that plugins will “take over” and add additional search functionality, such as OSM place names searching, data catalog searches, etc. I’m sure when QGIS 3.0 is released this will quickly become indispensable!

The upcoming QGIS 3.0 locator bar

Big thanks go out to Nathan’s wife, Stacey, who organized most of the event and without whom it probably would never have happened, and to Lutra Consulting who sponsored an awesome dinner for the attendees.

We’d love this to be the first of many. The mature European hackfests are attended by a huge swath of the community, including translators, documentation writers, and plugin developers (amongst others). If you’ve ever been interested in finding out how you can get more involved in the project it’s a great way to dive in and start contributing. There’s many QGIS users in this part of the world and we really want to encourage a community of contributors who “give back” to the project. So let Nathan or myself know if you’d be interested in attending other events like this, or helping to organize them locally yourself…

About label halos

A lot of cartographers have a love/hate relationship with label halos. On one hand they can be an essential technique for improving label readability, especially against complex background layers. On the other hand they tend to dominate maps and draw unwanted attention to the map labels.

In this post I’m going to share my preferred techniques for using label halos. I personally find this technique is a good approach which minimises the negative effects of halos, while still providing a good boost to label readability. (I’m also going to share some related QGIS 3.0 news at the end of this post!)

Let’s start with some simple white labels over an aerial image:

These labels aren’t very effective. The complex background makes them hard to read, especially the “Winton Shire” label at the bottom of the image. A quick and nasty way to improve readability is to add a black halo around the labels:

Sure, it’s easy to read the labels now, but they stand out way too much and it’s difficult to see anything here except the labels!

We can improve this somewhat through a better choice of halo colour:

This is much better. We’ve got readable labels which aren’t too domineering. Unfortunately the halo effect is still very prominent, especially where the background image varies a lot. In this case it works well for the labels toward the middle of the map, but not so well for the labels at the top and bottom.

A good way to improve this is to take advantage of blending (or “composition”) modes (which QGIS has native support for). The white labels will be most readable when there’s a good contrast with the background map, i.e. when the background map is dark. That’s why we choose a halo colour which is darker than the text colour (or vice versa if you’ve got dark coloured labels). Unfortunately, by choosing the mid-toned brown colour to make the halos blend in more, we are actually lightening up parts of this background layer and both reducing the contrast with the label and also making the halo more visible. By using the “darken” blend mode, the brown halo will only be drawn for pixels were the brown is darker then the existing background. It will darken light areas of the image, but avoid lightening pixels which are already dark and providing good contrast. Here’s what this looks like:

The most noticeable differences are the labels shown above darker areas – the “Winton Shire” label at the bottom and the “Etheridge Shire” at the top. For both these labels the halo is almost imperceptible whilst still subtly doing it’s part to make the label readable. (If you had dark label text with a lighter halo color, you can use the “lighten” blend mode for the same result).

The only issue with this map is that the halo is still very obvious around “Shire” in “Richmond Shire” and “McKinlay” on the left of the map. This can be reduced by applying a light blur to the halo:

There’s almost no loss of readability by applying this blur, but it’s made those last prominent halos disappear into the map. At first glance you probably wouldn’t even notice that there’s any halos being used here. But if we compare back against the original map (which used no halos) we can see the huge difference in readability:

Compare especially the Winton Shire label at the bottom, and the Richmond Shire label in the middle. These are much clearer on our tweaked map versus the above image.

Now for the good news… when QGIS 3.0 is released you’ll no longer have to rely on an external illustration/editing application to get this effect with your maps. In fact, QGIS 3.0 is bringing native support for applying many types of live layer effects to label buffers and background shapes, including blur. This means it will be possible to reproduce this technique directly inside your GIS, no external editing or tweaking required!

QGIS Composer Rewrite and Layout Engine crowdfund launched!

At North Road we believe that crowdfunding is a sustainable way to maintain and enhance open source software, like the QGIS open source GIS package. We’ve run a number of successful crowdfunding campaigns in the past, including support in QGIS for live layer effects, a point cluster renderer, and a unique value renderer for raster layers.

Now, we’re proud to announce our latest crowd funding endeavour, and our biggest to date, the QGIS Layout and Reporting Engine Campaign.

This campaign covers stage 1 of a large, ongoing project to modernise and expand on QGIS’ print composer and layout facilities. Over time QGIS’ composer functionality has grown extensively and now is capable of creating flexible, high quality cartographic outputs. However, we’ve now hit a limit where the current code architecture is prohibiting further improvements and important fixes. In order to add a reporting framework to QGIS, it is necessary for us to refactor and improve large sections of the composer code.

If this campaign is successful, we’ll be adding flexible report generation features to QGIS and cleaning up all the existing composer code. As part of these clean up, we’ll be taking the opportunity to tackle a number of current limitations which cannot be addressed in the current composition code:

  • Layouts will become unit aware, allowing for item placement and properties using millimetres, inches, pixels, centimetres, points, etc.
  • Layouts will have the ability to include mixed page sizes and orientations.
  • Plugins will be able to create custom composer item types (eg allow utilisation of 3rd party graphing and visualisation libraries!).
  • Individual layout items can be rasterised without affecting the rest of the layout. For instance, a map which requires rasterisation due to its use of blend modes will not require all other layout items (such as headings, legends, etc) to be rasterised. This will greatly benefit PDF outputs for complex map layouts.
  • The code refresh will allow more extensive use of data defined layout item properties.
  • A render caching system will be implemented for items, speeding up use of the layout designer and also paving the way for use of live paint effects on layout items (eg dynamic drop shadows).

Full details on what we have planned are available here: QGIS Layout and Reporting Engine Campaign.

To make it possible we need 30,000€ pledged before 31 May 2017. You can help make this a reality by supporting the campaign or by sharing the campaign page and increasing exposure to the campaign. Updates to follow!

Speeding up your PyQGIS scripts

I’ve recently spent some time optimising the performance of various QGIS plugins and algorithms, and I’ve noticed that there’s a few common performance traps which developers fall into when fetching features from a vector layer. In this post I’m going to explore these traps, what makes them slow, and how to avoid them.

As a bit of background, features are fetched from a vector layer in QGIS using a QgsFeatureRequest object. Common use is something like this:

request = QgsFeatureRequest()
for feature in vector_layer.getFeatures(request):
    # do something

This code would iterate over all the features in layer. Filtering the features is done by tweaking the QgsFeatureRequest, such as:

request = QgsFeatureRequest().setFilterFid(1001)
feature_1001 = next(vector_layer.getFeatures(request))

In this case calling getFeatures(request) just returns the single feature with an ID of 1001 (which is why we shortcut and use next(…) here instead of iterating over the results).

Now, here’s the trap: calling getFeatures is expensive. If you call it on a vector layer, QGIS will be required to setup an new connection to the data store (the layer provider), create some query to return data, and parse each result as it is returned from the provider. This can be slow, especially if you’re working with some type of remote layer, such as a PostGIS table over a VPN connection. This brings us to our first trap:

Trap #1: Minimise the calls to getFeatures()

A common task in PyQGIS code is to take a list of feature IDs and then request those features from the layer. A see a lot of older code which does this using something like:

for id in some_list_of_feature_ids:
    request = QgsFeatureRequest().setFilterFid(id)
    feature = next(vector_layer.getFeatures(request))
    # do something with the feature

Why is this a bad idea? Well, remember that every time you call getFeatures() QGIS needs to do a whole bunch of things before it can start giving you the matching features. In this case, the code is calling getFeatures() once for every feature ID in the list. So if the list had 100 features, that means QGIS is having to create a connection to the data source, set up and prepare a query to match a single feature, wait for the provider to process that, and then finally parse the single feature result. That’s a lot of wasted processing!

If the code is rewritten to take the call to getFeatures() outside of the loop, then the result is:

request = QgsFeatureRequest().setFilterFids(some_list_of_feature_ids)
for feature in vector_layer.getFeatures(request):
    # do something with the feature

Now there’s just a single call to getFeatures() here. QGIS optimises this request by using a single connection to the data source, preparing the query just once, and fetching the results in appropriately sized batches. The difference is huge, especially if you’re dealing with a large number of features.

Trap #2: Use QgsFeatureRequest filters appropriately

Here’s another common mistake I see in PyQGIS code. I often see this one when an author is trying to do something with all the selected features in a layer:

for feature in vector_layer.getFeatures():
    if not feature.id() in vector_layer.selectedFeaturesIds():
        continue

    # do something with the feature

What’s happening here is that the code is iterating over all the features in the layer, and then skipping over any which aren’t in the list of selected features. See the problem here? This code iterates over EVERY feature in the layer. If you’re layer has 10 million features, we are fetching every one of these from the data source, going through all the work of parsing it into a QGIS feature, and then promptly discarding it if it’s not in our list of selected features. It’s very inefficient, especially if fetching features is slow (such as when connecting to a remote database source).

Instead, this code should use the setFilterFids() method for QgsFeatureRequest:

request = QgsFeatureRequest().setFilterFids(vector_layer.selectedFeaturesIds())
for feature in vector_layer.getFeatures(request):
    # do something with the feature

Now, QGIS will only fetch features from the provider with matching feature IDs from the list. Instead of fetching and processing every feature in the layer, only the actual selected features will be fetched. It’s not uncommon to see operations which previously took many minutes (or hours!) drop down to a few seconds after applying this fix.

Another variant of this trap uses expressions to test the returned features:

filter_expression = QgsExpression('my_field &gt; 20')
for feature in vector_layer.getFeatures():
    if not filter_expression.evaluate(feature):
        continue

    # do something with the feature

Again, this code is fetching every single feature from the layer and then discarding it if it doesn’t match the “my_field > 20” filter expression. By rewriting this to:

request = QgsFeatureRequest().setFilterExpression('my_field &gt; 20')
for feature in vector_layer.getFeatures(request):
    # do something with the feature

we hand over the bulk of the filtering to the data source itself. Recent QGIS versions intelligently translate the filter into a format which can be applied directly at the provider, meaning that any relevant indexes and other optimisations can be applied by the provider itself. In this case the rewritten code means that ONLY the features matching the ‘my_field > 20’ criteria are fetched from the provider – there’s no time wasted messing around with features we don’t need.

 

Trap #3: Only request values you need

The last trap I often see is that more values are requested from the layer then are actually required. Let’s take the code:

my_sum = 0
for feature in vector_layer.getFeatures(request):
    my_sum += feature['value']

In this case there’s no way we can optimise the filters applied, since we need to process every feature in the layer. But – this code is still inefficient. By default QGIS will fetch all the details for a feature from the provider. This includes all attribute values and the feature’s geometry. That’s a lot of processing – QGIS needs to transform the values from their original format into a format usable by QGIS, and the feature’s geometry needs to be parsed from it’s original type and rebuilt as a QgsGeometry object. In our sample code above we aren’t doing anything with the geometry, and we are only using a single attribute from the layer. By calling setFlags( QgsFeatureRequest.NoGeometry ) and setSubsetOfAttributes() we can tell QGIS that we don’t need the geometry, and we only require a single attribute’s value:

my_sum = 0
request = QgsFeatureRequest().setFlags(QgsFeatureRequest.NoGeometry).setSubsetOfAttributes(['value'], vector_layer.fields() )
for feature in vector_layer.getFeatures(request):
    my_sum += feature['value']

None of the unnecessary geometry parsing will occur, and only the ‘value’ attribute will be fetched and populated in the features. This cuts down both on the processing required AND the amount of data transfer between the layer’s provider and QGIS. It’s a significant improvement if you’re dealing with larger layers.

Conclusion

Optimising your feature requests is one of the easiest ways to speed up your PyQGIS script! It’s worth spending some time looking over all your uses of getFeatures() to see whether you can cut down on what you’re requesting – the results can often be mind blowing!

Point cluster renderer crowdfunding – successful!

Great news! Thanks in part to some generous last minute pledges, our QGIS Point Cluster Renderer campaign has successfully reached its target. This means that QGIS 3.0 will now include a full feature and flexible cluster renderer.

In the meantime, we’d like to extend our warmest thanks to the following generous contributors, whose pledges have made this work possible:

  • Andreas Neumann
  • Qtibia Engineering (Tudor Barascu)
  • Karl-Magnus Jönsson
  • Geonesia (Nicolas Ponzo)

Plus numerous additional anonymous backers whose generous contributions are also highly valued. If you run into any of these funders at a QGIS user group or conference, make sure you treat them like the GIS rock-stars they are!

Keep an eye out on our social media accounts as we’ll be posting more video demonstrations of this work as it lands in the QGIS codebase.

BOTH

  • Page 1 of 3 ( 58 posts )
  • >>
  • geospatial

Back to Top

Sponsors