Page 1 of 1 (5 posts)

  • talks about »
  • pyqt

Tags

Last update:
Tue Dec 11 12:00:42 2018

A Django site.

QGIS Planet

Using Threads in PyQGIS3

While porting a plugin to QGIS3 I decided to also move all it’s threading infrastructure to QgsTasks. Here three possible variants to implement this.
the first uses the static method QgsTask.fromFunction and is simpler to use. A great quick solution. If you want need control you can look at the second solution that subclasses QgsTask. In this solution I also show how to create subtasks with interdependencies. The third variant, illustrates how to run a processing algorithm in a separate thread.
One thing to be very careful about is never to create widgets or alter gui in a task. This is a strict Qt guideline – gui must never be altered outside of the main thread. So your progress dialog must operate on the main thread, connecting to the progress report signals from the task which operates in the background thread. This also applies to “print” statements — these aren’t safe to use from a background thread in QGIS and can cause random crashes. Use the thread safe QgsMessageLog.logMessage() approach instead. Actually you should forget print and always use QgsMessageLog.

using QgsTask.fromFunction

this is a quick and simple way of running a function in a separate thread. When calling QgsTask.fromFunction() you can pass an on_finished argument with a callback to be executed at the end of run.

from time import sleep
import random
MESSAGE_CATEGORY = 'My tasks from a function'
def run(task, wait_time):
"""a dumb test function
to break the task raise an exception
to return a successful result return it. This will be passed together
with the exception (None in case of success) to the on_finished method
"""
QgsMessageLog.logMessage('Started task {}'.format(task.description()),
MESSAGE_CATEGORY, Qgis.Info)
wait_time = wait_time / 100
total = 0
iterations = 0
for i in range(101):
sleep(wait_time)
# use task.setProgress to report progress
task.setProgress(i)
total += random.randint(0, 100)
iterations += 1
# check task.isCanceled() to handle cancellation
if task.isCanceled():
stopped(task)
return None
# raise exceptions to abort task
if random.randint(0, 500) == 42:
raise Exception('bad value!')
return {
'total': total, 'iterations': iterations, 'task': task.description()
}
def stopped(task):
QgsMessageLog.logMessage(
'Task "{name}" was cancelled'.format(name=task.description()),
MESSAGE_CATEGORY, Qgis.Info)
def completed(exception, result=None):
"""this is called when run is finished. Exception is not None if run
raises an exception. Result is the return value of run."""
if exception is None:
if result is None:
QgsMessageLog.logMessage(
'Completed with no exception and no result '\
'(probably the task was manually canceled by the user)',
MESSAGE_CATEGORY, Qgis.Warning)
else:
QgsMessageLog.logMessage(
'Task {name} completed\n'
'Total: {total} ( with {iterations} '
'iterations)'.format(
name=result['task'],
total=result['total'],
iterations=result['iterations']),
MESSAGE_CATEGORY, Qgis.Info)
else:
QgsMessageLog.logMessage("Exception: {}".format(exception),
MESSAGE_CATEGORY, Qgis.Critical)
raise exception
# a bunch of tasks
task1 = QgsTask.fromFunction(
'waste cpu 1', run, on_finished=completed, wait_time=4)
task2 = QgsTask.fromFunction(
'waste cpu 2', run, on_finished=completed, wait_time=3)
QgsApplication.taskManager().addTask(task1)
QgsApplication.taskManager().addTask(task2)

Subclassing QgsTask

this solution gives you the full control over the task behaviour. In this example I also illustrate how to create subtasks dependencies.

import random
from time import sleep
from qgis.core import (
QgsApplication, QgsTask, QgsMessageLog,
)
MESSAGE_CATEGORY = 'My subclass tasks'
class MyTask(QgsTask):
"""This shows how to subclass QgsTask"""
def __init__(self, description, duration):
super().__init__(description, QgsTask.CanCancel)
self.duration = duration
self.total = 0
self.iterations = 0
self.exception = None
def run(self):
"""Here you implement your heavy lifting. This method should
periodically test for isCancelled() to gracefully abort.
This method MUST return True or False
raising exceptions will crash QGIS so we handle them internally and
raise them in self.finished
"""
QgsMessageLog.logMessage('Started task "{}"'.format(
self.description()), MESSAGE_CATEGORY, Qgis.Info)
wait_time = self.duration / 100
for i in range(101):
sleep(wait_time)
# use setProgress to report progress
self.setProgress(i)
self.total += random.randint(0, 100)
self.iterations += 1
# check isCanceled() to handle cancellation
if self.isCanceled():
return False
# simulate exceptions to show how to abort task
if random.randint(0, 500) == 42:
# DO NOT raise Exception('bad value!')
# this would crash QGIS
self.exception = Exception('bad value!')
return False
return True
def finished(self, result):
"""This method is automatically called when self.run returns. result
is the return value from self.run.
This function is automatically called when the task has completed (
successfully or otherwise). You just implement finished() to do whatever
follow up stuff should happen after the task is complete. finished is
always called from the main thread, so it's safe to do GUI
operations and raise Python exceptions here.
"""
if result:
QgsMessageLog.logMessage(
'Task "{name}" completed\n' \
'Total: {total} ( with {iterations} iterations)'.format(
name=self.description(),
total=self.total,
iterations=self.iterations),
MESSAGE_CATEGORY, Qgis.Success)
else:
if self.exception is None:
QgsMessageLog.logMessage(
'Task "{name}" not successful but without exception '\
'(probably the task was manually canceled by the user)'.format(
name=self.description()),
MESSAGE_CATEGORY, Qgis.Warning)
else:
QgsMessageLog.logMessage(
'Task "{name}" Exception: {exception}'.format(
name=self.description(), exception=self.exception),
MESSAGE_CATEGORY, Qgis.Critical)
raise self.exception
def cancel(self):
QgsMessageLog.logMessage(
'Task "{name}" was cancelled'.format(name=self.description()),
MESSAGE_CATEGORY, Qgis.Info)
super().cancel()
t1 = MyTask('waste cpu long', 10)
t2 = MyTask('waste cpu short', 6)
t3 = MyTask('waste cpu mini', 4)
st1 = MyTask('waste cpu Subtask 1', 5)
st2 = MyTask('waste cpu Subtask 2', 2)
st3 = MyTask('waste cpu Subtask 3', 4)
t2.addSubTask(st1, [t3, t1])
t1.addSubTask(st2)
t1.addSubTask(st3)
QgsApplication.taskManager().addTask(t1)
QgsApplication.taskManager().addTask(t2)
QgsApplication.taskManager().addTask(t3)

NEVER, EVER, EVER use print in the QgsTask outside from finished(). finished() is called on the main event loop

class MyTask(QgsTask):
def __init__(self, description, flags):
super().__init__(description, flags)
def run(self):
QgsMessageLog.logMessage('Started task {}'.format(self.description()))
#print('crashandburn')
return True
t1 = MyTask('waste cpu', QgsTask.CanCancel)
QgsApplication.taskManager().addTask(t1)

Call a Processing algorithm in a separate thread

You can simply execute a processing algorithm in a separate thread thanks to QgsProcessingAlgRunnerTask. This class takes a processing algorithm, its parameters, a context and a feedback objects and execute the algorithm. QgsProcessingAlgRunnerTask offers an executed signal to which you can connect and execute further code. executed sends two arguments bool successful and dict results. If you want to retrieve a memory layer you can pass the context as well by using partial or lambda.
If you’re wondering what parameter values you need to specify for an algorithm, and what values are acceptable, try running processing.algorithmHelp('qgis:randompointsinextent') in the python console. In QGIS 3.2 you’ll get a detailed list of all the parameter options for the algorithm and a summary of acceptable value types and formats for each. Another nice possibility is to run the algorithm from the gui and check the history after.

from functools import partial
from qgis.core import (QgsTaskManager, QgsMessageLog, QgsProcessingAlgRunnerTask,
QgsApplication, QgsProcessingContext, QgsProcessingFeedback, QgsProject)
MESSAGE_CATEGORY = 'My processing tasks'
def task_finished(context, successful, results):
if not successful:
QgsMessageLog.logMessage('Task finished unsucessfully', MESSAGE_CATEGORY, Qgis.Warning)
output_layer = context.getMapLayer(results['OUTPUT'])
# because getMapLayer doesn't transfer ownership the layer will be
# deleted when context goes out of scope and you'll get a crash.
# takeMapLayer transfers ownership so it's then safe to add it to the
# project and give the project ownership.
if output_layer.isValid():
QgsProject.instance().addMapLayer(
context.takeResultLayer(output_layer.id()))
alg = QgsApplication.processingRegistry().algorithmById(
'qgis:randompointsinextent')
context = QgsProcessingContext()
feedback = QgsProcessingFeedback()
params = {
'EXTENT': '4.63,11.57,44.41,48.78 [EPSG:4326]',
'MIN_DISTANCE': 0.1,
'POINTS_NUMBER': 100,
'TARGET_CRS': 'EPSG:4326',
'OUTPUT': 'memory:My random points'
}
task = QgsProcessingAlgRunnerTask(alg, params, context, feedback)
task.executed.connect(partial(task_finished, context))
QgsApplication.taskManager().addTask(task)

I hope this post can help you porting your plugins to QGIS3 and again if you need professional help for your plugins, don’t hesitate to contact us.

Porting QGIS plugins to API v3 – Strategy and tools

The Release of QGIS 3.0 was a great success and with the first LTR (3.4) scheduled for release this fall, it is now the perfect time to port your plugins to the new API.
QGIS 3.0 is the first major release since September 2013 when QGIS 2.0 was released. During the release cycles of all 2.x releases, the QGIS Python API remained stable. This means that a plugin or script meant to be used in QGIS 2.0 is still working in QGIS 2.18.
The need for a new major release was principally motivated by the update to newer core libraries such as Qt 5 and Python 3. But it also offered a unique opportunity to the development team to tackle long-standing issues and limitations which could not be fixed during the 2.x life cycle. Inevitably, this introduced multiple backward incompatibilities making scripts and plugins unusable in QGIS 3.
In this post, I’d like to share some notes from my latest ports. Obviously, if you need professional help for porting your plugins, don’t hesitate to contact us.

Step 0 – Unit tests

You should already have your code covered by unit tests, but I know, the world is not perfect and at times we have to cut edges and, unfortunately, often unit tests are the ones getting cut.
Porting to a new API version is a great moment to go write unit tests helping to make sure that your plugin will keep on working as before the port.

Step 1 – fix * imports

Before going on, please go and remove all your * imports (like from PyQt4.QtGui import *). They are bad and qgis2to3 cannot handle them. There is no need to already change them to the PyQ5 version, just remove them and add the propper PyQt4 imports. We’ll handle moving to PyQt5 in a later step.

From PEP8: Wildcard imports (from import *) should be avoided, as they make it unclear which names are present in the namespace, confusing both readers and many automated tools.

Step 2 – Versioning strategy

Since having a source code repository is a mandatory requirement for publishing a plugin on plugins.qgis.org, I assume you already know what code versioning is and why you absolutely should be using it.

APIv2 branch

Unless you absolutely want to make your code run on both API 2 and 3 (which might be possible) I strongly suggest to create a branch or your current version called qgis2, API2 or legacy or whatever you want to call it. From now on this branch will be responsible for all your future (probably mainly bugfixes) releases for the 2.x series of QGIS. Remember to edit the metadata.txt file and add your minimum and maximum version (not mandatory but nice for clarity):

qgisMinimumVersion=2.14
qgisMaximumVersion=2.18

Master branch

From now on your master branch will be where all your future development for the 3.x series will happen. Remember to edit the metadata.txt file and add your minimum version:

qgisMinimumVersion=3.0

Step 3 – install the helpers

We created a repository with two dedicated tools to help you migrate your QGIS 2 plugins to QGIS 3: qgis2to3 and qgis2apifinder. Both tools are distributed as a single Python package installable via

pip install qgis2to3

Please note that often for system-wide installation you need sudo.
All the sources and more information can be found at https://github.com/opengisch/qgis_2to3

Step 4 – Python 2 to Python 3 and PyQt4 to PyQt5

The qgis2to3 tool is a copy of the files found in QGIS scripts to allow for quick downloading and simple installation without the need of downloading the whole QGIS repository. This is a set of fixers for the python 2to3 command that will update your Python 2 code to Python 3. The additional fixers will also take care of the PyQt4 to PyQt5 porting as well as some other things.
Running the qgis2to3 command will show a number of changes required. These changes can be applied with -w flag

qgis2to3 -w /path/to/your/plugin

Step 5 – Check for API v2 usages

The qgisapi2finder tool helps you find usages of the QGIS API version 2 and gives hints about potential required changes for API version 3.
It is based on a machine parsing of https://qgis.org/api/api_break.html so the results are as good as the information there.
Also, being a simple text parser, it just gives a hint where to look at. It is by no means a complete tool to find all the possible API incompatibility.
Methods are matched using only their names and not their classes, so there might be various false positives. Also, if the same keyword has been edited in various classes, qgisapi2finder will show you all the available suggestions for that keyword.
You can run qgis2apifinder to get hints on the existence of obsolete code requiring manual porting and suggestions on how to actually deal with it. Please note that qgis2apifinder does hide some very frequent words like [‘layout’, ‘layer’, ‘fields’] from the analysis. You can show those with the --all flag.

qgis2apifinder --all /path/to/plugin
qgis2apifinder --all /path/to/plugin/file.py

Step 6 – update your code

From here on it is all looking at each hint, updating the code and rerunning your tests. A properly configured IDE (stay tuned) could also help in the process.
Some more information can be found at github.com/qgis/QGIS/wiki/Plugin-migration-to-QGIS-3
Also, take a look at the PyQGIS API documentation now online at python.qgis.org/master.
I hope this post and tool can help you porting your plugins to QGIS3 and again if you need professional help for porting your plugins, don’t hesitate to contact us.

QGIS2 compatibility plugin

Lately I’ve been spending time porting a bigger plugin from QGIS 2.8 to 3 while maintaining 2.8 compatibility. You can find it at https://github.com/opengisch/qgis2compat/ and http://plugins.qgis.org/plugins/qgis2compat/ One code to rule them all. My target was to have to edit the

Updating PyQt signals that use lambda in QGIS with 2to3

Just for the sake of documenting things, when running qgis 2to3 on a plugin I encountered a tricky situation regarding signals. [crayon-5b8f49bf38bed549859272/] The original code: [crayon-5b8f49bf38bfd417730445/] The generated code: [crayon-5b8f49bf38c03910524058/] so in do_load_project we get False instead of “my test

PSA: Please use new style Qt signals and slots not the old style

Don’t do this:

self.connect(self.widget, 
             SIGNAL("valueChanged(int)"), 
             self.valuechanged)

It’s the old way, the crappy way. It’s prone to error and typing mistakes. And who really wants to be typing strings as functions and arg names in it. Gross.

Do this:

self.widget.valueChanged.connect(self.valuechanged)
self.widget.valueChanged[str].connect(self.valuechanged)

Much nicer. Cleaner. Looks and feels like Python not some mash up between C++ and Python. The int argument is the default so it will use that. If you to pick the signal type you can use [type].

Don’t do this:

self.emit(SIGNAL("changed()", value1, value2))

Do this

class MyType(QObject):
   changed = pyqtSignal(str, int)

   def stuff(self):
       self.changed.emit(value1, value2)

pyqtSignal is a type you can use to define you signal. It will come with type checking, if you don’t want type checking just do pyqtSignal(object).

Please think of the poor kittens before using the old style in your code.


Filed under: pyqt, python, qgis Tagged: pyqt, qgis, qt

  • Page 1 of 1 ( 5 posts )
  • pyqt

Back to Top

Sponsors