QGIS Planet Tags
Just a few additional notes about how to build latest QGIS 3 master on Win 10 64 bits, the base document is: INSTALL
The environent configuration of the kit is:
<code>PATH=C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow;C:\Program Files (x86)\MSBuild\14.0\bin\amd64;C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\BIN\amd64;C:\WINDOWS\Microsoft.NET\Framework64\v4.0.30319;C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\VCPackages;C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE;C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\Tools;C:\Program Files (x86)\HTML Help Workshop;C:\Program Files (x86)\Microsoft Visual Studio 14.0\Team Tools\Performance Tools\x64;C:\Program Files (x86)\Microsoft Visual Studio 14.0\Team Tools\Performance Tools;C:\Program Files (x86)\Windows Kits\8.1\bin\x64;C:\Program Files (x86)\Windows Kits\8.1\bin\x86;C:\Program Files (x86)\Microsoft SDKs\Windows\v10.0A\bin\NETFX 4.6.1 Tools\x64\;C:\OSGEO4~1\apps\qt5\bin;C:\OSGEO4~1\apps\Python37;C:\OSGEO4~1\apps\Python37\Scripts;C:\OSGEO4~1\bin;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\system32\WBem;C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\bin;C:\Program Files\CMake\bin;c:\cygwin64\bin;C:\OSGeo4W64\apps\Python37\Scripts PYTHONHOME=C:\OSGEO4~1\apps\Python37 QT_QPA_PLATFORM_PLUGIN_PATH=C:\OSGeo4W64\apps\Qt5\plugins VCARCH=amd64 VCINSTALLDIR=C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\ VISUALSTUDIOVERSION=14.0 VS110COMNTOOLS=C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\Tools\ VS120COMNTOOLS=C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\Tools\ VS140COMNTOOLS=C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\Tools\ VSINSTALLDIR=C:\Program Files (x86)\Microsoft Visual Studio 14.0\ WINDIR=C:\WINDOWS WINDOWSLIBPATH=C:\Program Files (x86)\Windows Kits\8.1\References\CommonConfiguration\Neutral WINDOWSSDKDIR=C:\Program Files (x86)\Windows Kits\8.1\ WINDOWSSDKLIBVERSION=winv6.3\ WINDOWSSDKVERSION=\ WINDOWSSDK_EXECUTABLEPATH_X64=C:\Program Files (x86)\Microsoft SDKs\Windows\v10.0A\bin\NETFX 4.6.1 Tools\x64\ WINDOWSSDK_EXECUTABLEPATH_X86=C:\Program Files (x86)\Microsoft SDKs\Windows\v10.0A\bin\NETFX 4.6.1 Tools\</code>
Note: I’m not really sure if all those variables are necessary, but that worked for me.
The goal of the new API is twofold:
The new API is documented in https://qgis.org/api/classQgsAbstractProviderConnection.html and it provides a few specializations for DB connections (https://qgis.org/api/classQgsAbstractDatabaseProviderConnection.html) and an initial PR implementation for web service-based connections (https://github.com/qgis/QGIS/pull/33045).
While the whole of the desired refactoring work was too big for a single grant request, the first work package has been completed and the following data providers have been partially or totally refactored to make use of the new connections API:
The new API was also used to implement the automatic loading of layer dependencies (not part of the grant program).
For developers interested in working with the new API, a set Python tests are available to show how to use the methods:Â https://github.com/qgis/QGIS/blob/master/tests/src/python/test_qgsproviderconnection_ogr_gpkg.py (see also the postgres and spatialite companion tests).
There is still a large amount of work to be done in order to complete all the desired refactoring and to remove all the Python and C++ code that will be ultimately be made redundant. In particular, future work should be undertaken to:
For further information, the following paragraphs (taken from the original grant proposal) will provide full details about the background of this work.
We propose to start refactoring the DB-manager plugin functionality into core C++ implementation, reusing existing core API and replacing redundant duplicate functionality.
The clear advantages are:
The above mentioned work is too large to be completed within a single grant, so what we propose here is to start the refactoring needed in order to have a core stable C++ API that can be used by the application and the plugins and that will be available to fully move DB manager to C++ API in the future avoiding duplication of code and functionality.
The API will be exposed through the browser and it will be used by the DB manager instead of the Python implementation that is currently used.
From https://en.wikipedia.org/wiki/OpenCL:
OpenCL (Open Computing Language) is a framework for writing programs that execute across heterogeneous platforms consisting of central processing units (CPUs), graphics processing units (GPUs), digital signal processors (DSPs), field-programmable gate arrays (FPGAs) and other processors or hardware accelerators. OpenCL specifies programming languages (based on C99 and C++11) for programming these devices and application programming interfaces (APIs) to control the platform and execute programs on the compute devices. OpenCL provides a standard interface for parallel computing using task- and data-based parallelism.
Basically, you write a program and you execute it on a GPU (or, less frequently, on a CPU or on a DSP) taking advantage of the huge parallel programming capabilities of the modern graphic cards.
Depending on many different factors, the speed gain can vary to a great extent, but it is typically around one order of magnitude.
The work I’ve done consisted in integrating OpenCL support into QGIS and writing all the utilities to load, build and run OpenCL programs.
For now, I’ve ported the following QGIS core algorithms, all of them are availabe in processing:
Since the framework to support OpenCL is now in place, I think that more algorithms will be ported over the time.
During this development, even if was not in scope, the hillshade renderer has been optimized for speed and it can also benefit of OpenCL acceleration.
OpenCL support is optional and opt-in, to use it, you need to activate it into the QGIS options dialog like shown in the screenshot below:
Well, YMMV, but here are some figures for a big DEM raster, low values mean faster execution.
GDAL means CPU execution using the GDAL processing algorithm.
Of course it depends on your specific hardware and on your O.S., AMD, NVidia and Intel have different distributions channels, in general the driver for your graphic card will also provide the OpenCL driver, if your GPU is compatible, if OpenCL is not available on your current machine, try to Google for OpenCL, your O.S. and graphic card.
If there is no OpenCL support for your graphic card, you might try to install a driver for your GPU (Intel for example provides them) and you will probably have a decent acceleration even if not as much as you can get on a real graphic card.
This fact worths some more explanation: you might ask your self why running and algorithm directly on the CPU and running it on the same CPU but using OpenCL would make any difference and the reason why it is generally faster by using OpenCL is that OpenCL will run the algorithm in parallel on all cores of your CPU, while a normal application (and QGIS does not make an exception here) will use a single core.
Just a quick note: you’ll need to install the OpenCL headers and the ICD library:
sudo apt-get install opencl-headers ocl-icd-opencl-dev
I started this work as a proof of concept in my spare time (that it is not much, lately) and when I realized that it was promising, I submitted a QGIS grant proposal in order to allocate some working time to port more algorithms, write tests and polish the implementation.
This work would not be possible without all the generous sponsors and donors that feed the QGIS grant program year after year, many thanks to the QGIS community for this amazing support!
Jürgen Fischer was as usual very helpful and took care of the windows builds, now available in OSGeo4W packages.
Nyall Dawson helped with the code review and with testing the implementation on different cards and machines.
Matthias Kuhn reviewed the code.
Even Rouault pointed me to some highly efficient GDAL algorithm optimizations that I’ve been able to integrate in QGIS.
My main reasons for having Python data provider were:
This topic has been floating in my head for a while since I decided to give it a second look and I finally implemented it and merged for the next 3.2 release.
To make this possible I had to:
First, let me say that it wasn’t like a walk in the park: the Python bindings part is always like diving into woodoo and black magic recipes before I can get it to work properly.
For the Python provider sample implementation I decided to re-implement the memory (aka: scratch layers) provider because that’s one of the simplest providers and it does not depend on any external storage or backend.
For now, the main source of information is the API and the tests:
To register your own provider (PyProvider
in the snippet below) these are the basic steps:
metadata = QgsProviderMetadata(PyProvider.providerKey(), PyProvider.description(), PyProvider.createProvider) QgsProviderRegistry.instance().registerProvider(metadata)
To create your own provider you will need at least the following components:
QgsVectorDataProvider
)QgsAbstractFeatureSource
)QgsAbstractFeatureIterator
)Be aware that the implementation of a data provider is not easy and you will need to write a lot of code, but at least you could get some inspiration from the existing example.
Enjoy wirting data providers in Python and please let me know if you’ve fond this implementation useful!
Recently I was invited by the colleagues from OpenGIS.ch to lend a hand in a training session about QGIS server.
This was a good opportunity to update my presentation for QGIS3, to fix a few bugs and to explore the powerful capabilities of QGIS server and Python.
As a result, I published the full recipe of a Vagrant VM on github: https://github.com/elpaso/qgis3-server-vagrant
The presentation is online here: http://www.itopen.it/bulk/qgis3-server/
What’s worth mentioning is the sample plugins (I’ll eventually package and upload them to the official plugin site):
The VM uses 4 different (although similar) deployment strategies:
Have fun with QGIS server: it was completely refactored in QGIS 3 and it’s now better than ever!
# Assuming your subnet is 192.168.9 nmap -p 6000 192.168.1.* Nmap scan report for android-8899989888d02271.homenet.telecomitalia.it (192.168.99.50) Host is up (0.0093s latency). PORT STATE SERVICE 6000/tcp open X11Now, in QGIS you can open the plugin dialog through Vector -> GPS -> GPS Tools and enter the IP address and port of your GPS device:
/home/user/Qt/5.9.3/Src/
To build the sources, you can change into that directory and issue the following command – I assume that you have already installed all the dependencies normally needed to build C++ Qt programs – I’m using clang here but feel free to choose gcc, we are going to install the new Qt build into /opt/qt593
../configure -prefix /opt/qt593 -debug -opensource -confirm-license -ccache -platform linux-clangWhen done, you can build it with
make -j9 sudo make installTo build QGIS you also need three additional Qt packages QtWebKit from https://github.com/qt/qtwebkit (you can just download the zip):
/opt/qt593/bin/qmake WebKit.pro make -j9 sudo make installSame with QScintila2 from https://www.riverbankcomputing.com/software/qscintilla
/opt/qt593/bin/qmake qscintilla.pro make -j9 sudo make installQWT is also needed and it can be downloaded from https://sourceforge.net/projects/qwt/files/qwt/6.1.3/ but it requires a small edit in
qwtconfig.pri
before you can build it: set QWT_INSTALL_PREFIX = /opt/qt593_libs/qwt-6.1.3
to install it in a different folder than the default one (that would possibly overwrite a system install of QWT).
The build it with:/opt/qt593/bin/qmake qwt.pro make -j9 sudo make installIf everything went fine, you can now configure Qt Creator to use this new debug build of Qt: start with creating a new kit (you can probably clone a working Qt5 kit if you have one). What you need to change is the Qt version (the path to cmake) to point to your brand new Qt build,: