Crowdfunding: QGIS 3D for Open Source Digital Twins

Tag: c++

Software quality in QGIS

According to the definition of software quality given by french Wikipedia

An overall assessment of quality takes into account external factors, directly observable by the user, as well as internal factors, observable by engineers during code reviews or maintenance work.

I have chosen in this article to only talk about the latter. The quality of software and more precisely QGIS is therefore not limited to what is described here. There is still much to say about:

  • Taking user feedback into account,
  • the documentation writing process,
  • translation management,
  • interoperability through the implementation of standards,
  • the extensibility using API,
  • the reversibility and resilience of the open source model…

These are subjects that we care a lot and deserve their own article.

I will focus here on the following issue: QGIS is free software and allows anyone with the necessary skills to modify the software. But how can we ensure that the multiple proposals for modifications to the software contribute to its improvement and do not harm its future maintenance?

Self-discipline

All developers contributing to QGIS code doesn’t belong to the same organization. They don’t all live in the same country, don’t necessarily have the same culture and don’t necessarily share the same interests or ambitions for the software. However, they share the awareness of modifying a common good and the desire to take care of it.

This awareness transcends professional awareness, the developer not only has a responsibility towards his employer, but also towards the entire community of users and contributors to the software.

This self-discipline is the foundation of the quality of the contributions of software like QGIS.

However, to err is human and it is essential to carry out checks for each modification proposal.

Automatic checks

With each modification proposal (called Pull Request or Merge Request), the QGIS GitHub platform automatically launches a set of automatic checks.

Example of proposed modification

Result of automatic checks on a modification proposal

The first of these checks is to build QGIS on the different systems on which it is distributed (Linux, Windows, MacOS) by integrating the proposed modification. It is inconceivable to integrate a modification that would prevent the application from being built on one of these systems.

The tests

The first problem posed by a proposed modification is the following “How can we be sure that what is going to be introduced does not break what already exists?”

To validate this assertion, we rely on automatic tests. This is a set of micro-programs called tests, which only purpose is to validate that part of the application behaves as expected. For example, there is a test which validates that when the user adds an entry in a data layer, then this entry is then present in the data layer. If a modification were to break this behavior, then the test would fail and the proposal would be rejected (or more likely corrected).

This makes it possible in particular to avoid regressions (they are very often called non-regression tests) and also to qualify the expected behavior.

There are approximately 1.3 Million lines of code for the QGIS application and 420K lines of test code, a ratio of 1 to 3. The presence of tests is mandatory for adding functionality, therefore the quantity of test code increases with the quantity of application code.

In blue the number of lines of code in QGIS, in red the number of lines of tests

There are currently over 900 groups of automatic tests in QGIS, most of which run in less than 2 seconds, for a total execution time of around 30 minutes.

We also see that certain parts of the QGIS code – the most recent – are better covered by the tests than other older ones. Developers are gradually working to improve this situation to reduce technical debt.

Code checks

Analogous to using a spell checker when writing a document, we carry out a set of quality checks on the source code. We check, for example, that the proposed modification does not contain misspelled words or “banned” words, that the API documentation has been correctly written or that the modified code respects certain formal rules of the programming language.

We recently had the opportunity to add a check based on the clang-tidy tool. The latter relies on the Clang compiler. It is capable of detecting programming errors by carrying out a static analysis of the code.

Clang-tidy is, for example, capable of detecting “narrowing conversions”.

Example of detecting “narrowing conversions”

In the example above, Clang-tidy detects that there has been a “narrowing conversion” and that the value of the port used in the network proxy configuration “may” be corrupted. In this case, this problem was reported on the QGIS issues platform and had to be corrected.

At that time, clang-tidy was not in place. Its use would have made it possible to avoid this anomaly and all the steps which led to its correction (exhaustive description of the issue, multiple exchanges to be able to reproduce it, investigation, correction, review of the modification), meaning a significant amount of human time which could thus have been avoided.

Peer review

A proposed modification that would validate all of the automatic checks described above would not necessarily be integrated into the QGIS code automatically. In fact, its code may be poorly designed or the modification poorly thought out. The relevance of the functionality may be doubtful, or duplicated with another. The integration of the modification would therefore potentially cause a burden for the people in charge of the corrective or evolutionary maintenance of the software.

It is therefore essential to include a human review in the process of accepting a modification.

This is more of a rereading of the substance of the proposal than of the form. For the latter, we favor the automatic checks described above in order to simplify the review process.

Therefore, human proofreading takes time, and this effort is growing with the quantity of modifications proposed in the QGIS code. The question of its funding arises, and discussions are in progress. The QGIS.org association notably dedicates a significant part of its budget to fund code reviews.

More than 100 modification proposals were reviewed and integrated during the month of December 2023. More than 30 different people contributed. More than 2000 files have been modified.

Therefore the wait for a proofreading can sometimes be long. It is also often the moment when disagreements are expressed. It is therefore a phase which can prove frustrating for contributors, but it is an important and rich moment in the community life of a free project.

To be continued !

As a core QGIS developer, and as a pure player OpenSource company, we believe it is fundamental to be involved in each step of the contribution process.

We are investing in the review process, improving automatic checks, and in the QGIS quality process in general. And we will continue to invest in these topics in order to help make QGIS a long-lasting and stable software.

If you would like to contribute or simply learn more about QGIS, do not hesitate to contact us at infos+qgis@oslandia.com and consult our QGIS support proposal.

Learn More

(Fr) Oslandia recrute : ingénieur(e) développement C++ / Python – OSL2011B

Sorry, this entry is only available in French.

Learn More

Build and deploy c++ QGIS custom application on windows

After a lot of troubles, I managed to compile and deploy a QGIS c++ app on windows. This small guide will describe the steps I followed. This has been tested on win xp and windows 7, both in 32 bits.

Development environment

Your app must be built using MSVC 9.0 (2008) since QGIS in OSGeo’s package was built with it. Hence, MinGW cannot be used.

  1. Install Microsoft Visual Studio Express 2008.
  2. Install QGIS and Qt libs using OSGeo4W installer
  3. Install Qt Creator
  4. If you want a debugger,you should install CDB. This can be achieved by installing Windows SDK environment. In the installation process, only select Debugging toos for windows.

I wasn’t able to use the compiler yet, so I am not 100% sure about 4.

Now, if you want to build using Qt Creator, it must be started in a proper environment. Adapt this batch to launch Qt Creator:

ECHO Setting up QGIS DEV ENV

set PYTHONPATH=

set OSGEO4W_ROOT=C:\OSGeo4W
call "%OSGEO4W_ROOT%\bin\o4w_env.bat"

@set QMAKESPEC=win32-msvc2008
@set PATH=%OSGEO4W_ROOT%\bin;%OSGEO4W_ROOT%\apps\qgis-dev\bin;%PATH%

@set INCLUDE=%INCLUDE%;%OSGEO4W_ROOT%\include;%OSGEO4W_ROOT%\apps\qgis-dev\include
@set LIB=%LIB%;%OSGEO4W_ROOT%\lib;%OSGEO4W_ROOT%\apps\qgis-dev\lib

path %OSGEO4W_ROOT%\bin;%SYSTEMROOT%\System32;%SYSTEMROOT%;%SYSTEMROOT%\System32\wbem;C:\Progra~1\Git\bin;C:\Qt\qtcreator-3.0.1\bin;%PATH%

set VS90COMNTOOLS=C:\Program Files\Microsoft Visual Studio 9.0\Common7\Tools\
call "C:\Program Files\Microsoft Visual Studio 9.0\VC\vcvarsall.bat" x86

start "Qt Creator" /B C:\Qt\qtcreator-3.0.1\bin\qtcreator.exe %*

Then, you need to configure a proper kit in Qt Creator.

  1. Go to Options -> Build & Run -> Compilers and check that Microsoft Visual C++ Compiler 9.0 is correctly detected.
  2. Then in Qt Versions tab, add Qt from the OSGeO installation, normally c:\OSGeo4W\bin\qmake.exe
  3. In Debuggers tab, add cdb.exe found in c:\Debugging tools for windows\ 
  4. Finally, check in Kits that it is properly configured.

Building the application

This is what looks like an application project file.

QT += core gui xml
greaterThan(QT_MAJOR_VERSION, 4): QT += widgets
TARGET = hfp
TEMPLATE = app
SOURCES += YOURSOURCES
HEADERS +=YOUR HEADERS
FORMS += YOUR FORMS
RESOURCES += images/images.qrc

win32:CONFIG(Release, Debug|Release) {
 LIBS += -L"C:/OSGeo4W/lib/" -lQtCore4
 LIBS += -L"C:/OSGeo4W/lib/" -lQtGui4
 LIBS += -L"C:/OSGeo4W/lib/" -lQtXml4
 LIBS += -L"C:/OSGeo4W/apps/qgis-dev/lib/" -lqgis_core
 LIBS += -L"C:/OSGeo4W/apps/qgis-dev/lib/" -lqgis_gui
}
else:win32:CONFIG(Debug, Debug|Release) {
 PRE_TARGETDEPS += C:/OSGeo4W/lib/QtCored4.lib
 PRE_TARGETDEPS += C:/OSGeo4W/lib/QtGuid4.lib
 PRE_TARGETDEPS += C:/OSGeo4W/lib/QtXmld4.lib
 LIBS += -L"C:/OSGeo4W/lib/" -lQtCored4
 LIBS += -L"C:/OSGeo4W/lib/" -lQtGuid4
 LIBS += -L"C:/OSGeo4W/lib/" -lQtXmld4
 LIBS += -L"C:/OSGeo4W/apps/qgis-dev/lib/" -lqgis_core
 LIBS += -L"C:/OSGeo4W/apps/qgis-dev/lib/" -lqgis_gui
}
win32:{
 INCLUDEPATH += C:/OSGeo4W/include
 DEPENDPATH += C:/OSGeo4W/include
 INCLUDEPATH += C:/OSGeo4W/apps/qgis-dev/include
 DEPENDPATH += C:/OSGeo4W/apps/qgis-dev/include
 DEFINES += GUI_EXPORT=__declspec(dllimport) CORE_EXPORT=__declspec(dllimport)
}
unix {
 LIBS += -L/usr/local/lib/ -lqgis_core -lqgis_gui
 LIBS += -L/usr/local/lib/qgis/plugins/ -lgdalprovider
 INCLUDEPATH += /usr/local/include/qgis
 DEFINES += GUI_EXPORT= CORE_EXPORT=
}

Remarks

  • GUI_EXPORT and CORE_EXPORT must be set to __declspec(dllimport). I don’t know exactly what it means, but I found out reading this thread, with some hazardous tries. If you don’t set these, you won’t be able to call any variable defined as extern in QGIS (e.g. cursors).
  • Qt release libraries shall not be mixed up with debug config in your project. In other words, use release libs for release mode and debug libs for debug mode.

With this, you should be able to compile your QGIS application in Qt Creator!

You can find some coding examples on github which are a bit old but still useful to start.

Now, to get the whole potential of QGIS libs, you must initialize the QgsApplication in your main window class:

#if defined(Q_WS_WIN)
  QString pluginPath = "c:\\OSGeo4W\\apps\\qgis-dev\\plugins";
  QString prefixPath = "c:\\OSGeo4W\\apps\\qgis-dev\\";
#else
  QString pluginPath = "/usr/local/lib/qgis/plugins/";
  QString prefixPath = "/usr/local";
#endif

  QgsApplication::setPluginPath( pluginPath );
  QgsApplication::setPrefixPath( prefixPath, true);
  QgsApplication::initQgis();

Deploying on windows

Since QGIS is not to be installed on the target computer, the built app will not be able to find the path declared in previous code.
There is probably a better approach, but here is a way to solve this:
Change the path to

  QString pluginPath = "c:\\myapp\\qgis\plugins";
  QString prefixPath = "c:\\myapp\\qgis";

This means you must deploy the app to this exact location: c:\myapp. In this directory, you need to create a qgis folder in which you will copy c:\OSGeo4W\apps\qgis-dev\resources and c:\OSGeo4W\apps\qgis-dev\plugins.

Besides, this you will need to copy some DLLs to be able to run the applications. You might want to use the dependency walker to find which are needed.

The batch file hereafter creates a folder on the building machine that will contain all the needed files in my case (it might be different in your case).

rmdir c:\myapp /Q /S
mkdir c:\myapp
mkdir c:\myapp\iconengines
mkdir c:\myapp\qgis
mkdir c:\myapp\qgis\resources
mkdir c:\myapp\qgis\plugins

copy PATHTOMYAPP\build-myapp-Desktop-Release\release\myapp.exe c:\myapp\
copy c:\OSGeo4W\bin\QtCore4.dll c:\myapp\
copy c:\OSGeo4W\bin\QtGui4.dll c:\myapp\
copy c:\OSGeo4W\bin\QtXml4.dll c:\myapp\
copy c:\OSGeo4W\bin\QtNetwork4.dll c:\myapp\
copy c:\OSGeo4W\bin\QtSvg4.dll c:\myapp\
copy c:\OSGeo4W\bin\QtWebKit4.dll c:\myapp\

copy c:\OSGeo4W\bin\zlib_osgeo.dll c:\myapp\
copy c:\OSGeo4W\bin\msvcr71.dll c:\myapp\
copy c:\OSGeo4W\bin\phonon4.dll c:\myapp\
copy c:\OSGeo4W\bin\proj.dll c:\myapp\
copy c:\OSGeo4W\bin\geos_c.dll c:\myapp\
copy c:\OSGeo4W\bin\gdal110.dll c:\myapp\
copy c:\OSGeo4W\bin\ogdi_32b1.dll c:\myapp\
copy c:\OSGeo4W\bin\libexpat.dll c:\myapp\
copy c:\OSGeo4W\bin\xerces-c_3_1.dll c:\myapp\
copy c:\OSGeo4W\bin\LIBPQ.dll c:\myapp\
copy c:\OSGeo4W\bin\SSLEAY32.dll c:\myapp\
copy c:\OSGeo4W\bin\LIBEAY32.dll c:\myapp\
copy c:\OSGeo4W\bin\krb5_32.dll c:\myapp\
copy c:\OSGeo4W\bin\comerr32.dll c:\myapp\
copy c:\OSGeo4W\bin\k5sprt32.dll c:\myapp\
copy c:\OSGeo4W\bin\gssapi32.dll c:\myapp\
copy c:\OSGeo4W\bin\hdf_fw.dll c:\myapp\
copy c:\OSGeo4W\bin\mfhdf_fw.dll c:\myapp\
copy c:\OSGeo4W\bin\jpeg_osgeo.dll c:\myapp\
copy c:\OSGeo4W\bin\jpeg12_osgeo.dll c:\myapp\
copy c:\OSGeo4W\bin\netcdf.dll c:\myapp\
copy c:\OSGeo4W\bin\geotiff.dll c:\myapp\
copy c:\OSGeo4W\bin\libtiff.dll c:\myapp\
copy c:\OSGeo4W\bin\sqlite3.dll c:\myapp\
copy c:\OSGeo4W\bin\spatialite4.dll c:\myapp\
copy c:\OSGeo4W\bin\freexl.dll c:\myapp\
copy c:\OSGeo4W\bin\iconv.dll c:\myapp\
copy c:\OSGeo4W\bin\libxml2.dll c:\myapp\
copy c:\OSGeo4W\bin\LIBMYSQL.dll c:\myapp\
copy c:\OSGeo4W\bin\hdf5.dll c:\myapp\
copy c:\OSGeo4W\bin\szip.dll c:\myapp\
copy c:\OSGeo4W\bin\libcurl.dll c:\myapp\
copy c:\OSGeo4W\bin\zlib1.dll c:\myapp\
copy c:\OSGeo4W\bin\openjp2.dll c:\myapp\
copy c:\OSGeo4W\bin\spatialindex1.dll c:\myapp\
copy c:\OSGeo4W\bin\qwt5.dll c:\myapp\

copy c:\OSGeo4W\apps\qt4\plugins\iconengines\qsvgicon4.dll c:\myapp\iconengines\

copy C:\Progra~1\Git\bin\libiconv-2.dll c:\myapp\
copy C:\Progra~1\Git\bin\libintl-8.dll c:\myapp\

copy c:\OSGeo4W\apps\qgis-dev\bin\qgis_Core.dll c:\myapp\
copy c:\OSGeo4W\apps\qgis-dev\bin\qgis_gui.dll c:\myapp\
copy c:\OSGeo4W\apps\qgis-dev\bin\msvcp90.dll c:\myapp\

copy c:\windows\system32\msvcp100.dll c:\myapp\
copy c:\windows\system32\msvcr100.dll c:\myapp\

copy C:\OSGeo4W\apps\qgis-dev\resources\* c:\myapp\qgis\resources
copy C:\OSGeo4W\apps\qgis-dev\plugins\* c:\myapp\qgis\plugins

To be able to run the app, on a fresh windows XP, I had to install:

And copy the whole folder c:\myapp from the building machine to the target machine.

It seems that from Vista, the 2005 redistributable package is included. So, no need to install it.

And voilà!

Learn More