Page 1 of 40 (795 posts)

  • talks about »
  • qgis

Tags

Last update:
Fri Jul 3 09:35:11 2020

A Django site.

QGIS Planet

Versie 3.5.0 van de PDOKServicesPlugin

OK, this is a short post to let the dutchies know there is a new version 3.5.0 of the PDOKServicesPlugin to load some of our national data as WMS, WFS or WCS.
For example: the GeoMorphological map of The Netherlands, or the data of the accidents that happened from 2008 till 2018. Here combined in one scary map of my surroundings:
screenshot-20200619162849-1086x634
And… did you hear… “Pi is coming ….” (sinister Game of Thrones sound in the background…)
pi
Give the ‘packagers’ some time though …

(Fr) Financement mutualisé du logiciel libre: le cas QGIS

Sorry, this entry is only available in French.

Securely accessing enterprise ArcGIS Portal sites through QGIS

We were recently contacted for advice regarding our recommendations for securely accessing content on an enterprise ArcGIS Portal deployment from within QGIS. Fortunately, this setup is fully supported and works seamlessly in QGIS, thanks to the native integration of “OAuth2” authentication in QGIS!

This post details step-by-step instructions in setting up both ArcGIS Portal and QGIS to enable this integration. First, we’ll create a new desktop application on the Portal site in order to obtain the application-specific access keys for OAuth2 authentication. We’ll then create an authentication configuration in QGIS and associate this with a connection to the Portal site. Let’s dive in by doing the Portal configuration first…

Creating an application

Logon to the Portal, and from the “Content” tab, click the “Add Item” option. Select “An application” from the drop down list of options:

Set the type of the application as “Desktop

You can fill out the rest of this dialog as you see fit. Suggested values are:

  • Purpose: Ready to Use
  • Platform: Qt
  • URL: http://qgis.org
  • Tags: QGIS, Desktop, etc

Now – here comes a trick. Portal will force you to attach a file for the application. It doesn’t matter what you attach here, so long as it’s a zip file. While you could attach a zipped copy of the QGIS installer, that’s rather wasteful of server space! We’d generally just opt for a zip file containing a text file with a download link in it.

Click Add Item when you’re all done filling out the form, and the new application should be created on the Portal.

Registering the Application

The next step is to register the application on Portal, so that you can obtain the keys required for the OAuth2 logon using it. From the newly created item’s page, click on the “Settings” tab:

Scroll right to the bottom of this page, and you should see a “Register” button. Press this. Set the “App type” to “Native“.

Add two redirect URIs to the list (don’t forget to click “Add” after entering each!):

  1. The Portal’s public address, e.g. https://mydomain.com/portal
  2. http://127.0.0.1:7070

Finally, press the “Register” button in the dialog. If all goes well then the App Registration section in the item settings should now be populated with details. From here, copy the “App ID” and “Secret” strings, we’ll need these later:

Determine Request URLs

One last configuration setting we’ll need to determine before we fire up QGIS is the Portal’s OAuth Request and Token URLs. These are usually found by appending /sharing/rest/oauth2/authorize and /sharing/rest/oauth2/token to the end of your Portal’s URL.

For instance, if your public Portal URL is http://mydomain.com/portal, then the URLs will be:

Request URL: http://mydomain.com/portal/sharing/rest/oauth2/authorize
Token URL: http://mydomain.com/portal/sharing/rest/oauth2/token

You should be able to open both URLs directly in a browser. The Request URL will likely give a “redirect URL not specified” error, and the Token URL will give a “client_id not specified” error. That’s ok — it’s enough to verify that the URLs are correct.

We’re all done on the Portal side now, so time to fire up QGIS!

Creating an QGIS OAuth2 Authentication Configuration

From your QGIS application, select Options from the Settings menu. Select the Authentication tab. We need to create a new authentication configuration, so press the green + button on the right hand side of the dialog. You’ll get a new dialog prompting you for Authentication details.

There’s a few tricks to this setup. Firstly, it’s important to ensure that you use the exact same settings on all your client machines. This includes the authentication ID field, which defaults to an auto-generated random string. (While it’s possible to automatically deploy the configuration as part of a startup or QGIS setup script, we won’t be covering that here!).

So, from the top of the dialog, we’ll fill in the “Name” field with a descriptive name of the Portal site. You then need to “unlock” the “Id” field by clicking the little padlock icon, and then you’ll be able to enter a standard ID to identify the Portal. The Id field is very strict, and will only accept a 7 letter string!

Drop down the Authentication Type combo box, and select “OAuth2 Authentication” from the list of options. There’s lots of settings we need to fill in here, but here’s what you’ll need:

  • Grant flow: set to “Authorization Code”
  • Request URL: enter the Request URL we determined in the previous step, e.g. http://mydomain.com/portal/sharing/rest/oauth2/authorize
  • Token URL: enter the Token URL from the previous step, e.g. http://mydomain.com/portal/sharing/rest/oauth2/token
  • Refresh Token URL: leave empty
  • Redirect URL: leave as the default http://127.0.0.1:7070 value
  • Client ID: enter the App ID from the Portal item’s App Registration information (see earlier steps)
  • Client Secret: enter the App Secret from the Portal item’s App Registration information (see earlier steps)
  • Scope: leave empty
  • API Key: leave empty

That’s it — leave all the rest of the settings at their default values, and click Save.

You can close down the Options dialog now.

Adding the Portal Connection Details

Lastly, we’ll need to setup the Portal connection as an “ArcGISFeatureServer” and “ArcGISMapServer” connection in QGIS. This is done through the QGIS “Data Source Manager” dialog, accessed through the Layer menu. Click the “ArcGIS Feature Server” tab to start with, and then press “New” in the Server Connections group at the top of this dialog.

Enter a descriptive name for the connection, and then enter the URL for the ArcGIS server REST endpoint associated with your Portal:

Lastly, select the new OAuth2 configuration you just created under the “Authentication” group:

Click OK, and you’re done! When you try to connect to the newly added connection, you’ll automatically be taken to the Portal’s logon screen in order to authenticate with the server. After entering your details, you’ll then be connected securely to the server and will have access to all items which are shared with your user account on the Portal!

You can then repeat this step for and create a similar connection under the “ArcGIS Map Server” tab.

We’ve regularly use this setup for our enterprise clients, and have found it to work flawlessly in recent QGIS versions! If you’ve found this useful and are interested in other “best-practice” recommendations for mixed Open-Source and ESRI workplaces, don’t hesitate to contact us to discuss your requirements… at North Road we specialise in ensuring flawless integration between ESRI based systems and the Open Source geospatial software stack.

Movement data in GIS #30: synchronized trajectory animations with QGIS temporal controller

QGIS Temporal Controller is a powerful successor of TimeManager. Temporal Controller is a new core feature of the current development version and will be shipped with the 3.14 release. This post demonstrates two key advantages of this new temporal support:

  1. Expression support for defining start and end timestamps
  2. Integration into the PyQGIS API

These features come in very handy in many use cases. For example, they make it much easier to create animations from folders full of GPS tracks since the files can now be loaded and configured automatically:

Script & Temporal Controller in action (click for full resolution)

All tracks start at the same location but at different times. (Kudos for Andrew Fletcher for recordings these tracks and sharing them with me!) To create an animation that shows all tracks start simultaneously, we need to synchronize them. This synchronization can be achieved on-the-fly by subtracting the start time from all track timestamps using an expression:

directory = "E:/Google Drive/QGIS_Course/05_TimeManager/Example_Dayrides/"

def load_and_configure(path):
    path = os.path.join(directory, filename)
    uri = 'file:///' + path + "?type=csv&escape=&useHeader=No&detectTypes=yes"
    uri = uri + "&crs=EPSG:4326&xField=field_3&yField=field_2"
    vlayer = QgsVectorLayer(uri, filename, "delimitedtext")
    QgsProject.instance().addMapLayer(vlayer)

    mode = QgsVectorLayerTemporalProperties.ModeFeatureDateTimeStartAndEndFromExpressions
    expression = """to_datetime(field_1) -
    make_interval(seconds:=minimum(epoch(to_datetime("field_1")))/1000)
    """

    tprops = vlayer.temporalProperties()
    tprops.setStartExpression(expression)
    tprops.setEndExpression(expression) # optional
    tprops.setMode(mode)
    tprops.setIsActive(True)

for filename in os.listdir(directory):
    if filename.endswith(".csv"):
        load_and_configure(filename)

The above script loads all CSV files from the given directory (field_1 is the timestamp, field_2 is y, and field_3 is x), enables sets the start and end expression as well as the corresponding temporal control mode and finally activates temporal rendering. The resulting config can be verified in the layer properties dialog:

To adapt this script to other datasets, it’s sufficient to change the file directory and revisit the layer uri definition as well as the field names referenced in the expression.


This post is part of a series. Read more about movement data in GIS.

TimeManager is dead, long live the Temporal Controller!

TimeManager turns 10 this year. The code base has made the transition from QGIS 1.x to 2.x and now 3.x and it would be wrong to say that it doesn’t show ;-)

Now, it looks like the days of TimeManager are numbered. Four days ago, Nyall Dawson has added native temporal support for vector layers to QGIS. This is part of a larger effort of adding time support for rasters, meshes, and now also vectors.

The new Temporal Controller panel looks similar to TimeManager. Layers are configured through the new Temporal tab in Layer Properties. The temporal dimension can be used in expressions to create fancy time-dependent styles:

temporal1

TimeManager Geolife demo converted to Temporal Controller (click for full resolution)

Obviously, this feature is brand new and will require polishing. Known issues listed by Nyall include limitations of supported time fields (only fields with datetime type are supported right now, strings cannot be used) and worse performance than TimeManager since features are filtered in QGIS rather than in the backend.

If you want to give the new Temporal Controller a try, you need to install the current development version, e.g. qgis-dev in OSGeo4W.


Update from May 16:

Many of the limitations above have already been addressed.

Last night, Nyall has recorded a one hour tutorial on this new feature, enjoy:

QGIS video tutorials: election maps, hydrology, and more

Mapping spatial decision patterns, such as election results, is always a hot topic. That’s why we decided to include a recipe for election maps in our QGIS Map Design books. What’s new is that this recipe is now available as a free video tutorial recorded by Oliver Burdekin:

This video is just one of many recently published video tutorials that have been created by QGIS community members.

For example, Hans van der Kwast and Kurt Menke have recorded a 7-part series on QGIS for Hydrological Applications:

and Klas Karlsson’s Youtube channel is also always worth a follow:

For the Pythonically inclined among you, there is also a new version of Python in QGIS on the Automating GIS-processes channel:

 

(Fr) Entretien avec Vincent Picavet

Sorry, this entry is only available in French.

SLYR ESRI to QGIS compatibility suite – April 2020 update

Since the last update, our “SLYR” ESRI to QGIS compatibility suite has gained a ton of new functionality, including full support for conversion of ArcMap MXD documents (with page layouts!). In this update, we’ll explore some of the new functionality available in the tool — but instead of focusing solely on SLYR, this time we’ll also explore the enhancements we’ve been making in QGIS itself that have helped improve the quality of ArcMap document conversion.

While many of these enhancements are already available to all users of QGIS 3.12, others are exciting additions to the upcoming QGIS 3.14 release. Let’s dive in!

Improved legend customisation

One shortcoming we realised early on during our work on SLYR was that QGIS map legends just didn’t offer a comparable level of customisation as ArcMap legends. We could convert the basic layout of a legend, but we just couldn’t get the legend appearance in QGIS sufficiently close to its original appearance in the MXD file.

To address this we’ve been extending QGIS’ inbuilt legend support by adding finer control over the legend layout and spacing.

As a result, one exciting addition we’ve recently made for QGIS 3.14 is adding the ability to customise legend patch shapes and sizes on an item-by-item basis! Previously, legends in QGIS were rather boring, with all polygon layers showing as a plain rectangle and line layers as a horizontal line.

Now, users have full control over setting custom shapes for their legend patches! This makes for much more user-friendly legends, as you can now show representative shapes in your legends — e.g. a river symbol can be shown as a wiggly line, instead of an unrealistic straight horizontal line.

You can also override the size of a legend patch on an item-by-item basis too, which allows for further control over the final legend appearance. Checkout the screencast below showing both these features in action (naturally, using a legend from a converted MXD document… ArcMap users will likely recognise the fonts and patch shapes used here!)

We really wanted custom legend patch shapes to be a full first-class citizen in QGIS, so we also added support for managing them in user’s style databases. This makes it easy to setup your own libraries of custom legend shapes and share them with others. As a nice bonus, the SLYR tool even offers support for converting area and line patch shapes while converting ESRI .style databases:

Marker north arrows

Another issue we ran into while converting ArcMap page layouts was converting north arrows. QGIS used a very different approach to north arrows compared with ArcMap — in QGIS, north arrows were always based on existing SVG files, while ArcMap uses a rotated marker symbol for north arrows. Both approaches have their advantages and disadvantages, but we struggled to get good results when trying to convert ArcMap’s marker based approach to QGIS’ SVG based approach.

In the end, we weren’t happy with the result, so we took the step of implementing full support for marker symbol north arrows in QGIS 3.14. Now QGIS users have a choice of both north arrow styles — you can still create north arrows direct from SVG files, but you’ve also now got the flexibility to create them from standard marker symbols instead!

Adding support for marker based north arrows in QGIS allows us to get an perfect match when converting ESRI Page Layouts with north arrows:

Hollow and stepped line scale bars

While adding support for scale bar conversion to SLYR, we identified that some scale bar styles which are widely used in ArcMap just weren’t possible to reproduce in QGIS. Accordingly, from QGIS 3.14 on, we added native support for “Stepped line” and “Hollow” scale bar styles:

Embedded images in QGIS print layouts

ArcMap offers users the ability to directly embed images inside symbol definitions or page layouts. Whilst QGIS has offered embedded image support in symbols for a number of releases, this previously wasn’t possible to do in print layouts.

This was an issue for us while converting ArcMap page layouts, because we didn’t have any way to represent embedded images in QGIS layouts. Accordingly, for QGIS 3.14 we’ve added native support for directly embedding images (either raster images or SVG pictures) inside a page layout:

One handy consequence of this improvement is that it’s now possible to create completely self contained print layout templates for QGIS — you no longer have to separately distribute any required images (such as company logos) along with your QPT templates!

Naturally, our SLYR plugin now automatically converts any embedded images it finds in an ArcMap page layout and correctly creates a converted, embedded version of the image in the QGIS print layout.

Scalebar numeric formats

Another missing customisation in QGIS’ scale bar functionality was allowing users control over the scale bar’s number format. Previously, QGIS offered no customisation for these numbers, so you got only what QGIS decided you wanted. We improved this in QGIS 3.12 by offering users the ability to control exactly how they want their scale bar numbers to appear. There’s options for manually controlling the thousand and decimal separators, rounding, and much more:

This enhancement allowed us to get an exact match when converting ArcMap scale bars — the converted results should appear identical to their original ArcMap appearance!

Random marker fill

Until recently, one of the few remaining layer symbolisation gaps between QGIS and ArcMap was that QGIS had no symbology option for randomised marker placements for fill symbols. This was a big issue for us, because without it there just wasn’t any way that SLYR could convert layers styled with ArcMap’s “dot density renderer”  or using a marker fill’s “random offset” option.

So for QGIS 3.12, we add inbuilt support for a new “Random marker fill” symbol type:

This new symbol layer type allows for randomised (or stable seed-based) placement of markers inside polygon features. You’re given the option of either an absolute number of points to show in the feature, or a density-based count which retains its dot density regardless of the map’s scale!

Aside from being a useful symbology option in it’s own right, adding this functionality allowed us to accurately convert random markers or dot density renders from ArcMap to QGIS.

Other enhancements

The highlights above are only a small subset of the work we’ve done in QGIS to improve its interoperability with ArcMap via the SLYR plugin! Some of the other work we’ve done includes:

  • Many improvements to QGIS’ bad layer handling and automatic repair functionality. For instance, QGIS now emulates ArcMap’s helpful behaviour where ALL similar broken layer paths in a document are fixed automatically after fixing the path to one broken layer. Handling broken layer paths was a pain point for our customers, so we’ve sought to make this as painless as possible.
  • Support for plugins to handle pasting content into QGIS print layouts. We use this in SLYR to offer the ability to directly copy and paste content from ArcMap page layouts into a QGIS print layout.
  • Support for plugins to hook into the standard QGIS “Open Project” dialog, offering support for opening projects of their own custom types. We use this to allow users to directly open MXD, MXT, PMF and SXD files from the QGIS Open Project menu action.
  • We’ve worked closely with the upstream proj project, to ensure that coordinate reference systems from ESRI documents are correctly matched to known EPGS/ESRI CRS definitions in certain circumstances.

Other new features in SLYR

Aside from all the goodness we’ve explored above, the latest versions of SLYR offer TONS of new functionality for conversion of ArcMap documents, including:

  • Full support for joins and relations when converting MXD documents
  • Print layouts, including support for conversion of data driven pages to QGIS print atlases and support for multi-map page layouts using multiple data frames.
  • Support for reading MXD document metadata (and converting this to QGIS document metadata)
  • Support for dragging and dropping layers direct from ArcMap or ArcCatalog to a QGIS window, respecting all the layer styling.
  • Support for AVL style conversion
  • A new tool for dumping the full structure of MXD or LYR files to a json document. This is very handy for digging right into the full internals of the documents and for diagnosing corrupted documents.
  • Full support for conversion of vector and raster layers
  • Support for converting MXD, MXT and PMF documents
  • Support for converting ArcScene SXD documents to 2-dimensional QGIS maps

Read more are the SLYR home page, or contact us today to discuss purchasing SLYR and your licensing needs!

Who is behind QGIS at Oslandia ?

You are using QGIS and look for support services to improve your experience and solve problems ? Oslandia is here to help you with our full QGIS editor service range ! Discover our team members below.

You will probably interact first with our pre-sales engineer Bertrand Parpoil. He leads Geographical Information System projects for 15 years for large corporations, public administrations or hi-tech SME. Bertrand will listen to your needs and explore your use cases, to offer you the best set of services.

Régis Haubourg also takes part in the first steps of projects to analyze your usages and improve them. GIS Expert, he knows QGIS by heart and will make the most of its capabilities. As QGIS Community Manager at Oslandia, he is very active in the QGIS community of developers and contributors. He is president of the Francophone OSGeo local chapter ( OSGeo-fr ), QGIS voting member, organizes the French QGIS day conference in Montpellier, and participates to QGIS community meetings. Before joining Oslandia, he led the migration to QGIS and PostGIS at the Adour-Garonne Water Agency, and now guides our clients with their GIS migrations to OpenSource solutions. Régis is also a great asset when working on water, hydrology and other specific thematic subjects.

Loïc Bartoletti develops QGIS, specializing in features corresponding to his fields of interests : network management, topography, urbanism, architecture… We find him contributing to advanced vector editing in QGIS, writing Python plugins, namely for DICT management. Pushing CAD and migrations from CAD tools to GIS and QGIS is one of his major goals. He will develop your custom applications, combining technical expertise and functional competences. When bored, Loïc packages software on FreeBSD.

Vincent Mora is senior developer in Python and C++, as well as PostGIS expert. He has a strong experience in numerical simulation. He likes coupling GIS (PostGIS, QGIS) with 3D numerical computing for risk management or production optimization. Vincent is an official QGIS committer and can directly integrate your needs into the core of the software. He is also GDAL committer and optimizes low-level layers of your applications. Among numerous activities, Vincent serves as lead developer of the development team for Hydra Software, a tool dedicated to unified hydraulics and hydrology modelling and simulation based on QGIS.

Hugo Mercier is an officiel QGIS committer too for several years. He regularly talks in international conferences on PostGIS, QGIS and other OpenSource GIS softwares. He will implement your needs with new QGIS features, develop innovative plugins ( like QGeoloGIS ) and design and build your new custom applications, solving all kind of technological challenges.

Paul Blottière completes our QGIS committers : very active on core development, Paul has refactored the QGIS server component to bring it to an industry-grade quality level. He also designed and implemented the infrastructure allowing to guarantee QGIS server performances. He dedicated himself to QGIS server OGC certification, especially for WMS (1.3). Thanks to this work QGIS is now a reference OGC implementation.

Julien Cabièces recently joined Oslandia, and quickly dived into QGIS : he contributes to the core of this Desktop GIS, on the server component, as well as applications linked to numerical simulation. Coming from a satellite imagery company with industrial applications, he uses his flexibility to answer all your needs. He brings quality and professionalism to your projects, minimizing risks for large production deployments.

You may also meet Vincent Picavet. Oslandia’s founder is a QGIS.org voting member, and is involved in the project’s evolution and the organization of the community.

Aside from these core contributors, all other Oslandia members also master QGIS integrate this tool into their day-to-day projects.

Bertrand, Régis, Loïc, Vincent (x2), Hugo, Paul et Julien are in tune with you and will be happy to work together for your migrations, application development, and all your desires to contribute to the QGIS ecosystem. Do not hesitate to contact us !

(Fr) Oslandia recrute : Ingénieur(e) développement d’applications SIG ( Python / SQL / QGIS )

Sorry, this entry is only available in French.

QGIS Processing, Model Designer and ETL Campaign crowdfund launched!

QGIS Processing offers a rich and expandable set of algorithms which can operate on spatial data, along with a powerful Model Designer which allows users to string together these algorithms to create custom workflows.

Since its introduction in QGIS 2, the Processing framework has seen an intensive amount of development and optimisation efforts. In recent QGIS releases it offers a very user-friendly way of performing complex spatial data processing tasks, all without requiring ANY expensive third-party tools or software licenses!

At North Road we are passionate about the QGIS Processing framework, and have invested considerable effort in this framework over the past 5 years. We’re proud to announce that our latest crowd-funding campaign is focused on further expanding the capabilities and flexibility of Processing and the Processing Model Designer!

Unlike a typical crowdfunding campaign, where a specific funding target and deadline is set, we’re running this campaign a little differently. Instead, this campaign is taking the form of a “à la carte” menu of Processing enhancements. These range from small “paper-cut” style fixes, through to larger architectural improvements, and are each individually priced accordingly. We are asking backers to pick individual enhancements from this “menu of enhancements” and fund that enhancement’s development in full. In order to make this campaign affordable for a wide range of backers, we’ve included a huge range of enhancements which vary in price from smaller amounts to larger amounts.

You can read the full details of the campaign and browse the list of proposed enhancements at the campaign page.

QGIS Abstract Connections API

 

The goal of the new API is twofold:

  1. provide a unified way to store and retrieve data provider connections in the QGIS settings
  2. provide an abstract set of methods to perform most common operation on DB data sources (e.g. browse tables, drop/create a table/schema, run arbitrary SQL commands etc.)

 

The new API is documented in https://qgis.org/api/classQgsAbstractProviderConnection.html and it provides a few specializations for DB connections (https://qgis.org/api/classQgsAbstractDatabaseProviderConnection.html) and an initial PR implementation for web service-based connections (https://github.com/qgis/QGIS/pull/33045).

 

While the whole of the desired refactoring work was too big for a single grant request, the first work package has been completed and the following data providers have been partially or totally refactored to make use of the new connections API:

  • postgres
  • geopackage (OGR)
  • spatialite

 

The new API was also used to implement the automatic loading of layer dependencies (not part of the grant program).

 

For developers interested in working with the new API, a set Python tests are available to show how to use the methods:  https://github.com/qgis/QGIS/blob/master/tests/src/python/test_qgsproviderconnection_ogr_gpkg.py (see also the postgres and spatialite companion tests).

 

There is still a large amount of work to be done in order to complete all the desired refactoring and to remove all the Python and C++ code that will be ultimately be made redundant. In particular, future work should be undertaken to:

  • port all remaining data providers to the new API
  • refactor and eliminate the remaining DB-manager connectors to make use of the abstract API
  • eliminate duplicate and untested code inside the Processing framework for working with Postgres databases and port the code to the new, stable, well-tested API
  • refactor and eliminate the remaining QGIS browser data items to make use of the abstract API 

 

For further information, the following paragraphs (taken from the original grant proposal) will provide full details about the background of this work.

Background/motivation

  • DB-Manager is an important part of the QGIS interface, which allows browsing/previews of different DB-based data sources, complex queries, management of layers, import-export etc., DB creation and deletion etc.
  • After the QGIS 3.0 release, improvements within the core browser widgets implemented in C++ have resulted in a (constantly growing) degree of overlapping functionality between the browser and db manager.
  • After QGIS 3 API improvements concerning layer import and export functionality, there are many duplicated implementations between browser and db manager – some functionality is better in browser, some functionality is better in db manager. Users are forced to choose between two competing semi-complete alternatives, instead of having one, complete, well integrated solution.
  • There are no unit tests for DB-Manager and this leads to frequent regressions, which (aside from being frustrating for users) consume a substantial part of our development time and budget during the bugfixing programs. Furthermore the nature of large Python codebases like db manager makes it very easy to accidentally break functionality with no warning or errors during development.

 

Proposed solution

We propose to start refactoring the DB-manager plugin functionality into core C++ implementation, reusing existing core API and replacing redundant duplicate functionality.

The clear advantages are:

  • no duplicate functionality, so it’s easier for users to understand and use
  • more usage of well-tested and well-maintained core C++ API
  • testability and immediate feedback on API breaks (an advantage of C++ is that the application won’t even build if an API is changed or accidentally misused)
  • better performance
  • the ability to expose database management functionality via stable PyQGIS API, allowing other plugins and scripts to utilise this functionality. In future, Processing algorithms may also be developed which would take advantage of these functions (e.g. “create schema”, “drop table”, “vacuum table” algorithms)
  • DB management functionality would be available within the main QGIS window (from the Browser panel), instead of as a separate dialog.

 

Grant proposal package

The above mentioned work is too large to be completed within a single grant, so what we propose here is to start the refactoring needed in order to have a core stable C++ API that can be used by the application and the plugins and that will be available to fully move DB manager to C++ API in the future avoiding duplication of code and functionality.

  • create an interface for databases that expose the required functions to a coherent API
  • add missing tests and documentation for the a.m. API
  • porting some basic functions from db manager to the new api:
    • create table (with native field types support)
    • create schema
    • delete table
    • Rename table

The API will be exposed through the browser and it will be used by the DB manager instead of the Python implementation that is currently used.

QGIS Snapping improvements

A few months ago, we proposed to the QGIS grant program to make improvements to the snap cache in QGIS. The community vote selected our project which was funded by QGIS.org. Developments are now mostly finished.

In short, snapping is crucial for editing geospatial features. It is the only way to ensuring they are topologically related, ie, connected vertices have exactly the same coordinates even if manual digitizing on screen is imprecise by nature.  Snapping correctly supposes QGIS have in memory an indexed cache of the geometries to snap to. And maintainting this cache when data is modified, sometimes by another user or database logic, can be a real challenge. This it exactly what this work adresses.

The proposal was divided into two different tasks:

  • Manage circular dependencies
  • Relax the snap cache index build

Manage cicular data dependencies

Data dependencies

Data dependency is an existing feature that allows you to configure QGIS to reload layers (and their snapping cache) when a layer is modified.

It is useful when you store your data in a database and you set up triggers to maintain consistency between the different tables of your data model.

For instance, say you have topological informations containing lines and nodes. Nodes are part of lines and lines go through nodes. Then, you move a node in QGIS, and save your modifications to the database. In order to keep the data consistent, a trigger updates the geometry of the line going through the modified node.

Node 2 is modified, Line 1 is updated accordingly

QGIS, as a database client, has no information that the line layer currently displayed in the canvas needs to be refreshed after the trigger. Although the map canvas will be up to date, because QGIS fetches data for display without any caching system, the snapping cache is not and you’ll end up with ghost snapping highlights issues.

Snapping highlights (light red) differ from real line (orange)

Defining a dependency between nodes and lines layers tells QGIS that it has to refresh the line layer when a node is modified.

Dependencies configuration: Lines layer will be refreshed whenever Nodes layer is modified

It also have to work the other way, modifying a line should update the nodes to ensure they still are on the line.

Circular data dependencies

So here we are, lines depend on nodes which depend on lines which depend on nodes which…

That’s what circular dependencies is about. This specific behavior was previously forbidden and needed a special way to deal with it. Thanks to this recent development, it is now possible.

It’s also possible to add the layer itself as one of its own dependencies. It helps dealing with specific cases where one feature modification could lead to a modification of another feature in the same layer (to keep consistency on road networks for instance).

Road 2 is modified, Road 1 is updated accordingly

This feature is available in the next QGIS LTR version 3.10.

Relax the snapping cache index build

If you work in QGIS with huge projects displaying a lot of vector data, and you enable snapping while editing these data, you probably already met this dialog:

Snap indexing dialog

This dialog informs you that data are currently being indexed so you can snap on them while you will edit feature geometry. And for big projects, this dialog can last for a really long time. Let’s work on speeding it up!

What’s a snap index?

Let’s say you want to move a line and snap it onto another one. While you drag your line with the mouse, QGIS will look for an existing geometry beneath the mouse cursor (with a certain pixel tolerance) every time you move your mouse. Without spatial index, QGIS will have to go through every geometry in your layer to check if the given geometry is beneath the cursor position. This would be very ineffective.

In order to prevent this, QGIS keeps an index where vector data are stored in a way that it can quickly find out what geometry is beneath the mouse cursor. The building of this data structure takes time and that is what the progress dialog is about.

Firstly: Parallelize snap index build

If you want to be able to snap on all layers in your project, then QGIS will have to build one snap index for each layer. This operation was made sequentially meaning that if you have for instance 20 layers and the index building last approximatively 3 seconds for each, then the whole index building will last 1 minute. We made modifications to QGIS so that index building could be done in parallel. As a result, the total index building time could theoretically be 3 seconds!

4 layers snap index being built in parallel

However, parallel operations are limited by the number of CPU cores of your machine, meaning that if you have 4 cores (core i7 for instance) then the total time will be up to 4 times faster than when the building is sequential (and last 15 seconds in our example).

Secondly: relax the snap build

For big projects, parallelizing index building is not enough and still takes too much time. Futhermore, to reduce snap index building, an existing optimisation was to build the spatial index for a specific area of interest (determined according to the displayed area and layer size). As a consequence, when you’ve done waiting for an index currently building and you move the map or zoom in/out, you could possibly trigger another snap index building and wait again.

So, the idea was to avoid waiting at all. Snap index is now built whenever it needs to (when you first enable snapping, when you move or zoom) but the user doesn’t have to wait for the build to be over and can continue what it was doing (creating feature, moving…). Snapping highlights will be missing when the index is currently being built and will appear gradually as soon as they finished. That’s what we call the relaxing mode.

No waiting dialog, snapping highlights appears as soon as snap index is ready

This feature has been merged into current QGIS master and will be present in future QGIS 3.12 release. We keep working on this feature in order to make it more stable and efficient.

What’s next

We’ll continue to improve this feature in the coming days, if you have the chance to test it and encounter issues please let us know on the QGIS tracker. If you think about a missing feature or just want to know more about QGIS, feel free to contact us at [email protected]. And please have a look at our support offering for QGIS.

Many thanks to QGIS grant program for funding these new features. Thanks also to all the people involved in reviewing the code and helping to better understand the existing mechanism.

 

De Ruimtelijke Plannen plugin profiteert van Open Source

Rececently the ruimtelijkeplannen plugin for using dutch spatial zoning plans in QGIS was renewed.
A lot of extra functionality was added, sponsored by LBP|SIGHT, a company which uses this plugin frequently.
As it happened, we now have a small list list of organizations who contributed to the development of this plugin, illustrating the power of the OpenSource model:

Kort verslag oprichtingsvergadering QGIS gebruikersgroep

Sorry, this entry is only available in Dutch.

Lid worden?

Sorry, this entry is only available in Dutch.

SLYR ESRI to QGIS compatibility suite – November 2019 update

It’s a been a month full of huge improvements since the last update, and we have some exciting news to share about our SLYR ESRI to QGIS compatibility suite. With the recently published plugin version 3.7, MXD conversion has moved from a “beta” state to being fully supported and available out-of-the-box for all users!

Based on our massive library of reference files (almost 10,000 files covering a huge range of ArcGIS versions and features!), the tool is now able to successfully convert 96% of LYR files and 94.5% of MXD documents. This is a significant milestone, and with it we decided that MXD conversion support is now stable enough to move out of its previous beta state.

Aside from this milestone, the 3.7 release brings many more enhancements and improvements, including:

  • SLYR now has full support for PMF published map documents created by ArcGIS Publisher, along with a new Processing algorithm to convert from a PMF document to a QGS projects
  • We’ve also added support for converting ArcScene SXD scenes to QGS projects. This conversion is 2-dimensional only for now, but we plan on adding 3D conversion when QGIS’ 3D support further matures.
  • We now convert all data frames contained within MXD documents, instead of just the first data frame. Currently, these are exposed as their own individual groups within the project layer tree (when we enable support for page layout conversion we’ll be automatically creating corresponding map themes from each data frame).
  • We’ve added support for reading many more layer types, including raster catalog layers, topology layers, terrain layers, and LAS dataset layers. While QGIS doesn’t have support for these layer types, we need to fully parse them in order to convert the rest of the MXD document contents. Whenever an unsupported layer type like these are encountered the plugin shows a warning advising users which layers could not be successfully converted.
  • We’ve also added support for reading TIN layers. Although previous QGIS versions had no means to read ESRI tin layers, thanks to work done in the MDAL library the upcoming QGIS 3.10.1 release adds full support for reading these data files! Accordingly, we’ll be unlocking support for converting TIN layers contained within MXD documents following the 3.10.1 release.
  • Full support for WMTS and tiled internet layers
  • Support for reading MXD documents which have repaired by the MXD Doctor utility
  • Support for layers with a geopackage source
  • Conversion of ImageServer based layers (since QGIS only has basic support for ESRI ImageServers, we convert these layers to their equivalent MapServer versions wherever possible)
  • Basic support for representation renderers. Although QGIS has no capability to utilise the symbology linked with a representation renderer, we’ve added support for rendering these layers using any geometry overrides which may be present for the features.
  • Conversion support for simple scale dependent renderers (these are a funny beast, which can’t be created directly through the ArcMap interface and which require custom ArcObjects code to create! That said, we’ve encountered a few examples of these inside our test library so have added support for converting them to the equivalent QGIS rule based renderer).
  • We added a new “random marker fill” symbol type to the upstream QGIS project, which will be available in QGIS 3.12 along with support in SLYR for conversion of ESRI random marker fills.

So what’s next for SLYR? Over the remainder of 2019 we’ll be working furiously toward 100% conversion rates for LYR and MXD files. We’ll also start rolling out conversion support for page layouts to QGIS print layouts, and support for automatic conversion of ArcMap TIN layers to QGIS mesh layers.

Keep an eye on this blog and our Twitter channel for further updates!

 

QGIS Print Layouts Graphs and Charts Campaign – Complete!

Last week saw the exciting release of version 3 of the QGIS DataPlotly plugin, which incorporates all the work done as a result of our Print Layouts Graphs and Charts crowdfunding campaign crowd funding campaign. Now, beautiful charts and graphs are available directly within QGIS print layouts, and all it takes is the easy installation of the “DataPlotly” plugin from your QGIS install!

In this post we’ll showcase the functionality which has been added during this campaign, and which is available today in the plugin.

UI modernisation and tweaks

First up, during our work on this plugin we’ve invested some time in refreshing the plugin’s UI to ensure it follows all the widget conventions used elsewhere in QGIS. Now, the plugin blends seamlessly into your QGIS window, and all the chart setting widgets behave in just the way you’re used to. We’ve also used this opportunity to fix a number of issues the plugin had when running on hi-dpi displays (such as Apple retina displays)!

 

Dynamic charting of selected and visible features

If you’ve used earlier versions of the DataPlotly plugin, you’ll likely notice that there’s many new options shown in the screenshot above. Possibly the most exciting of these is the new “Use only selected features” and “Use only visible features” checkboxes. When either of these options are enabled, then your chart will immediately respond to changes in layer selections or changes to the visible map canvas extent respectively. Previously, interactivity in the plugin only went one way (from the chart to the canvas) – but now the charts are truly interactive, and respond dynamically to changes in the canvas too!

 

Improved handling of “data defined” settings

During the plugin refactoring, we reworked how “data defined” settings are handled within charts. If you’re not familiar with these, “data defined” settings are QGIS’ approach for exposing per-feature control over the map rendering process. In DataPlotly charts, we expose this functionality to allow per-feature control over the chart appearance (e.g., showing different scatter plot dot colors based on feature attributes). The new code uses the same code model as QGIS itself, so data-defined settings in your charts now have full access to the whole suite of QGIS expression functions and variables that you’re used to! Additionally, QGIS data-defined assistants are fully supported in the charts too. Ultimately, this enables some very advanced styling options, such as charts which dynamically change color and appearance on every page of your print atlas…

Charts in print layouts

We’ve previously covered this feature in depth, but the DataPlotly v3 release brings print-layout based charts to the masses! When a chart is inserted into a print layout, some additional options are available for controlling the plot behavior:

These new options allow you to link the chart to a map item within the layout, which lets you filter the content of the chart to only include features visible within the map. If your print layout is setup as an atlas export, you can also filter out included features to only show those which are geographically located inside the current atlas feature.

Our partners from Faunalia demonstrate this in the screencast below:

 

Saving plot configuration

An often-requested functionality previous missing from the plugin was the ability to save and restore plot configuration. Now, plot configuration is automatically saved within your QGIS project and restored when you reopen the project. You no longer have to re-create all your charts from scratch at every session (ouch!). We also added the ability to export chart configuration to XML files, allowing you to share and reuse chart configuration across projects.

Behind the scenes work

Aside from all the wonderful new features added to the plugin, we’ve extensively refactored most of the plugin backend. Unit tests and CI infrastructure have been added, which will ensure the plugin remains stable and regression-free in future releases. The code cleanup and simplification has drastically lowered the barrier of contribution to the project, and we’ve already seen new contributors adding more new features to the plugin as a result of this! (Kudos to Simon Gröchenig, who added the new “Feature subset” expression option you can see in the above screenshots!).

Project sponsors

All this work is thanks to the backers of our crowdfunding campaign. Without their contributions this work would not have been possible! In no particular order, our thanks go out to:

  • Federico Gianoli
  • Papercraft Mountains
  • Liam McCrae
  • Henry Walshaw
  • Raúl Sangonzalo
  • Ferdinando Urbano
  • pitsch-ing.ch
  • Carbon-X
  • Gabriel Diosan
  • Rene Giovanni Borella
  • Enrico Bertonati
  • Guido Ingwer
  • David Addy
  • Gerd Jünger
  • Andreas Neumann
  • Stefano Campus
  • Michael Jabot
  • Korto
  • Enrico Ferreguti
  • Carlo A. Nicolini
  • Salvatore Fiandaca
  • Alberto Grava
  • Hans van der Kwast
  • Ben Hur Pintor
  • Silvio Grosso
  • Nobusuke Iwasaki
  • Alasdair Rae
  • Manori Senanayake
  • Canton de Neuchâtel
  • Matthias Daues
  • Alteri Seculo
  • SunGIS Ltd.
  • Stu Smith
  • Keolis Rennes
  • Gabriel Diosan
  • Aiden Price
  • Giacomo Ponticelli
  • Diane Fritz
  • Gemio Bissolati
  • Claire Birnie
  • Nicolas Roelandt
  • Rocco Pispico
  • Gabriel Bengtsson
  • Birds Eye View
  • Barend Köbben
  • Roberto Marzocchi (GTER)
  • Yoichi Kayama
  • Alessandro Sarretta
  • Luca Angeli
  • Luca Bellani
  • giswelt
  • Stefan Giese
  • Ben Harding
  • Joao Gaspar
  • Romain Lacroix
  • Ryan Cooper
  • Daniele Bonaposta
  • QGIS Swedish User Group
  • Nino Formica
  • Michael Gieding
  • Amedeo Fadini
  • Andrew Hannell
  • Stefano
  • Phil Wyatt
  • Brett Edmond Carlock
  • Transitec

Keep an eye out on the North Road blog for future crowd-funding initiatives. Coming soon: a QGIS Processing grab-bag of ETL modelling improvements!

Oprichting QGIS Gebruikersgroep Nederland

Sorry, this entry is only available in Dutch.

(Fr) Rechercher une adresse avec QGIS

Sorry, this entry is only available in French.

  • Page 1 of 40 ( 795 posts )
  • >>
  • qgis

Back to Top

Sponsors