Page 1 of 1 (10 posts)

  • talks about »
  • linux

Tags

Last update:
Sat Oct 25 02:45:17 2014

A Django site.

QGIS Planet

Compiling QGIS 2.0.1 for Fedora 19 in a few steps

Thanks to Volker Fröhlich’s efforts, a source code RPM package (SRPM) of QGIS 2.0.1 is now available for Fedora. If you are not yet F20 user (like me), you can just take the F20 package and compile it for F19 (or even F18) since there will be no backport of QGIS 2 to F19 (it comes with QGIS 1.8). But: we do want QGIS 2 on Fedora19!

Solution: compile it yourself.

1. 1. Preparations

The best way is  to use “mock” which is used to recompile SRPMS in a separate local environment (“chroot”) without cluttering the system with extra packages needed for the compilation (run as “root”):

su
yum install mock

2. 2. Get the source code

Next download the SRPM package from the Koji  server:
QGIS:  http://koji.fedoraproject.org/koji/buildinfo?buildID=467757 (–> src – download) or check here for more recent versions.

3. 3. Compile it locally as RPM package

The general compilation command (“mock”) would be:

mock -r my_fedora_version_config --rebuild my_source_rpm.src.rpm

So, check for Fedora version config name which is suitable for your system (“my_fedora_version_config“)

ls /etc/mock/

In my case of a 64bit machine, it is “fedora-19-x86_64″. Hence we can compile QGIS 2.0.1 directly from the SRPM file:

mock -r fedora-19-x86_64 --rebuild qgis-2.0.1-2.fc20.src.rpm

Note: the compilation takes 40min on my tiny core i3 laptop (ASUS X202). Use the time to donate some coins to the QGIS project :-)

4. 4. Install and enjoy

Once the compilation job is done, i.e. the binary RPM files are available for installation. To install the freshly compiled QGIS 2.0.1 RPMs, run:

cd /var/lib/mock/fedora-19-x86_64/result/

# an existing QGIS1.8 installation will be replaced: 
yum localinstall qgis-2.0.1-2.fc19.x86_64.rpm \
qgis-grass-2.0.1-2.fc19.x86_64.rpm qgis-python-2.0.1-2.fc19.x86_64.rpm

# consider to cleanup (or keep it for the next update, it is about 1.5GB):
rm -rf /var/lib/mock/fedora-19-x86_64/
# leave the "root" shell
exit

Now we can happily use QGIS 2.0.1 on Fedora 19!

qgis

qgis201_on_fedora19

Howto shrink a remote root ext3 filesystem on Debian wheezy

This howto describes how to resize a root ext3 filesystem on a remote Debian wheezy server.

This howto is not an original work but only an update of an older howto by Stefan @ https://thunked.org/. This version is specifically adapted to a server running Debian wheezy.

DISCLAIMER:

THERE IS A DECENT CHANCE THAT IF YOU FUCK THIS UP YOUR
REMOTE SYSTEM SIMPLY WONT BOOT AT ALL.  I URGE YOU TO TEST
THIS LOCALLY BEFORE USING THIS METHOD ON A PRODUCTION
SYSTEM. ESPECIALLY IF YOUR SYSTEM IS NOT DEBIAN WHEEZY,
SINCE THAT'S THE ONLY ONE I HAVE TESTED.

THE QUICK WAY:

If you don't want to read the whole thing you can only
execute the commands I run and probably skip the
explanations.

I’ve only done this on Debian Wheezy. If you’re using another distro the initrd layout and init scripts may be a bit different. However, I suspect it looks very similar on almost every distro out there. On Debian my root partition is an ext3 partition.

The general idea is pretty simple: you can’t shrink a mounted partition and it’s impossible to unmount or replace your root partition in a live system, so we have to resize the partition before it is mounted. What we’ll do to accomplish this is change the initrd image to make the init scripts resize the root partition before mounting it. This is by far the most flexible and easy method to resize your root partition I could think of. Most suggestions I found on google required you to create separate OS on a new root partition and boot into that, but I did not have any space to create a new root partition on my remote machine.

Unpacking the initrd image is fairly straight forward:

$ mkdir ~/initrd; cd ~/initrd
$ gunzip -c /boot/initrd.img-3.2.0-3-amd64 | cpio -i --make-directories
62631 blocks
$ ls -l
total 40
drwxr-xr-x 2 root root 4096 Sep 11 20:23 bin
drwxr-xr-x 3 root root 4096 Sep 11 20:23 conf
drwxr-xr-x 6 root root 4096 Sep 11 20:23 etc
-rwxr-xr-x 1 root root 6797 Sep 11 20:23 init
drwxr-xr-x 7 root root 4096 Sep 11 20:23 lib
drwxr-xr-x 2 root root 4096 Sep 11 20:23 lib64
drwxr-xr-x 2 root root 4096 Sep 11 20:23 run
drwxr-xr-x 2 root root 4096 Sep 11 20:23 sbin
drwxr-xr-x 6 root root 4096 Sep 11 20:23 scripts

First, we have to copy all the programs we need to resize our partition onto the initial ram disk. For my ext3 file system I need e2fsck and resize2fs. The programs are depending on a few libraries, so you’ll need to copy those to the new initrd image too. Libraries can also depend on other libraries, make sure you recursively check dependencies until you don’t have any missing dependencies anymore.

$ ldd /sbin/e2fsck
    linux-vdso.so.1 =>  (0x00007fff3594c000)
    libext2fs.so.2 => /lib/x86_64-linux-gnu/libext2fs.so.2 (0x00007f7cdddd5000)
    libcom_err.so.2 => /lib/x86_64-linux-gnu/libcom_err.so.2 (0x00007f7cddbd1000)
    libblkid.so.1 => /lib/x86_64-linux-gnu/libblkid.so.1 (0x00007f7cdd9a9000)
    libuuid.so.1 => /lib/x86_64-linux-gnu/libuuid.so.1 (0x00007f7cdd7a4000)
    libe2p.so.2 => /lib/x86_64-linux-gnu/libe2p.so.2 (0x00007f7cdd59c000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f7cdd214000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f7cdcff8000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f7cde01e000)
$ ldd /sbin/resize2fs
    linux-vdso.so.1 =>  (0x00007fffa216a000)
    libe2p.so.2 => /lib/x86_64-linux-gnu/libe2p.so.2 (0x00007fa01f7a2000)
    libext2fs.so.2 => /lib/x86_64-linux-gnu/libext2fs.so.2 (0x00007fa01f55f000)
    libcom_err.so.2 => /lib/x86_64-linux-gnu/libcom_err.so.2 (0x00007fa01f35a000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fa01efd3000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fa01edb7000)
    /lib64/ld-linux-x86-64.so.2 (0x00007fa01f9b0000)

We only copy the libraries that are not yet present in the initramfs. Fortunately in our case there are no recursive dependencies of those libraries.

$ for i in libext2fs.so.2.4 libcom_err.so.2.1 libe2p.so.2.3; do \
     cp -i /lib/x86_64-linux-gnu/$i lib/x86_64-linux-gnu/; \
  done
$ cd lib/x86_64-linux-gnu/
$ ln -s libcom_err.so.2.1 libcom_err.so.2
$ ln -s libext2fs.so.2.4 libext2fs.so.2
$ ln -s libe2p.so.2.3 libe2p.so.2
$ cd ~/initrd/

$ cp /sbin/e2fsck ~/initrd/bin/
$ cp /sbin/resize2fs ~/initrd/bin/

Next, we need to edit the init script. Debian uses busybox in its initrd image so the init script will be interpreted by a bourne shell. If you look through the init script file you’ll find the moment where the script mounts the root file system:

$ cat scripts/local
...
    # FIXME This has no error checking
    # Mount root
    if [ "${FSTYPE}" != "unknown" ]; then
            mount ${roflag} -t ${FSTYPE} ${ROOTFLAGS} ${ROOT} ${rootmnt}
    else
            mount ${roflag} ${ROOTFLAGS} ${ROOT} ${rootmnt}
    fi
...

Simply add in the commands to resize the file system right before the the file system is mounted. Resize2fs in Debian does not want to resize the file system before it is forcefully checked. It may be wise to add the -p or -y flag to e2fsck. -y will answer yes to all questions, this could prevent a hung system but may cause more damage to your files or filesystem. The -p flag will only automatically answer yes to safe operations. resize2fs takes two parameters, the first is the block device that has the ext2 or ext3 file system and the second is the new size you want to give it. By default the size is in blocks, but you can append a unit to change that. ‘K’ for kilobytes, ‘M’ for megabytes, ‘G’ for gigabytes and ‘T’ for terabytes. If you don’t specify a size, it will enlarge the file system to the total size of the partition or logical volume. After adding the commands the init script will look something like this:

    #RESIZEROOTFS MODIFIED!!! DONT RUN MORE THAN ONCE
    _log_msg "Starting e2fsck"
    /bin/e2fsck -p -f -C 0 /dev/sda4 || true
    _log_msg "Starting resize2fs"
    /bin/resize2fs /dev/sda4 100G || true

    # FIXME This has no error checking
    # Mount root
    if [ "${FSTYPE}" != "unknown" ]; then
            mount ${roflag} -t ${FSTYPE} ${ROOTFLAGS} ${ROOT} ${rootmnt}
    else
            mount ${roflag} ${ROOTFLAGS} ${ROOT} ${rootmnt}
    fi

If you have access to the system’s console, you might want to add a “-C 0” to e2fsck’s parameters. That will show you the progress of the fscheck.

There’s one last thing we have to do before re-packing the initrd image. e2fsck and resize2fs will fail if there is no /etc/mtab file available so we’ll have to make sure /etc/mtab exists.

$ touch ~/initrd/etc/mtab
$ cd ~/initrd/
$ find ./ | cpio -H newc -o > /tmp/initrd.cpio
64097 blocks
$ gzip -c /tmp/initrd.cpio > /boot/initrd-resize.img

And lastly, we need to add a new default boot option in grub. In Debian grub’s configuration file is constructed from various bits under /etc/grub.d. The resulting total configuration file is put under /boot/grub/grub.cfg. Open up the grub.cfg file in your favorite text editor and look for the default entry. It’s usually the first one, but may vary. Add a copy of the original entry to /etc/grub.d/40_custom and change the initrd image to the one we just created.

$ cat /etc/grub.d/40_custom
#!/bin/sh
exec tail -n +3 $0
# This file provides an easy way to add custom menu entries.  Simply type the
# menu entries you want to add after this comment.  Be careful not to change
# the 'exec tail' line above.

menuentry 'resize' --class debian --class gnu-linux --class gnu --class os {
        insmod gzio
        insmod part_gpt
        insmod ext2
        set root='(hd0,gpt1)'
        search --no-floppy --fs-uuid --set=root 0aa8bc27-17e3-4ae2-a9cf-497ab444970b
        echo    'Loading Linux 3.2.0-3-amd64 ...'
        linux   /vmlinuz-3.2.0-3-amd64 root=UUID=660f79dc-c152-4e15-ad61-7075b42de609 ro  quiet
        echo    'Loading initial ramdisk ...'
        initrd  /initrd-resize.img
}

Now make sure that it’s this entry that will be booted into by default. Set GRUB_DEFAULT in /etc/default/grub to the name of the entry you’ve just created:

$ cat /etc/default/grub
# If you change this file, run 'update-grub' afterwards to update
# /boot/grub/grub.cfg.
# For full documentation of the options in this file, see:
# info -f grub -n 'Simple configuration'

#GRUB_DEFAULT=0
GRUB_DEFAULT="resize"
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT="quiet"
GRUB_CMDLINE_LINUX=""
...

Finally, you can reboot the system. When it comes back online (if it comes back grin) your file system will be resized. Be aware that e2fsck and resizefs take a long time on big disks. On my system the two took 2 hours for a 1.5T filesystem. Thus don’t prematurely reboot your system if it doesn’t come up again quickly.

Don’t forget to remove the new grub entry, so your file system doesn’t get resized every time you boot.

Original howto at https://thunked.org/general/howto-shrink-a-remote-root-ext3-filesystem-t96.html

Original howto Written by Stefan @ https://thunked.org/

This version for Debian wheezy by Tomáš Pospíšek

Celebrating 20 years of Linux!

Celebrating 20 years of Linux!

Celebrating 20 years of Linux!

Celebrating 20 years of Linux!

Visually sorting images under Linux

to visually sort images under Linux doesn’t seem to be a trivial task. I duckduckgo‘ed for a long time and had a look at various image and file managing applications before finding gthumb. And even there, you first need to create a “catalog” and within the catalog a “library” which will finally allow you to manually sort your images. All of which is not documented.

Once you’ve sorted your images, you’d possibly want to export the sorting? Again, no trace of any help or documentation: gthumb catalogs are saved under $HOME/.local/share/gthumb/catalogs/foobar.catalog.

Tomáš Pospíšek

Visually sorting images under Linux

to visually sort images under Linux doesn’t seem to be a trivial task. I duckduckgo‘ed for a long time and had a look at various image and file managing applications before finding gthumb. And even there, you first need to create a “catalog” and within the catalog a “library” which will finally allow you to manually sort your images. All of which is not documented.

Once you’ve sorted your images, you’d possibly want to export the sorting? Again, no trace of any help or documentation: gthumb catalogs are saved under $HOME/.local/share/gthumb/catalogs/foobar.catalog.

Tomáš Pospíšek

Visually sorting images under Linux

to visually sort images under Linux doesn’t seem to be a trivial task. I duckduckgo‘ed for a long time and had a look at various image and file managing applications before finding gthumb. And even there, you first need to create a “catalog” and within the catalog a “library” which will finally allow you to manually sort your images. All of which is not documented.

Once you’ve sorted your images, you’d possibly want to export the sorting? Again, no trace of any help or documentation: gthumb catalogs are saved under $HOME/.local/share/gthumb/catalogs/foobar.catalog.

Tomáš Pospíšek

A workflow for creating beautiful relief shaded dems using GDAL

Sometimes I create hillshades using the QGIS hillshade plugin and then overlay the original DEM over it. I set the DEM to have a false colour pallette and set it to be semi-transparent to produce something like this:

Typical usage of a hillshade with false colour overlay

That is all well and good but a bit impractical. It would be much nice to have the colour pallette permanetly assigned to the hillshade. Also I want to be able to clip and mask the resulting raster to the boundary specified in a shapefile. Fortunately, GDAL provides all the tools you need to make this happen. There are already some nice articles (like this one) that describe parts of this workflow but I am writing this because I wanted to note the additional steps that I took to make this work well for me.

Before you begin

Before you begin you should have:

  1. a raster containing digital elevation data (DEM) - in my case its called 'alt.tif'
  2. a vector layer (typically a shapefile) containing the area of interest for your final product - in my case its called 'tanzania.shp'

Create the hillshade image

The first thing we need to do is generate a hillshade. There is a nice python plugin for doing this in QGIS, you can do it in GRASS using the QGIS-GRASS plugin. But in this article I'm going for an all-GDAL approach so we will be using the handy **gdaldem** command line application.  I won't be explaining the parameters I have used here as they are well explained in the gdaldem manual pages.

So to create our hillshade we do something like this:

::
gdaldem hillshade alt.tif shade.tif -z 5 -s 111120 -az 90

Which will produce something like this: colorbrewer would be a good place to start if you want to learn more.  Another favourite site of mine is colourlovers.com and for this tutorial I decided to use a pallette from there to see how it would turn out.

Once you have selected a suitable colour pallette (around 5 or 6 colour classes should do it), the next thing you need to do is get some information about the range of values contained in your DEM. Once again you can easily point and click your way to this in QGIS, but here is the way to get it in gdal from the command line:

::
gdalinfo -mm alt.tif

The resulting output includes the computed min/max for Band 1 - which is what we are after:

::
Band 1 Block=1334x3 Type=UInt16, ColorInterp=Gray
Computed Min/Max=1.000,5768.000 NoData Value=65535

Ok so our minimum height is 1m and maximum is 5768m - Tanzania is the home of Kilimanjaro after all! So lets split that range up into 5 classes to match the 'Landcarpet Europe' colourlover pallette I selected. I set nearly white as an additional colour for the highest altitude range.

::
65535 255 255 255 5800 254 254 254 3000 121 117 10 1500 151 106 47 800 127 166 122 500 213 213 149 1 218 179 122

The value in the first column is the base of the scale range. The subsequent values are RGB triplets for that range. I saved this as a text file called 'ramp.txt' in the same directory as my 'alt.tiff' dataset. You will notice that I made the value ranges closer together at lower altitudes and wider appart at higher altitudes. You may want to experiment a little to get pleasing results - especially if you have a relatively small number of high lying terrain pixels and the rest confined to lower lying areas.

Also note that I assigned the highest terrain 'nearly white' so that I could reserve white (RGB: 255 255 255) for the nodata value (65535) in this dataset. We will use the fact that white is only used for nodata to our advantage when we do a clip further on in these instructions.

Ok now we can use gdaldem to generate the terrain map:

::
gdaldem color-relief alt.tif ramp.txt relief.tif

This is what my relief map looked like:

The terrain colour map I produced

Don't worry about the fact that it does not resemble the colour pallette you have chosen - it will do in the next step!

Merging the two rasters

The next step is to combine the two products. I used Frank Warmerdam's handy hsv_merge.py script for this purpose.

::
./hsv_merge.py relief.tif shade.tif colour_shade.tif

Which produced this:

The result of merging my hillshade and my terrain colour map

You may have noticed that it is only at this point that the colours of our product resemble the original pallette we used.

One little gotcha with the hsv_merge.py script is that it throws away our nodata values, so what was sea before (and nodata in my original alt.tif dataset) is now white (RGB: 255 255 255).

Clipping and masking

You may have everything you need from the above steps. Alternatively you can also clip and mask the dataset using a shapefile.

::
gdalwarp -co compress=deflate -dstnodata 255 -cutline Tanzania.shp
colour_shade.tif colour_shade_clipped.tif

My final image is now a compressed tif with nodata outside of the country of Tanzania and looks like this:

Final result: A false coloured elevation map for Tanzania

A final note

One of the things I love about the command line is the repeatable and reusable workflows it allows for. I can distill everything in this blog post into a sequence of commands and replay them any time. If you are still stuck doing things in a GUI only way, give BASH a try - you can really do powerful geospatial data processing with it!

  • Page 1 of 1 ( 10 posts )
  • linux

Back to Top

Sponsors