Page 1 of 2 (22 posts)

  • talks about »
  • programming

Tags

Last update:
Fri Nov 15 17:05:17 2019

A Django site.

QGIS Planet

QGIS speaks a lot of languages

QGIS is a real cosmopolitan. Born in Alaska sixteen years ago, it has spread all over the world since. Thanks to its open source mentality, it finds not only in economically strong countries big usergroups. No question, that beside all the developers, there is a bunch of brave translators giving everything to make and keep QGIS multilingual. It’s translated in over forty languages – even to Mandarin Chinese and Esperanto. Not only the application, but also its plugins.
And since the feature-loaded long term release 3.4.0 even the QGS-Projects themselves.  Thanks to the friendly support of QGIS Usergroup Switzerland and the QGEP Project.

How it comes

Plugins are often shipped with pre-configured project files. To provide them in the users individual language, you’ve been required to translate the project manually in the properties and store it separately. When you needed to change something, you have been coerced to update every single file. This is a big effort and fault-prone. So there appeared the idea to have translation files for each required language, and when the user opens the only one project, it will be translated to his specific language. And that’s, what this new functionality does.

How it’s done

Like QGIS and the plugins, the projects are translated with the Qt translation process. Means, it makes the translation according to a Qt Compiled Translation Source File (.qm file). When the user opens a project, QGIS checks for a .qm file laying in the same folder like the .qgs file, having the same name like the .qgs file and having the language-code as postfix of the users language (the language configured in the QGIS settings).
So when the user opens a project named “citybees.qgs” that is originally in English, but his QGIS language is German, it checks for a file named “citybees_de.qm”. The project’s layer names, field-aliases, container names and much more will be translated to German and the project will be automatically stored as “citybees_de.qgs”. So the user has his German project version and can use and edit it like he wishes. Super easy.

Start from the beginning

Swiss people love honey and so they are diligent beekeepers. So, let’s assume you want to provide a project about beekeeping in cities to your Swiss customers. Because in Switzerland people talk four different languages, you need to have the project multilingual in German, French and Italian. We skip Romansh, not because it’s less important or in the Romansh speaking parts are no cities, but because QGIS does not support Romansh (if you’ll propose this one day, you will have my vote).
Anyway. Let’s see what you have to do, to deliver the projects with the .qm file for German, French or Italian.

1. Create the project

You create your project about beekeeping and store it as “citybees.qgs”. You don’t have to care about languages at the moment. You name everything in your language. Assumed it’s English, you name the layers “apiary” and “area” and the fields “fid”, “bee-species”, “beekeeper” and so on.
On changes you will edit always this project and no translated projects.

2. Generate Ts File

In the project properties in the section General there is the part to generate a translation source file. First, you select the source language, to have this information in programs you’ll edit the file (like Qt Linguist or Transifex) afterwards. Per default the language of your QGIS is selected here.

When pressing Generate TS File you will find in your projects folder the new file “citybees.ts”.
But wait a minute, why are we generating a .ts file when we need a .qm file?
The .ts file is the translation source file and it’s the uncompiled .qm file. The .qm file contains compact binary format code Qt can make the translation of programs with, but you and your brave translator cannot read it. So you create a .ts file looking like this:

<!DOCTYPE TS>
<TS sourcelanguage="en_US">
 <context>
  <name>project:layers:apiary__offline__1f89f4fd_49da_4eb9_90b3_1f7b5e82840c</name>
  <message>
   <source>apiary</source>
   <translation type="unfinished"/>
  </message>
 </context>
 <context>
  <name>project:layers:apiary__offline__1f89f4fd_49da_4eb9_90b3_1f7b5e82840c:fieldaliases</name>
  <message>
   <source>fid</source>
   <translation type="unfinished"/>
  </message>
 </context>
 <context>
  <name>project:layers:apiary__offline__1f89f4fd_49da_4eb9_90b3_1f7b5e82840c:fieldaliases</name>
  <message>
   <source>bee_species</source>
   <translation type="unfinished"/>
  </message>
 </context>
 <context>
  <name>project:layers:apiary__offline__1f89f4fd_49da_4eb9_90b3_1f7b5e82840c:fieldaliases</name>
  <message>
   <source>beekeeper</source>
   <translation type="unfinished"/>
  </message>
 </context>
[...]

You see it’s simple XML code that contains mainly untranslated text in the <source> element and empty space for the translated text in the <translation> element.
You could enter your translations directly in this file using the text editor and then build the .qm file with the command lrelease, but it’s preferable to use tools like Qt Linguist orweb-based services like Transifex or Weblate.

3. Translate your File in Qt Linguist

You open Qt Linguist an you select the target language of the .qm file you want to build in the end. Let’s choose German.
The file opens and you see a list of entries described by the Context. The context is, where the strings are located in the QGIS project.

The string “beekeeper” stays in the context project:layers:apiary_1f7b5e82839c:fieldaliases and this means it’s a field or alias of the layer apiary_1f7b5e82839c in the project.
The translation is done simply over the graphical interface. To confirm your translation you can set the check mark.

4. Finally build your .qm file

You compile the translation – means build a .qm file – by simply select Release as… in the Qt Linguist and store it as “citybees_de.qm”.

And now your customer will be able to open your project in German 🙂

What’s translated

Most of the needed parameters like layer names and fields are translated. There could be still some strings in your use case that are not like for example action titles or labels. But the solution is designed, that it’s extendable for more project parameters (see the next chapter). So don’t worry if you will find parameters that cannot be translated yet. They possibly will be in the future.

Translatable strings:

  • layer names
  • layer group names
  • form attributes like tab titles and group box titles
  • relation names
  • field names and aliases
  • value relations

With field names and aliases it has a special behavior: while we should not translate the field names itself because they can be used as identification, the aliases are translated only. In case there is no alias in the original project, the translation of the field name would be written as an alias in the translated project. The field name stays the same. So you can just generate and translate without the fear of overwriting field names.

Content translation

There is a possibility to have a translation of the content as well. In particular using the value relation widget.
Let’s assume, you want to have the values in the field bee_species translated as well. Means the following bee species should be German:
– European dark bee
– Carniolan honey bee
– Buckfast bee

You can solve this by using the value relation widget. Means you create a non geometrical layer “beespecies” and enter the following values:

nomenclatura name_en name_de name_fr name_it
Apis mellifera mellifera European dark bee Dunkle Europäische Biene abeille noire ape nera
Apis mellifera carnica Carniolan honey bee Kärntner Biene abeille carniolienne ape carnica
Apis mellifera Buckfast bee Buckfastbiene abeille Buckfast ape Buckfast

And configure the field bee_species of apiary as value relation widget:

And now comes the magic:

When you create the .ts file it includes not only the field name bee_species but also the referenced value name_en:

 <context>
  <name>project:layers:apiary__offline__1f89f4fd_49da_4eb9_90b3_1f7b5e82840c:fields:bee_species:valuerelationvalue</name>
  <message>
   <source>name_en</source>
   <translation type="unfinished"/>
  </message>
 </context>

When you now “translate” the name_en to name_de the field is referenced to the German values of the entry:

Getting technical

Let’s have a quick look into the source code, shall we?

Generate Ts File

When you press the Generate Ts File button in the project properties, in the background happens the following:
First QGIS scans for the translatable strings in the currently loaded project and collects everything in a QgsTranslationContext object. This object contains the filename of the .ts file and all the collected translatable strings.
The strings are collected by firing a signal called requestForTranslatableObjects delivering the QgsTranslationContext. Every object that contains translatable strings connects a slot registering the strings in the received QgsTranslationContext:

void registerTranslatableObjects( QgsTranslationContext *translationContext )
{
  const QList<QgsLayerTreeLayer *> layers = mRootGroup->findLayers();
  for ( const QgsLayerTreeLayer *layer : layers )
  {
    translationContext->registerTranslation( QStringLiteral( "project:layers:%1" ).arg( layer->layerId() ), layer->name() );
[...]

This will be a growing list of strings as new features are added and missing bits are discovered.

Translate by QTranslator

The translation is made using the QTranslator. It’s loaded with the .qm file on reading the project and on loading of every single translatable string, it’s called to translate:

QString layername = mTranslator->translate( QStringLiteral( "project:layers:%1" ).arg( node.namedItem( QStringLiteral( "id" ) ).toElement().text() ), node.namedItem( QStringLiteral( "layername" ) ).toElement().text(), disambiguation, n );

That’s it

I hope you liked reading and if you have questions or inputs, feel free to add a comment.
Enjoy this cosmopolitious feature! 🙂

Increasing the stability of processing algorithms

Processing just got a new testing framework to improve the long-term stability of this important plugin. And you can help to improve it, even if you are not a software developer! This is yet another piece in our never-stopping crusade to

Syntactic sugar for PyQGIS

PyQGIS now supports a nice new addition for handling edit sessions in layers. This way, changes get committed automatically at the end of a successful (python) edit session.

with edit(layer):
    do your changes here()
See more ›

Generating state machines with Dia

Dia is a nice cross platform application for diagram drawing. It can be scripted via Python, which opens the possibility to generate code from Dia diagrams.

Below we’ll create a python plugin that generates C code from UML state machine diagrams. Doing the same for other languages should be trivial.

The first thing you need is Unai Estébanez Sevilla’s nice finite state machine code generator. The version that we are using here has been abstracted in order to be able to produce code in various languages.

This python plugin implements an exporter to C code. To use it, you need to put it, along with the base exporter into your local plugins directory under ~/.dia/python.

Let’s have a quick look at the code.

First we import the dia python module and the exporter base functionality:

import dia
import uml_stm_export

Then we create our C exporter class that inherits from the generic exporter:

class CDiagramRenderer(uml_stm_export.SimpleSTM):

Next we define how the beginning of our generated code file should look like. That could include general infrastructure independent of the state machine diagram at hand. In our case, we want to encapsulate the generated state machine code within a function:

CODE_PREAMBLE="void config_stm(STM_t* stm) {"

We also define the postamble to close the function. After that come generic functions that implement the class constructor init(self) and functions responsible for calling the dia object parser begin_render(self,data,filename).

Now we define our output generator end_render(self). We first traverse dia’s objects in order to find the state machine’s initial state:

for transition in self.transitions:
   if(transition.source == "INITIAL_STATE"):

The initial state state gets a special treatment: we have a special function call generated for it:

f.write("    add_initial_state( stm, %s, %s );\n" %
    (initial_state.name, initial_state.doaction))

Next we traverse all states and output code that will create them, along with functions to be called within that state to decide on where to transition next:

for key in self.states.keys():
    f.write("    add_state( stm, %s, %s );\n"
        % (state.name, state.doaction))

And finally we output all the transitions between states:

for transition in self.transitions:
    f.write("    add_transition( stm, %s, %s, %s );\n" %
        (transition.source, transition.trigger, transition.target))

and that’s nearly it. At the end of our generator we make sure to register it with dia:

dia.register_export("State Machine Cstma Dump", "c", CDiagramRenderer())

Done! Simple, isn’t it?

Finally please permit me to thank all the people that created such a powerful tool free for us to use:

  • Unai Estébanez Sevilla for the original STM generator
  • Steffen Macke and Hans Breuer, Dia’s current busy maintaners
  • Alexander Larsson, Dia’s original author
  • all the other contributors to Dia and free software
  • Panter for inviting me to their fabulous work week in Greece where most of the hacking on the generator was done and Combitool who supported this work by needing a state machine generator in their current project.

PS: Unai’s original text generator is now also “just” a “simple” addon

Generating state machines with Dia

Dia is a nice cross platform application for diagram drawing. It can be scripted via Python, which opens the possibility to generate code from Dia diagrams.

Below we’ll create a python plugin that generates C code from UML state machine diagrams. Doing the same for other languages should be trivial.

The first thing you need is Unai Estébanez Sevilla’s nice finite state machine code generator. The version that we are using here has been abstracted in order to be able to produce code in various languages.

This python plugin implements an exporter to C code. To use it, you need to put it, along with the base exporter into your local plugins directory under ~/.dia/python.

Let’s have a quick look at the code.

First we import the dia python module and the exporter base functionality:

import dia
import uml_stm_export

Then we create our C exporter class that inherits from the generic exporter:

class CDiagramRenderer(uml_stm_export.SimpleSTM):

Next we define how the beginning of our generated code file should look like. That could include general infrastructure independent of the state machine diagram at hand. In our case, we want to encapsulate the generated state machine code within a function:

CODE_PREAMBLE="void config_stm(STM_t* stm) {"

We also define the postamble to close the function. After that come generic functions that implement the class constructor init(self) and functions responsible for calling the dia object parser begin_render(self,data,filename).

Now we define our output generator end_render(self). We first traverse dia’s objects in order to find the state machine’s initial state:

for transition in self.transitions:
   if(transition.source == "INITIAL_STATE"):

The initial state state gets a special treatment: we have a special function call generated for it:

f.write("    add_initial_state( stm, %s, %s );\n" %
    (initial_state.name, initial_state.doaction))

Next we traverse all states and output code that will create them, along with functions to be called within that state to decide on where to transition next:

for key in self.states.keys():
    f.write("    add_state( stm, %s, %s );\n"
        % (state.name, state.doaction))

And finally we output all the transitions between states:

for transition in self.transitions:
    f.write("    add_transition( stm, %s, %s, %s );\n" %
        (transition.source, transition.trigger, transition.target))

and that’s nearly it. At the end of our generator we make sure to register it with dia:

dia.register_export("State Machine Cstma Dump", "c", CDiagramRenderer())

Done! Simple, isn’t it?

Finally please permit me to thank all the people that created such a powerful tool free for us to use:

  • Unai Estébanez Sevilla for the original STM generator
  • Steffen Macke and Hans Breuer, Dia’s current busy maintaners
  • Alexander Larsson, Dia’s original author
  • all the other contributors to Dia and free software
  • Panter for inviting me to their fabulous work week in Greece where most of the hacking on the generator was done and Combitool who supported this work by needing a state machine generator in their current project.

PS: Unai’s original text generator is now also “just” a “simple” addon

Reducing magic in Ruby's Forwardable class implementation

Abstract

Using string-eval in Ruby for metaprogramming is unnecessarily obscuring. Ruby’s more modern and specific metaprogramming methods should be used instead whenever possible. This problem is illustrated on the example of Ruby’s Forwardable class.

In detail…

Ruby’s Forwardable class is using metaprogramming to forward calls from a frontend interface to an instance in the back executing the call.

Metaprogramming is the discipline of making code that creates code. This task allready is rather abstract and hard to grasp in itself. Having hard to grasp code is a liability. One of the goals of writing code is allways to keep the code as simple and as well understandable as possible.

Additionaly, metaprogramming code itself is difficult to read and understand: that is because the metaprogramming code will not necessarily express what the code it is creating is about, but only how it is creating that code. As such the code it is creating can be invisible to you as a reader of the source code - the created code will only start to exist at runtime.

One would therefore expect that programmers would try especially hard when they metaprogram to make that particular kind of code expressive and easy to understand.

Another consequence of the fact that the code produced by metaprogramming is not necessarily visible, is that debugging becomes more difficult: when analyzing problems you’ll not only be unsure how the programm works, but in addition, you won’t even be sure how the code that is executed looks like - since it is only generated at runtime.

This post is focusing on the last problem: debugging of metaprogrammed code.

There are two approaches to metaprogramming. One is to have as far as possible compile-time parseable code and the other is to let the code only be parsed at runtime.

As of version 1.9.2, Ruby’s Forwardable class is using the latter. The metaprogramming code in Ruby 1.8.7 looks like this:

    module_eval(<

As said, this has the consequence of the metaprogrammed code being completely invisible to the parser and other tools such as editors and debuggers.

This results in the following:

$ cat queue.rb
require 'rubygems'
require 'forwardable'
require 'ruby-debug'

class Queue
  extend Forwardable

  def initialize
    @q = [ ]    # prepare delegate object
  end

  # setup preferred interface, enq() and deq()...
  def_delegator :@q, :push, :enq
  def_delegator :@q, :shift, :deq

  # support some general Array methods that fit Queues well
  def_delegators :@q, :clear, :first, :push, :shift, :size
end

q = Queue.new
debugger # ------ DEBUGGING FROM HERE ON -----
q.enq 1, 2, 3, 4, 5
q.push 6

q.shift    # => 1
while q.size > 0
  puts q.deq
end

q.enq "Ruby", "Perl", "Python"
puts q.first
q.clear
puts q.first


$ ruby queue.rb

queue.rb:24
q.enq 1, 2, 3, 4, 5

(rdb:1) step
(__FORWARDABLE__):2

(rdb:1) list =
*** No sourcefile available for (__FORWARDABLE__)

(rdb:1) step
(__FORWARDABLE__):3

In other words, you are rather lost allready - otherwise you probably wouldn’t be stepping through your code - and in that situation it happens that your debugger gets completely lost as well, since it does not know any more where in the code it is and what it exactly is executing.

That’s nothing the programmer wishes for. In a situation where you are debugging you want to have a maximally clear view of all state, including what code you are currently executing.

Chaning that situation requires making as much of the metaprogrammed code visible to the parser, which is the second approach to metaprogramming mentioned previously:

$ cat forwardable2.rb
...
    self.send(:define_method, ali) do |*args,█|
      begin
        instance_variable_get(accessor).__send__(method, *args,█)
      rescue Exception
        [email protected]_if{|s| /^\\(__FORWARDABLE__\\):/ =~ s} unless Forwardable2::debug
        Kernel::raise
      end
    end

Note that it’s the same code as before, except that we do not do eval("string") any more, but instead are using specific, more modern metaprogramming tools provided by standard Ruby.

The result is the following:

$ ruby queue.rb

queue.rb:22
q.enq 1, 2, 3, 4, 5

(rdb:1) step
/usr/lib/ruby/1.8/forwardable2.rb:149
begin

(rdb:1) list =
[144, 153] in /usr/lib/ruby/1.8/forwardable2.rb
   144      accessor = accessor.id2name if accessor.kind_of?(Integer)
   145      method = method.id2name if method.kind_of?(Integer)
   146      ali = ali.id2name if ali.kind_of?(Integer)
   147
   148      self.send(:define_method, ali) do |*args,█|
=> 149        begin
   150          instance_variable_get(accessor).__send__(method, *args,█)
   151        rescue Exception
   152          [email protected]_if{|s| /^\\(__FORWARDABLE__\\):/ =~ s} unless Forwardable2::debug
   153          Kernel::raise

(rdb:1) step
/usr/lib/ruby/1.8/forwardable2.rb:150
instance_variable_get(accessor).__send__(method, *args,█)

Allready much, much better.

Of course, with the string-eval approach to metaprogramming Ruby itself could do better by saving the string that is being evaled to be able to refer to it later at step-through time. However currently we don’t have this option.

Tomáš Pospíšek

Reducing magic in Ruby's Forwardable class implementation

Abstract

Using string-eval in Ruby for metaprogramming is unnecessarily obscuring. Ruby’s more modern and specific metaprogramming methods should be used instead whenever possible. This problem is illustrated on the example of Ruby’s Forwardable class.

In detail…

Ruby’s Forwardable class is using metaprogramming to forward calls from a frontend interface to an instance in the back executing the call.

Metaprogramming is the discipline of making code that creates code. This task allready is rather abstract and hard to grasp in itself. Having hard to grasp code is a liability. One of the goals of writing code is allways to keep the code as simple and as well understandable as possible.

Additionaly, metaprogramming code itself is difficult to read and understand: that is because the metaprogramming code will not necessarily express what the code it is creating is about, but only how it is creating that code. As such the code it is creating can be invisible to you as a reader of the source code - the created code will only start to exist at runtime.

One would therefore expect that programmers would try especially hard when they metaprogram to make that particular kind of code expressive and easy to understand.

Another consequence of the fact that the code produced by metaprogramming is not necessarily visible, is that debugging becomes more difficult: when analyzing problems you’ll not only be unsure how the programm works, but in addition, you won’t even be sure how the code that is executed looks like - since it is only generated at runtime.

This post is focusing on the last problem: debugging of metaprogrammed code.

There are two approaches to metaprogramming. One is to have as far as possible compile-time parseable code and the other is to let the code only be parsed at runtime.

As of version 1.9.2, Ruby’s Forwardable class is using the latter. The metaprogramming code in Ruby 1.8.7 looks like this:

    module_eval(<

As said, this has the consequence of the metaprogrammed code being completely invisible to the parser and other tools such as editors and debuggers.

This results in the following:

$ cat queue.rb
require 'rubygems'
require 'forwardable'
require 'ruby-debug'

class Queue
  extend Forwardable

  def initialize
    @q = [ ]    # prepare delegate object
  end

  # setup preferred interface, enq() and deq()...
  def_delegator :@q, :push, :enq
  def_delegator :@q, :shift, :deq

  # support some general Array methods that fit Queues well
  def_delegators :@q, :clear, :first, :push, :shift, :size
end

q = Queue.new
debugger # ------ DEBUGGING FROM HERE ON -----
q.enq 1, 2, 3, 4, 5
q.push 6

q.shift    # => 1
while q.size > 0
  puts q.deq
end

q.enq "Ruby", "Perl", "Python"
puts q.first
q.clear
puts q.first


$ ruby queue.rb

queue.rb:24
q.enq 1, 2, 3, 4, 5

(rdb:1) step
(__FORWARDABLE__):2

(rdb:1) list =
*** No sourcefile available for (__FORWARDABLE__)

(rdb:1) step
(__FORWARDABLE__):3

In other words, you are rather lost allready - otherwise you probably wouldn’t be stepping through your code - and in that situation it happens that your debugger gets completely lost as well, since it does not know any more where in the code it is and what it exactly is executing.

That’s nothing the programmer wishes for. In a situation where you are debugging you want to have a maximally clear view of all state, including what code you are currently executing.

Chaning that situation requires making as much of the metaprogrammed code visible to the parser, which is the second approach to metaprogramming mentioned previously:

$ cat forwardable2.rb
...
    self.send(:define_method, ali) do |*args,█|
      begin
        instance_variable_get(accessor).__send__(method, *args,█)
      rescue Exception
        [email protected]_if{|s| /^\\(__FORWARDABLE__\\):/ =~ s} unless Forwardable2::debug
        Kernel::raise
      end
    end

Note that it’s the same code as before, except that we do not do eval("string") any more, but instead are using specific, more modern metaprogramming tools provided by standard Ruby.

The result is the following:

$ ruby queue.rb

queue.rb:22
q.enq 1, 2, 3, 4, 5

(rdb:1) step
/usr/lib/ruby/1.8/forwardable2.rb:149
begin

(rdb:1) list =
[144, 153] in /usr/lib/ruby/1.8/forwardable2.rb
   144      accessor = accessor.id2name if accessor.kind_of?(Integer)
   145      method = method.id2name if method.kind_of?(Integer)
   146      ali = ali.id2name if ali.kind_of?(Integer)
   147
   148      self.send(:define_method, ali) do |*args,█|
=> 149        begin
   150          instance_variable_get(accessor).__send__(method, *args,█)
   151        rescue Exception
   152          [email protected]_if{|s| /^\\(__FORWARDABLE__\\):/ =~ s} unless Forwardable2::debug
   153          Kernel::raise

(rdb:1) step
/usr/lib/ruby/1.8/forwardable2.rb:150
instance_variable_get(accessor).__send__(method, *args,█)

Allready much, much better.

Of course, with the string-eval approach to metaprogramming Ruby itself could do better by saving the string that is being evaled to be able to refer to it later at step-through time. However currently we don’t have this option.

Tomáš Pospíšek

Reducing magic in Ruby's Forwardable class implementation

Abstract

Using string-eval in Ruby for metaprogramming is unnecessarily obscuring. Ruby’s more modern and specific metaprogramming methods should be used instead whenever possible. This problem is illustrated on the example of Ruby’s Forwardable class.

In detail…

Ruby’s Forwardable class is using metaprogramming to forward calls from a frontend interface to an instance in the back executing the call.

Metaprogramming is the discipline of making code that creates code. This task allready is rather abstract and hard to grasp in itself. Having hard to grasp code is a liability. One of the goals of writing code is allways to keep the code as simple and as well understandable as possible.

Additionaly, metaprogramming code itself is difficult to read and understand: that is because the metaprogramming code will not necessarily express what the code it is creating is about, but only how it is creating that code. As such the code it is creating can be invisible to you as a reader of the source code - the created code will only start to exist at runtime.

One would therefore expect that programmers would try especially hard when they metaprogram to make that particular kind of code expressive and easy to understand.

Another consequence of the fact that the code produced by metaprogramming is not necessarily visible, is that debugging becomes more difficult: when analyzing problems you’ll not only be unsure how the programm works, but in addition, you won’t even be sure how the code that is executed looks like - since it is only generated at runtime.

This post is focusing on the last problem: debugging of metaprogrammed code.

There are two approaches to metaprogramming. One is to have as far as possible compile-time parseable code and the other is to let the code only be parsed at runtime.

As of version 1.9.2, Ruby’s Forwardable class is using the latter. The metaprogramming code in Ruby 1.8.7 looks like this:

    module_eval(<

As said, this has the consequence of the metaprogrammed code being completely invisible to the parser and other tools such as editors and debuggers.

This results in the following:

$ cat queue.rb
require 'rubygems'
require 'forwardable'
require 'ruby-debug'

class Queue
  extend Forwardable

  def initialize
    @q = [ ]    # prepare delegate object
  end

  # setup preferred interface, enq() and deq()...
  def_delegator :@q, :push, :enq
  def_delegator :@q, :shift, :deq

  # support some general Array methods that fit Queues well
  def_delegators :@q, :clear, :first, :push, :shift, :size
end

q = Queue.new
debugger # ------ DEBUGGING FROM HERE ON -----
q.enq 1, 2, 3, 4, 5
q.push 6

q.shift    # => 1
while q.size > 0
  puts q.deq
end

q.enq "Ruby", "Perl", "Python"
puts q.first
q.clear
puts q.first


$ ruby queue.rb

queue.rb:24
q.enq 1, 2, 3, 4, 5

(rdb:1) step
(__FORWARDABLE__):2

(rdb:1) list =
*** No sourcefile available for (__FORWARDABLE__)

(rdb:1) step
(__FORWARDABLE__):3

In other words, you are rather lost allready - otherwise you probably wouldn’t be stepping through your code - and in that situation it happens that your debugger gets completely lost as well, since it does not know any more where in the code it is and what it exactly is executing.

That’s nothing the programmer wishes for. In a situation where you are debugging you want to have a maximally clear view of all state, including what code you are currently executing.

Chaning that situation requires making as much of the metaprogrammed code visible to the parser, which is the second approach to metaprogramming mentioned previously:

$ cat forwardable2.rb
...
    self.send(:define_method, ali) do |*args,█|
      begin
        instance_variable_get(accessor).__send__(method, *args,█)
      rescue Exception
        [email protected]_if{|s| /^\\(__FORWARDABLE__\\):/ =~ s} unless Forwardable2::debug
        Kernel::raise
      end
    end

Note that it’s the same code as before, except that we do not do eval("string") any more, but instead are using specific, more modern metaprogramming tools provided by standard Ruby.

The result is the following:

$ ruby queue.rb

queue.rb:22
q.enq 1, 2, 3, 4, 5

(rdb:1) step
/usr/lib/ruby/1.8/forwardable2.rb:149
begin

(rdb:1) list =
[144, 153] in /usr/lib/ruby/1.8/forwardable2.rb
   144      accessor = accessor.id2name if accessor.kind_of?(Integer)
   145      method = method.id2name if method.kind_of?(Integer)
   146      ali = ali.id2name if ali.kind_of?(Integer)
   147
   148      self.send(:define_method, ali) do |*args,█|
=> 149        begin
   150          instance_variable_get(accessor).__send__(method, *args,█)
   151        rescue Exception
   152          [email protected]_if{|s| /^\\(__FORWARDABLE__\\):/ =~ s} unless Forwardable2::debug
   153          Kernel::raise

(rdb:1) step
/usr/lib/ruby/1.8/forwardable2.rb:150
instance_variable_get(accessor).__send__(method, *args,█)

Allready much, much better.

Of course, with the string-eval approach to metaprogramming Ruby itself could do better by saving the string that is being evaled to be able to refer to it later at step-through time. However currently we don’t have this option.

Tomáš Pospíšek

Reducing magic in Ruby's Forwardable class implementation

Abstract

Using string-eval in Ruby for metaprogramming is unnecessarily obscuring. Ruby’s more modern and specific metaprogramming methods should be used instead whenever possible. This problem is illustrated on the example of Ruby’s Forwardable class.

In detail…

Ruby’s Forwardable class is using metaprogramming to forward calls from a frontend interface to an instance in the back executing the call.

Metaprogramming is the discipline of making code that creates code. This task allready is rather abstract and hard to grasp in itself. Having hard to grasp code is a liability. One of the goals of writing code is allways to keep the code as simple and as well understandable as possible.

Additionaly, metaprogramming code itself is difficult to read and understand: that is because the metaprogramming code will not necessarily express what the code it is creating is about, but only how it is creating that code. As such the code it is creating can be invisible to you as a reader of the source code - the created code will only start to exist at runtime.

One would therefore expect that programmers would try especially hard when they metaprogram to make that particular kind of code expressive and easy to understand.

Another consequence of the fact that the code produced by metaprogramming is not necessarily visible, is that debugging becomes more difficult: when analyzing problems you’ll not only be unsure how the programm works, but in addition, you won’t even be sure how the code that is executed looks like - since it is only generated at runtime.

This post is focusing on the last problem: debugging of metaprogrammed code.

There are two approaches to metaprogramming. One is to have as far as possible compile-time parseable code and the other is to let the code only be parsed at runtime.

As of version 1.9.2, Ruby’s Forwardable class is using the latter. The metaprogramming code in Ruby 1.8.7 looks like this:

    module_eval(<

As said, this has the consequence of the metaprogrammed code being completely invisible to the parser and other tools such as editors and debuggers.

This results in the following:

$ cat queue.rb
require 'rubygems'
require 'forwardable'
require 'ruby-debug'

class Queue
  extend Forwardable

  def initialize
    @q = [ ]    # prepare delegate object
  end

  # setup preferred interface, enq() and deq()...
  def_delegator :@q, :push, :enq
  def_delegator :@q, :shift, :deq

  # support some general Array methods that fit Queues well
  def_delegators :@q, :clear, :first, :push, :shift, :size
end

q = Queue.new
debugger # ------ DEBUGGING FROM HERE ON -----
q.enq 1, 2, 3, 4, 5
q.push 6

q.shift    # => 1
while q.size > 0
  puts q.deq
end

q.enq "Ruby", "Perl", "Python"
puts q.first
q.clear
puts q.first


$ ruby queue.rb

queue.rb:24
q.enq 1, 2, 3, 4, 5

(rdb:1) step
(__FORWARDABLE__):2

(rdb:1) list =
*** No sourcefile available for (__FORWARDABLE__)

(rdb:1) step
(__FORWARDABLE__):3

In other words, you are rather lost allready - otherwise you probably wouldn’t be stepping through your code - and in that situation it happens that your debugger gets completely lost as well, since it does not know any more where in the code it is and what it exactly is executing.

That’s nothing the programmer wishes for. In a situation where you are debugging you want to have a maximally clear view of all state, including what code you are currently executing.

Chaning that situation requires making as much of the metaprogrammed code visible to the parser, which is the second approach to metaprogramming mentioned previously:

$ cat forwardable2.rb
...
    self.send(:define_method, ali) do |*args,█|
      begin
        instance_variable_get(accessor).__send__(method, *args,█)
      rescue Exception
        [email protected]_if{|s| /^\\(__FORWARDABLE__\\):/ =~ s} unless Forwardable2::debug
        Kernel::raise
      end
    end

Note that it’s the same code as before, except that we do not do eval("string") any more, but instead are using specific, more modern metaprogramming tools provided by standard Ruby.

The result is the following:

$ ruby queue.rb

queue.rb:22
q.enq 1, 2, 3, 4, 5

(rdb:1) step
/usr/lib/ruby/1.8/forwardable2.rb:149
begin

(rdb:1) list =
[144, 153] in /usr/lib/ruby/1.8/forwardable2.rb
   144      accessor = accessor.id2name if accessor.kind_of?(Integer)
   145      method = method.id2name if method.kind_of?(Integer)
   146      ali = ali.id2name if ali.kind_of?(Integer)
   147
   148      self.send(:define_method, ali) do |*args,█|
=> 149        begin
   150          instance_variable_get(accessor).__send__(method, *args,█)
   151        rescue Exception
   152          [email protected]_if{|s| /^\\(__FORWARDABLE__\\):/ =~ s} unless Forwardable2::debug
   153          Kernel::raise

(rdb:1) step
/usr/lib/ruby/1.8/forwardable2.rb:150
instance_variable_get(accessor).__send__(method, *args,█)

Allready much, much better.

Of course, with the string-eval approach to metaprogramming Ruby itself could do better by saving the string that is being evaled to be able to refer to it later at step-through time. However currently we don’t have this option.

Tomáš Pospíšek

extending RedCloth markup

There are various approaches when trying to extend the Textile markup that RedCloth understands with own tags or syntax. Some approaches documented on the net have changed or don’t work any more, since RedCloth has been rewritten in Version 4.

Below is a fairly robust aproach, that is based on the assumption, that RedCloth leaves HTML tags inside the markup untouched and passes them on to the application consuming the translated markup.

The below code has been used in the Madek project. Madek, a Ruby On Rails application, uses irwi which provides a Wiki to Madek. And finally irwi uses RedCloth to do the rendering from Textile to HTML. That’s where we hook in.

We want to have a few special tags, which make the life of the Madek Wiki admin simpler, by letting him write the following inside the Textile markup:

[media=210      | Das Huhn]
[screenshot=210 | Das Huhn]
[video=210      | Das Huhn]

That will finally produce this HTML output:

<a href="/media_entries/210">Das Huhn</a>
<img src="/media_entries/210/image" title="Das Huhn"/>
<video src="/media_entries/210/image" title="Das Huhn"/>
  <a href='/media_entries/210'>(see video)</a>
</video>

Here’s the implementation:

class RedClothMadek

  ActionView::Base.sanitized_allowed_tags << 'video'

  def initialize
    require 'redcloth'
  end

  def format( text )
    ::RedCloth.new( replace_madek_tags(text) ).to_html
  end

  # Transforms the follwing Textile markups:
  # 
  #   [media=210      | Das Huhn] -> <a href="/media_entries/210">Das Huhn</a>
  #   [screenshot=210 | Das Huhn] -> <img src="/media_entries/210/image" title="Das Huhn"/>
  #   [video=210      | Das Huhn] -> <video src="/media_entries/210/image" title="Das Huhn"/>
  #                                    <a href='/media_entries/210'>(see video)</a>
  #                                  </video>
  #
  def replace_madek_tags( text )

    # unfortunately having multiple matches in gsub doesn't seem to work, therefore
    # we fall back to $1 $2
    #
    text.gsub(/[\s*media\s*=\s*(\d+)\s*\|\s*([^]]+)\s*]/) { |number,txt|

           "<a href='/media_entries/#{$1}'>#{h($2)}</a>"                 }.

         gsub(/[\s*screenshot\s*=\s*(\d+)\s*\|\s*([^]]+)\s*]/) { |number,title|

           "<img src='/media_entries/#{$1}/image' title='#{h($2)}'/>"    }.

         gsub(/[\s*video\s*=\s*(\d+)\s*\|\s*([^]]+)\s*]/) { |number,title|
           "<video src='/media_entries/#{$1}/image' title='#{h($2)}'>" +
             "<a href='/media_entries/#{$1}'>(see Wideo)</a>" +
           "</video>"  }
  end

end

And finally, irwi needs to be told to use that formatter instead of RedCloth directly. From config/environment.rb:

require "#{Rails.root}/lib/red_cloth_madek.rb"
Irwi.config.formatter = RedClothMadek.new

Tomáš Pospíšek

extending RedCloth markup

There are various approaches when trying to extend the Textile markup that RedCloth understands with own tags or syntax. Some approaches documented on the net have changed or don’t work any more, since RedCloth has been rewritten in Version 4.

Below is a fairly robust aproach, that is based on the assumption, that RedCloth leaves HTML tags inside the markup untouched and passes them on to the application consuming the translated markup.

The below code has been used in the Madek project. Madek, a Ruby On Rails application, uses irwi which provides a Wiki to Madek. And finally irwi uses RedCloth to do the rendering from Textile to HTML. That’s where we hook in.

We want to have a few special tags, which make the life of the Madek Wiki admin simpler, by letting him write the following inside the Textile markup:

[media=210      | Das Huhn]
[screenshot=210 | Das Huhn]
[video=210      | Das Huhn]

That will finally produce this HTML output:

<a href="/media_entries/210">Das Huhn</a>
<img src="/media_entries/210/image" title="Das Huhn"/>
<video src="/media_entries/210/image" title="Das Huhn"/>
  <a href='/media_entries/210'>(see video)</a>
</video>

Here’s the implementation:

class RedClothMadek

  ActionView::Base.sanitized_allowed_tags << 'video'

  def initialize
    require 'redcloth'
  end

  def format( text )
    ::RedCloth.new( replace_madek_tags(text) ).to_html
  end

  # Transforms the follwing Textile markups:
  # 
  #   [media=210      | Das Huhn] -> <a href="/media_entries/210">Das Huhn</a>
  #   [screenshot=210 | Das Huhn] -> <img src="/media_entries/210/image" title="Das Huhn"/>
  #   [video=210      | Das Huhn] -> <video src="/media_entries/210/image" title="Das Huhn"/>
  #                                    <a href='/media_entries/210'>(see video)</a>
  #                                  </video>
  #
  def replace_madek_tags( text )

    # unfortunately having multiple matches in gsub doesn't seem to work, therefore
    # we fall back to $1 $2
    #
    text.gsub(/[\s*media\s*=\s*(\d+)\s*\|\s*([^]]+)\s*]/) { |number,txt|

           "<a href='/media_entries/#{$1}'>#{h($2)}</a>"                 }.

         gsub(/[\s*screenshot\s*=\s*(\d+)\s*\|\s*([^]]+)\s*]/) { |number,title|

           "<img src='/media_entries/#{$1}/image' title='#{h($2)}'/>"    }.

         gsub(/[\s*video\s*=\s*(\d+)\s*\|\s*([^]]+)\s*]/) { |number,title|
           "<video src='/media_entries/#{$1}/image' title='#{h($2)}'>" +
             "<a href='/media_entries/#{$1}'>(see Wideo)</a>" +
           "</video>"  }
  end

end

And finally, irwi needs to be told to use that formatter instead of RedCloth directly. From config/environment.rb:

require "#{Rails.root}/lib/red_cloth_madek.rb"
Irwi.config.formatter = RedClothMadek.new

Tomáš Pospíšek

extending RedCloth markup

There are various approaches when trying to extend the Textile markup that RedCloth understands with own tags or syntax. Some approaches documented on the net have changed or don’t work any more, since RedCloth has been rewritten in Version 4.

Below is a fairly robust aproach, that is based on the assumption, that RedCloth leaves HTML tags inside the markup untouched and passes them on to the application consuming the translated markup.

The below code has been used in the Madek project. Madek, a Ruby On Rails application, uses irwi which provides a Wiki to Madek. And finally irwi uses RedCloth to do the rendering from Textile to HTML. That’s where we hook in.

We want to have a few special tags, which make the life of the Madek Wiki admin simpler, by letting him write the following inside the Textile markup:

[media=210      | Das Huhn]
[screenshot=210 | Das Huhn]
[video=210      | Das Huhn]

That will finally produce this HTML output:

<a href="/media_entries/210">Das Huhn</a>
<img src="/media_entries/210/image" title="Das Huhn"/>
<video src="/media_entries/210/image" title="Das Huhn"/>
  <a href='/media_entries/210'>(see video)</a>
</video>

Here’s the implementation:

class RedClothMadek

  ActionView::Base.sanitized_allowed_tags << 'video'

  def initialize
    require 'redcloth'
  end

  def format( text )
    ::RedCloth.new( replace_madek_tags(text) ).to_html
  end

  # Transforms the follwing Textile markups:
  # 
  #   [media=210      | Das Huhn] -> <a href="/media_entries/210">Das Huhn</a>
  #   [screenshot=210 | Das Huhn] -> <img src="/media_entries/210/image" title="Das Huhn"/>
  #   [video=210      | Das Huhn] -> <video src="/media_entries/210/image" title="Das Huhn"/>
  #                                    <a href='/media_entries/210'>(see video)</a>
  #                                  </video>
  #
  def replace_madek_tags( text )

    # unfortunately having multiple matches in gsub doesn't seem to work, therefore
    # we fall back to $1 $2
    #
    text.gsub(/[\s*media\s*=\s*(\d+)\s*\|\s*([^]]+)\s*]/) { |number,txt|

           "<a href='/media_entries/#{$1}'>#{h($2)}</a>"                 }.

         gsub(/[\s*screenshot\s*=\s*(\d+)\s*\|\s*([^]]+)\s*]/) { |number,title|

           "<img src='/media_entries/#{$1}/image' title='#{h($2)}'/>"    }.

         gsub(/[\s*video\s*=\s*(\d+)\s*\|\s*([^]]+)\s*]/) { |number,title|
           "<video src='/media_entries/#{$1}/image' title='#{h($2)}'>" +
             "<a href='/media_entries/#{$1}'>(see Wideo)</a>" +
           "</video>"  }
  end

end

And finally, irwi needs to be told to use that formatter instead of RedCloth directly. From config/environment.rb:

require "#{Rails.root}/lib/red_cloth_madek.rb"
Irwi.config.formatter = RedClothMadek.new

Tomáš Pospíšek

annotating third party web pages

Problem: I have a web site/page that I visit regularily which I want to annotate with my notes.

More specifically, I was regularly searching through the Homegate real estate hub looking for a new home. It goes without saying that I was again and again forgetting which objects I had already looked at, which objects were really interesting and I should check out more closely.

Therefore the need to annotate search results.

The here presented approach should be applicable for annotation of other web pages as well. It’s based on the observation, that restful web applications need to operate with asset IDs. These asset IDs can be reused to enrich the asset locally with additional data, such as notes. Thus we’re looking for those IDs in specific elements of the page and add a bit of HTML markup to those places.

Of course that aproach only works as long as the web site doesn’t heavily change its markup and doesn’t rendomly change asset IDs.

Here’s a screenshot of regular Homegate search results:

And here’s the same page after scripting it with Greasemonkey:

You’ll notice the input field with the comment in it.

The idea is simple: add an input box to each search result, where you write your comment. When the focus leaves the input box, the comment is stored to localStorage.

Starting with Greasemonkey is quite easy. There are howtos and templates to start from, such as this one.

However, regular JavaScripting and Greasemonkey JavaScripting do not work in exactly the same way:

One of the differences is that Greasemonkey creates a separate JavaScript environment, in which the Greasemonkey scripts are executed. That is calling Javascript contained in the page from Greasemonkey and the inverse calling Greasemonkey scripts from the page is not possible by default. This is on purpose, so that the web page can not detect and not interfere with the Greasemonkey scripts, because you want your Greasemonkey scripts to work allways on some page, whether or not that page likes it or not. Therefore no interference is possible.

Greasemonkey however provides a standard way to access the web page’s scripts, and that’s through the “unsafeWindow” object, which is a reference to the web page’s environment.

I had two mechanisms I had to make accessible using the “unsafeWindow” handle:

  • the first was accessing JQuery, which is included by default by the Homegate page. Since I needed to use JQuery functions in my Greasemonkey script, I got a reference to it via the standard Greasemonkey precedure:
    var jQuery = unsafeWindow['jQuery'];
  • the second mechanism that needed to cross the boundaries between the web page and Greasemonkey was callbacks from the web page to my Greasemonkey script. This is necessary, because I’m attaching “input” elements to each search result, which contain a note and which, “onblur”, need to call a function that saves the content of the input box. Here’s part that constructs the input element:
    jQuery("<input onclick='event.cancelBubble = true;'" +
                 " onblur='saveComment(this, immoID);'>").insertAfter(immoElement);

And this is the function that gets called back by “onblur”:

    unsafeWindow.saveComment = function(element, immoID) {
      unsafeWindow.localStorage.setItem(immoID, element.value);
    };

The next interesting thing you’ll note is usage of ‘locaStorage’. Support for the latter in browsers does not seem mature yet. One problem I’ve encountered when developing under Firefox 3.6 was that saving to ‘localStorage’ was not possible when cookies were disabled (see this report). Thus you’ll need to permanently enable cookies for Homegate in order for the script to be able to save its data.

Finally, while developing, a major problem was, that Firefox did not show me errors in the Greasemonkey scripts. Thus either the script would work or not work and fail completely silently. That made debugging a bit painful.

So now, here’s the script.

Tomáš Pospíšek

PS: This script also lives at userscripts.org

annotating third party web pages

Problem: I have a web site/page that I visit regularily which I want to annotate with my notes.

More specifically, I was regularly searching through the Homegate real estate hub looking for a new home. It goes without saying that I was again and again forgetting which objects I had already looked at, which objects were really interesting and I should check out more closely.

Therefore the need to annotate search results.

The here presented approach should be applicable for annotation of other web pages as well. It’s based on the observation, that restful web applications need to operate with asset IDs. These asset IDs can be reused to enrich the asset locally with additional data, such as notes. Thus we’re looking for those IDs in specific elements of the page and add a bit of HTML markup to those places.

Of course that aproach only works as long as the web site doesn’t heavily change its markup and doesn’t rendomly change asset IDs.

Here’s a screenshot of regular Homegate search results:

And here’s the same page after scripting it with Greasemonkey:

You’ll notice the input field with the comment in it.

The idea is simple: add an input box to each search result, where you write your comment. When the focus leaves the input box, the comment is stored to localStorage.

Starting with Greasemonkey is quite easy. There are howtos and templates to start from, such as this one.

However, regular JavaScripting and Greasemonkey JavaScripting do not work in exactly the same way:

One of the differences is that Greasemonkey creates a separate JavaScript environment, in which the Greasemonkey scripts are executed. That is calling Javascript contained in the page from Greasemonkey and the inverse calling Greasemonkey scripts from the page is not possible by default. This is on purpose, so that the web page can not detect and not interfere with the Greasemonkey scripts, because you want your Greasemonkey scripts to work allways on some page, whether or not that page likes it or not. Therefore no interference is possible.

Greasemonkey however provides a standard way to access the web page’s scripts, and that’s through the “unsafeWindow” object, which is a reference to the web page’s environment.

I had two mechanisms I had to make accessible using the “unsafeWindow” handle:

  • the first was accessing JQuery, which is included by default by the Homegate page. Since I needed to use JQuery functions in my Greasemonkey script, I got a reference to it via the standard Greasemonkey precedure:
    var jQuery = unsafeWindow['jQuery'];
  • the second mechanism that needed to cross the boundaries between the web page and Greasemonkey was callbacks from the web page to my Greasemonkey script. This is necessary, because I’m attaching “input” elements to each search result, which contain a note and which, “onblur”, need to call a function that saves the content of the input box. Here’s part that constructs the input element:
    jQuery("<input onclick='event.cancelBubble = true;'" +
                 " onblur='saveComment(this, immoID);'>").insertAfter(immoElement);

And this is the function that gets called back by “onblur”:

    unsafeWindow.saveComment = function(element, immoID) {
      unsafeWindow.localStorage.setItem(immoID, element.value);
    };

The next interesting thing you’ll note is usage of ‘locaStorage’. Support for the latter in browsers does not seem mature yet. One problem I’ve encountered when developing under Firefox 3.6 was that saving to ‘localStorage’ was not possible when cookies were disabled (see this report). Thus you’ll need to permanently enable cookies for Homegate in order for the script to be able to save its data.

Finally, while developing, a major problem was, that Firefox did not show me errors in the Greasemonkey scripts. Thus either the script would work or not work and fail completely silently. That made debugging a bit painful.

So now, here’s the script.

Tomáš Pospíšek

PS: This script also lives at userscripts.org

annotating third party web pages

Problem: I have a web site/page that I visit regularily which I want to annotate with my notes.

More specifically, I was regularly searching through the Homegate real estate hub looking for a new home. It goes without saying that I was again and again forgetting which objects I had already looked at, which objects were really interesting and I should check out more closely.

Therefore the need to annotate search results.

The here presented approach should be applicable for annotation of other web pages as well. It’s based on the observation, that restful web applications need to operate with asset IDs. These asset IDs can be reused to enrich the asset locally with additional data, such as notes. Thus we’re looking for those IDs in specific elements of the page and add a bit of HTML markup to those places.

Of course that aproach only works as long as the web site doesn’t heavily change its markup and doesn’t rendomly change asset IDs.

Here’s a screenshot of regular Homegate search results:

And here’s the same page after scripting it with Greasemonkey:

You’ll notice the input field with the comment in it.

The idea is simple: add an input box to each search result, where you write your comment. When the focus leaves the input box, the comment is stored to localStorage.

Starting with Greasemonkey is quite easy. There are howtos and templates to start from, such as this one.

However, regular JavaScripting and Greasemonkey JavaScripting do not work in exactly the same way:

One of the differences is that Greasemonkey creates a separate JavaScript environment, in which the Greasemonkey scripts are executed. That is calling Javascript contained in the page from Greasemonkey and the inverse calling Greasemonkey scripts from the page is not possible by default. This is on purpose, so that the web page can not detect and not interfere with the Greasemonkey scripts, because you want your Greasemonkey scripts to work allways on some page, whether or not that page likes it or not. Therefore no interference is possible.

Greasemonkey however provides a standard way to access the web page’s scripts, and that’s through the “unsafeWindow” object, which is a reference to the web page’s environment.

I had two mechanisms I had to make accessible using the “unsafeWindow” handle:

  • the first was accessing JQuery, which is included by default by the Homegate page. Since I needed to use JQuery functions in my Greasemonkey script, I got a reference to it via the standard Greasemonkey precedure:
    var jQuery = unsafeWindow['jQuery'];
  • the second mechanism that needed to cross the boundaries between the web page and Greasemonkey was callbacks from the web page to my Greasemonkey script. This is necessary, because I’m attaching “input” elements to each search result, which contain a note and which, “onblur”, need to call a function that saves the content of the input box. Here’s part that constructs the input element:
    jQuery("<input onclick='event.cancelBubble = true;'" +
                 " onblur='saveComment(this, immoID);'>").insertAfter(immoElement);

And this is the function that gets called back by “onblur”:

    unsafeWindow.saveComment = function(element, immoID) {
      unsafeWindow.localStorage.setItem(immoID, element.value);
    };

The next interesting thing you’ll note is usage of ‘locaStorage’. Support for the latter in browsers does not seem mature yet. One problem I’ve encountered when developing under Firefox 3.6 was that saving to ‘localStorage’ was not possible when cookies were disabled (see this report). Thus you’ll need to permanently enable cookies for Homegate in order for the script to be able to save its data.

Finally, while developing, a major problem was, that Firefox did not show me errors in the Greasemonkey scripts. Thus either the script would work or not work and fail completely silently. That made debugging a bit painful.

So now, here’s the script.

Tomáš Pospíšek

PS: This script also lives at userscripts.org

JMeter Series

This is a short series of howtos for and a critique of JMeter v2.4.

The following articles have been done:

Tomáš Pospíšek, 4.1.2011

Waiting for a page change in JMeter

(This article is part of the JMeter Series)

While testing a Rails application there was a situation where a background worker (DJB or background task) would get an order to execute and eventually the completed order would appear on a page.

Thus we needed to wait for the page to be updated and continue the test after.

User Defined Variables

I’m predefining used variables here - see “We set up our variables” in the Extracting text from a page and using it somewhere else in JMeter article for an exaplanation why.

Clear Loop Variable

This step is not strictly necessary in our example here, however if you want to reuse this sequence of steps more than one time - that is call it from different places, then you need to clear the loop variable first.

Debug

Since creating these steps was not easy, I’ve put a debug statement in so that I see, when JMeter calls into these steps and to see how the variables are set.

The While Controler

Here we really start the loop.

I’m not sure whether if would be better to use BeanShell commands here instead - using JavaScript to test the loop condition works in any case…

Getting the page

We get the page we want to check.

Extract the data we want to check from the page and assign it to our loop variable

Make sure our request is not being cached

It’s better - as far as we can - to be sure we force the web application to recreate the page we’re waiting for to change. I’m not sure I’ve done it right here, however it works:

Wait a bit before re-looping

We wait two seconds here before retrying.

You can download this JMeter test from here

Tomáš Pospíšek, 4.1.2011

Extracting text from a page and using it somewhere else in JMeter

(This article is part of the JMeter Series)

In the following we’ll do these things:

  1. we go to a form submit page
  2. we order something
  3. we get that something we ordered

In more detail:

We use a “Cookie Manager” to carry forward cookies between calls

We set up our variables

The idea here being that we predefine the variables, so they would show up in the “Debug Sampler” which makes debugging easier, because you see at each step, whether the variable has the correct value or not. I do predefine the variables with a “non-value” so that I can see immediately whether the variable has been already assigned something during the execution of the test or not.

We go to a form submit page

We extract the AUTHTOKEN from that form

(The auth token is being used by the web application to prevent cross-site-scripting)

Notice that the extractor is set to extract the string from the HTTP reply-body.

We also extract the session cookie

This is because we want to pass the cookie to an outside application, so that it can call the web app from within the same session.

Notice that this time we told the extractor to extract the string from the HTTP reply-headers.

We now use all our extracted parameters to submit our order

Note that:

  • we are passing the AUTHTOKEN along with the HTTP POST
  • since we are not sure whether possibly the AUTHTOKEN contains some problematic character (in my case it was an equal sign ‘=’ that was interferring with the parameter encoding) we tell JMeter to URL-encode the string.
  • we also pass along another parameter telling the web app what we’re interested in getting back from our order
  • we are using HTTPS

Submitting the order will redirect us to the page that’ll show us the resulting order.

We extract the ORDER_ID from HTTP headers which would otherwise redirect us to the resulting page

The headers contain the location of the resulting order page where we would be redirected to.

Finally we download the resulting artefact with curl

Note that:

  • we are passing the SESSION_COOKIE to curl to be able to download the artifact in the same (potentially authenticated) session.

  • we also construct the download URL from the ORDER_ID

You can download this JMeter test from here

Tomáš Pospíšek, 4.1.2011

The JMeter "Workbench", a trapdoor for the newbie

(This article is part of the JMeter Series)

Upon starting JMeter you’ll see two branches: “Test Plan” and “WorkBench”.

“Test Plan” is the place where your tests will live.

What the purpose of “WorkBench” is, is not really clear. It seems to be meant to be a place to do your throw-away experimentation.

The really crucial “trap” of the “WorkBench” is however, that JMeter will throw away whatever you put into the “WorkBench” upon exit. JMeter will not save the contents of the “WorkBench” if you tell it to “Save” your work and you’ll loose whatever is in there. It will only save the contents of the “Test Plan” branch.

So be ware of putting anything in the “Work Bench”. You’re bound to get burned.

Tomáš Pospíšek, 27.12.2010

Making your JMeter Test modular

(This article is part of the JMeter Series)

As tests get larger, or as steps need to be repeated you’ll want to structure your tests into distinct entities - these seem to be called “Modules” in JMeter.

However, there is no “Module” element JMeter. As a “Module” you can however use the “Simple Controller”. It allows you to drop other elements into it and to name them as a whole. I don’t know whether it has additional features such as providing scoping of any kind.

Thus if you need to “call” the same set of steps from different places, then you can place them in a “Simple Controller”. However you’ll need to place that controller somewhere and as such it will get executed in that place and order. You can prevent it being executed by disabling it:

even though being disabled, components inside a Controller still can be called and executed. Therefore, you can use a disabled “Simple Controller” as a repository for reusable components or modules:

On the picture you can see a “module repository” and a call to a specific “Module” inside it.

Tomáš Pospíšek, 30.12.2010

  • Page 1 of 2 ( 22 posts )
  • >>
  • programming

Back to Top

Sponsors