Skip to content

Wrapping GObject for pygtk – a real world example

01/08/2009

How I wrapped scintilla

In this article I will tell you my journey wrapping pygtkscintilla, I think this could be useful to tell other people which mistakes I made and how I’ve solved them.

Know your enemy

To solve a problem you have to gather all possible information relate to this problem. Scintilla has a scary interface that have nothing to do with a well-formatted gtk-widget.

The interface is a very important thing. If you wanto to wrap scintilla you have to learn the basics of its usage.  Fortunately there was bait, an example  to test scintilla, it’s cast gold.

Here’s the code:

/* bait.c */
#include <gtk/gtk.h>

#include <Scintilla.h>
#include <SciLexer.h>
#define PLAT_GTK 1
#include <ScintillaWidget.h>

static int exit_app(GtkWidget*w, GdkEventAny*e, gpointer p) {
 gtk_main_quit();
 return w||e||p||1;    // Avoid warnings
}

int main(int argc, char **argv) {
 GtkWidget *app;
 GtkWidget *editor;
 ScintillaObject *sci;

 gtk_init(&argc, &argv);
 app = gtk_window_new(GTK_WINDOW_TOPLEVEL);
 editor = scintilla_new();
 sci = SCINTILLA(editor);

 gtk_container_add(GTK_CONTAINER(app), editor);
 gtk_signal_connect(GTK_OBJECT(app), "delete_event",
 GTK_SIGNAL_FUNC(exit_app), 0);

 scintilla_set_id(sci, 0);
 gtk_widget_set_usize(editor, 500, 300);

#define SSM(m, w, l) scintilla_send_message(sci, m, w, l)

 SSM(SCI_STYLECLEARALL, 0, 0);

And this is the Makefile associated:

##### Makefile #####
# Make file for bait on Linux or compatible OS
# Released to the public domain 2000 by Neil Hodgson neilh@scintilla.org
# This makefile tested with GCC 3.2 and GNOME 2.0

.SUFFIXES: .c .o .h .a

INCLUDEDIRS=-I../scintilla/include
CXXFLAGS= -DGTK -DSCI_LEXER -W -Wall -fPIC
LEXEROBJS=$(wildcard ../scintilla/gtk/Lex*.o)

all: bait

.c.o:
 gcc `pkg-config --cflags gtk+-2.0` $(INCLUDEDIRS) $(CXXFLAGS) -c $< -o $@
bait: bait.o $(LEXEROBJS) ../scintilla/bin/scintilla.a
 gcc `pkg-config --libs gtk+-2.0 gthread-2.0` -lstdc++ -DGTK $^ -o $@
clean:
 rm -rf bait *.o
&#91;/sourcecode&#93;

I noticed:
<ol>
	<li>How to initialize a scintilla editor</li>
	<li>How to interact with it. It uses just the SSM macro:  scintilla_send_message function.</li>
	<li>With the Makefile associated, how to compile and run it.</li>
</ol>
Once I've done this,  I needed a good interface to respond codegen (utility used to wrap gtk widgets, explained in the <a href="https://pygabriel.wordpress.com/2009/07/22/wrapping-gobjects-for-pygtk/" target="_blank">wrapping tutorial</a>).
<h3>It's time to start wrapping</h3>
There was 2 possibilities:
<ol>
	<li>Write myself the widget in C.</li>
	<li>Search the web for someone that have already done this.</li>
</ol>
I'm really lazy, I searched a lot on the web and I've found some alternatives:
<ul>
	<li>The geany editor: there was some files that wrapped the scintilla interface in a more convenient manner, however it wasn't well-formatted because they don't use intensively the C/GObject system.</li>
	<li>The anjuta editor:  They use a well formatted editor widget that wraps scintilla but there were interdependencies related to anjuta.</li>
	<li>The most natural GtkScintilla: someone have done this work long time ago, after some days I noticed that it was a very good starting point.</li>
</ul>
note: I discovered all this things browsing in the project files, reading headers and source files ( I felt like a true hacker).

I haven't modified the gtkscintilla.h header and I generated definition files. Next I wrote a simple override file and the initialization module (if you don't understand this first read the <a href="https://pygabriel.wordpress.com/2009/07/22/wrapping-gobjects-for-pygtk/" target="_blank">wrapping tutorial</a>).

With no problem I generated the wrapper, cool! (pygtk team have done a great work with codegen, I'm sad they didn't write any documentation)

Now comes the hard part...
<h3>Compiling</h3>
Welcome in the hell. Scintilla is distribuited as a static library, scintilla.a but the python module I needed was a shared library. I browsed in a sea of pain, errors, errors and errors, my module wouldn't compile.

The problem is that you can't link a static library to a shared library, the compiler tell you that you have to recompile your files with the -fPIC option. I ignored what the compiler told me, and I tried a lot of  dirty ways to make it work, without success...

After about 4 days of depression I was sorry with the wisdom of gcc compiler and I was gone in the scintilla/gtk directory to modify the makefile that compiles the scintilla stuff.

In the original makefile there was the compilation command without the -fPIC option and the command "ar" to make a static library.

The compilation commands:


.cxx.o:
 $(CC) $(CONFIGFLAGS) $(CXXFLAGS) -c $<
.c.o:
 $(CCOMP) $(CONFIGFLAGS) $(CXXFLAGS) -w -c $<

&#91;/sourcecode&#93;

Becomes:

&#91;sourcecode language='bash'&#93;
 .cxx.o:
 $(CC) $(CONFIGFLAGS) $(CXXFLAGS) -fPIC -c $<
 .c.o:
 $(CCOMP) $(CONFIGFLAGS) $(CXXFLAGS) -fPIC -w -c $<

&#91;/sourcecode&#93;

And I modified the creation of the library from static (compressed with ar) to shared (compiled with gcc -shared)

Before:

&#91;sourcecode language="bash"&#93;
COMPLIB= ../bin/scintilla.a
AR= ar
// other things //

$(COMPLIB): DocumentAccessor.o WindowAccessor.o KeyWords.o StyleContext.o \
 CharClassify.o Decoration.o Document.o PerLine.o CallTip.o \
 ScintillaBase.o ContractionState.o Editor.o ExternalLexer.o PropSet.o PlatGTK.o \
 KeyMap.o LineMarker.o PositionCache.o ScintillaGTK.o CellBuffer.o ViewStyle.o \
 RESearch.o RunStyles.o Style.o Indicator.o AutoComplete.o UniConversion.o XPM.o \
 $(MARSHALLER) $(LEXOBJS)
 $(AR) rc $@ $^
&#91;/sourcecode&#93;

After:

&#91;sourcecode language='bash'&#93;
COMPLIB= ../bin/scintilla.so # MODIFIED
AR= ar
// other things //

$(COMPLIB): DocumentAccessor.o WindowAccessor.o KeyWords.o StyleContext.o \
 CharClassify.o Decoration.o Document.o PerLine.o CallTip.o \
 ScintillaBase.o ContractionState.o Editor.o ExternalLexer.o PropSet.o PlatGTK.o \
 KeyMap.o LineMarker.o PositionCache.o ScintillaGTK.o CellBuffer.o ViewStyle.o \
 RESearch.o RunStyles.o Style.o Indicator.o AutoComplete.o UniConversion.o XPM.o \
 $(MARSHALLER) $(LEXOBJS)
 gcc -shared -o $@ $^ # MODIFIED
&#91;/sourcecode&#93;

After recompiling scintilla I obtained my shared library scintilla.so !!! After this I clearly compiled and linked (reference the <a href="https://pygabriel.wordpress.com/2009/07/31/wrapping-pygtk-widgets-what-you-should-know/">basics tutorial</a>) my python module and run it without (too much) problems!

I post the pygtkscintilla makefile that compiled and linked the python module:


DEFS=`pkg-config --variable=defsdir pygtk-2.0`
CFLAGS=-I/usr/include/python2.6 `pkg-config --cflags gtk+-2.0 pygtk-2.0 gthread-2.0`  -I. -I..
CXXFLAGS=-DGTK2  
LDFLAGS=`pkg-config --libs gtk+-2.0 pygtk-2.0 pygobject-2.0 gthread-2.0`
LEXEROBJS=$(wildcard ~/workspace/pygtkscintilla/scintilla/gtk/Lex*.o)

all: gtkscintilla-wrap.c gtkscintilla-module.o gtkscintilla.o gtkscintilla.so

gtkscintilla-wrap.c: gtkscintilla.defs gtkscintilla.override
 python /usr/share/pygobject/2.0/codegen/codegen.py \
 --override gtkscintilla.override \
 --prefix gtkscintilla gtkscintilla.defs > $@

# Here's the magic!
gtkscintilla.so: gtkscintilla-wrap.o gtkscintilla.o  gtkscintilla-module.o
 gcc -shared  $(LDFLAGS) -L~/workspace/pygtkscintilla/scintilla/bin -lscintilla  $^ -o $@

gtkscintilla-wrap.o: gtkscintilla-wrap.c
 gcc $(CFLAGS) -fPIC -c $^ -o $@
gtkscintilla.o: gtkscintilla.c
 gcc $(CFLAGS) -fPIC -c $^ -o $@
gtkscintilla-module.o: gtkscintilla-module.c
 gcc  $(CFLAGS) -fPIC -c   $^ -o $@

clean:
 rm -f gtkscintilla.so gtkscintilla-wrap.c gtkscintilla.o gtkscintilla-module.o

Ok I’ve finished telling my story. I hope that my errors can help you!

Advertisements

Wrapping pygtk widgets – what you should know

31/07/2009

In past days I’ve encountered a series of problems. In this article I will explain what kind of problem one can found and how to solve this problems.

I’ve some hints:

  • Study the C syntax and the pointers , skip all the algorithmic part: it’s not useful for wrapping a python module.
  • It’s useful to  study some C++  for design purposes, the C manuals doesn’t face well the multiple-file programs, compilation,  headers topics… In C++ this is necessary and well explained in manuals.
  • The most important thing you should know is how to read (and write ) simple Makefiles,  I’ve tried to learn autotools but they are very hard to understand.

Libraries and python modules

A python module (from a C wrapper)  is a shared library. What’s shared, and what’s library? Let’s explain with a simple example.

Suppose you are a mathematician and you’ve coded in C your ugly formulas.  You want to use this formulas in other programs.

The best solution is to split your program in three files:

  • main.c : Run the program, using the functions you specify in other files.( includes headers files like ugly-functions.h to define the fuctions prototypes)
  • ugly-functions.h : here are the function prototipes, each function is declared and explained in an human-readable format ( a lot of  comments)
  • ugly-functions.c : here are the function implementations, you code what each function do.

Well, it’s time to compile your boring program. You can compile with:

gcc ugly-functions.c main.c -o boring-program

Another way, more interesting:

gcc -c ugly-functions.c # The output is a file called ugly-functions.o
gcc -c main.c           # main.o
gcc main.o ugly.functions.o -o boring-program # This phase is called linking

With the first two commands you create object files.  With the last command you link the so called object-files in an executable form.

The -c option tells gcc to don’t call the linker. The linker links the function implementations to their name. (otherwise main.c couldn’t find the function implementations).

Static libraries

Now you can found other formulas and code them in another file:  good-formulas.c.

You make your header file(.h), and your source file (.c), now you can compile with the same command.

gcc -c ugly-functions.c # The output is a file called ugly-functions.o
gcc -c good-formulas.c  # good-formulas.o
gcc -c main.c           # main.o
gcc main.o ugly.functions.o good-formulas.o -o interesting-program

Ok you have obtained your interesting program with your new good formulas.  Now you may want to transform them in libraries.

Static libraries

To make a static library, you can group (archive) your object-files in a single file with this command:


ar rcs libmine.a ugly-functions.o good-formulas.o


now with this archive you can link your functions to the main program in this way:

gcc main.o libmine.a -o interesting-program

OK. You’ve done our static library. You’ve copied in the final executable all the object-files.  When there are a lot of object-files the executable will growth in dimension. Here comes the shared libraries.

A note before proceeding, static libraries are discouraged in general.

Shared libraries

Now there’s the most important part of the tutorial.  Because python modules are shared libraries.

To make a shared library, you have to recompile your .c files in a special manner, with the -fPIC option:  this makes the library to be loaded and referenced at run time.

gcc -c -fPIC ugly-functions.c
gcc -c -fPIC good-formulas.c

Now we can create the library, libmine.so :

gcc -shared ugly-functions.o good-formulas.o -o libmine.so

Now it’s time to link, you can use the same method:

gcc main.o libmine.so -o interesting-program

There’s another way to link the program, that is mostly used:

gcc -L/directory/containing/library -lmine main.o -o interesting-program

The -l option specifies which library to link when you use this you have to omit the lib prefix and the .so suffix  (if there isn’t the lib prefix you have to omit only the .so suffix).

The -L option specifies the path where to find the library.

Remember this options that are widely used in Makefiles.

Now this library can be used by a lot of programs and it’s linked at runtime but it has to be reachable. If you run the program you will catch this error:

./interesting-program: error while loading shared libraries: libmine.so:
 cannot open shared object file: No such file or directory

You have to tell to the system where to find the library, you can put it in a standard location like /usr/lib or you can use the variable LD_LIBRARY_PATH.

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/directory/containing/library

Re-run and it should be ok. There are a lot of other problems you can encounter, mostly related to the file you have linked in the library.

What you should take at home

When you compile python module in a shared library you can encounter a lot of problems:

  • Sure you that you have compiled ALL the object files with the -fPIC option
  • You must include all the libraries that contains the functions defined. Otherwise you will encounter “undefined symbol” errors.
  • Others that now I can’t remember

Eureka eureka eureka!!! pygtkscintilla coming soon!!!!

29/07/2009

Well after some days of pain and blood finally I compiled and run a pygtkscintilla module!!! This gives me a great great satisfaction and happiness!!!

🙂 🙂 🙂

Help, wrapping Scintilla for pygtk

28/07/2009

Just why?

Scintilla is a the powerful text editor widget.  I’s used by a lot of applications, it’s something like this:

Scintilla is for GTK+ but this f*** widget doesn’t follow standards from GTK+/GObject.

In the actual state of things there isn’t a good pygtk IDE/editor written in pygtk (there’s pida but uses embedded editors  like vim and emacs), and I think the reason is that there isn’t a scintilla wrapper. To make a comparison with pyqt there’s the fairly complete eric.

The code source editor for pygtk  is pygtksourceview but this widget hasn’t the code-completion feature and the code folding feauture that in my opinion are fundamental. These features will be implented in near(?) future and I will use it, but I (maybe we) need a powerful editor now.

I want to wrap scintilla for pygtk, it was wrapped sometime ago by Roberto Cavada but it’s outdated  and his mail isn’t reachable.

The current state of things

  • found GtkScintilla widget, a well-formatted GObject scintilla distribution. It’s outdated but it’s a good starting point to write another GObject to wrap.
  • I generate the wrapper  (cfr Wrapping Gobject) for GtkScintilla but I can’t compile it … frustrating.
  • In next days I will test a little the scintilla distribution to focus the design and to learn to compile all the things… I need.

There’s a lot of work to do, it’s very hard, every help is accepted (everyone can partecipate, also without experience. Just tell me in comments or via e-mail).

Wish me good luck!

Redirecting the stdout on a gtk.TextView

27/07/2009

The problem

Suppose you want to spawn a system process in python and you want to intercept the stdout of the process and redirecting it on a gtk.TextView.
For example in this tutorial you will see how to redirect a generic command ( in the example there is a typical blocking process).

The solution

The most correct solution of the problem is not so immediate, if you search around you will get some answers related to threads and so on… If you study threads is a good thing but we can do it without knowing threads :). Expecially because threads in python can be buggy in some particular cases ( like in pygtk itself).

Spawning a process

To spawn a process the best solution ever is to use the subprocess module. The syntax is simple:

  • popen_object = subprocess.Popen(command, shell = True, stdout = subprocess.PIPE)
  • We tell to subprocess to spawn a process with the command “command”
  • The option shell = True tell subprocess that the command is a shell command
  • We also want to redirect the stdout to a pipe  ->  that is the popen_object.
import subprocess
proc = subprocess.Popen("ls ../", shell = True,stdout=subrocess.PIPE) # ls ../ is an example

The proc object is a Popen object, it has some interesting methods and attributes, in particular we can access
the stdout of the process,  (like) it’s a file.

Interesting methods and attributes:

  • proc.communicate() will wait until the process finish and give us the stdout
  • proc.kill() will kill the process
  • proc.stdout is a file descriptor (an opened file) that we can use

There are a lot of other features, if you are interested (maybe you are) look at the documentation.

Redirecting the stdout

To read and redirect the stdout of the process to the TextView widget we will use the glib library that’s shipped with pygtk. This library has a function glib.io_add_watch that is pretty smart:

glib.io_add_watch

    def glib.io_add_watch(fd, condition, callback, ...)
fd : a Python file object or an integer file descriptor ID
condition : a condition mask
callback : a function to call
... : additional arguments to pass to callback
Returns : an integer ID of the event source

The glib.io_add_watch() function arranges for the file (specified by fd) to be monitored by the main loop for the specified conditionfd may be a Python file object or an integer file descriptor. The value of condition is a combination of:

glib.IO_IN There is data to read.
glib.IO_OUT Data can be written (without blocking).
glib.IO_PRI There is urgent data to read.
glib.IO_ERR Error condition.
glib.IO_HUP Hung up (the connection has been broken, usually for pipes and sockets).

Additional arguments to pass to callback can be specified after callback. The idle priority may be specified as a keyword-value pair with the keyword “priority”. The signature of the callback function is:

  def callback(source, cb_condition, ...)

where source is fd, the file descriptor; cb_condition is the condition that triggered the signal; and, ... are the zero or more arguments that were passed to the glib.io_add_watch() function.

If the callback function returns FALSE it will be automatically removed from the list of event sources and will not be called again. If it returns TRUE it will be called again when the condition is matched.

  • The fd is an opened file like our  proc.stdout!!
  • the condition is something like “there’s something to read”:  in computer language glib.IO_IN
  • the callback is a normal function with this prototype:  callback(fd, condition_triggered), the condition triggered tell us why the callback was called (there’s something to read, someone has interrupted the process and so on…)
  • NOTE: this particular callback has to return True or False, from the reference:
    “If the callback function returns FALSE it will be automatically removed from the list of event sources and will not be called again. If it returns TRUE it will be called again when the condition is matched.”

Creating you smart TextView

the plan is to create a special textview, we pass a command, and this Textview presents itself giving us the command stdout. Well we know that her constructor has one argument:

#Test driven developement, write the test:
def test():
    # This command will download one project of mine deheh
    ctv = CommandTextView("svn co https://fitta.svn.sourceforge.net/svnroot/fitta fitta")
    win=gtk.Window()
    win.connect("delete-event", lambda wid,event: gtk.main_quit()) # Defining callbacks with lambdas
    win.set_size_request(200,300)
    win.add(ctv)
    win.show_all()
    ctv.run()
    gtk.main()
if __name__=='__main__' : test()

We need also a method run that starts all the things, now let’s write the class, in the constructor simply
we will save the command as a variable and we’ll do the initialization stuff,  in the run we will use subprocess to spawn the process and we will use glib.io_add_watch to fetch the stdout.

import gtk,glib
import subprocess

class CommandTextView(gtk.TextView):
    ''' Nice TextView that reads the output of a command syncronously '''
    def __init__(self, command):
        '''command : the shell command to spawn'''
        super(CommandTextView, self).__init__()
        self.command = command
    def run(self):
        ''' Runs the process '''
        proc = subprocess.Popen(self.command, stdout = subprocess.PIPE) # Spawning
        glib.io_add_watch(proc.stdout, # file descriptor
                          glib.IO_IN,  # condition
                          self.write_to_buffer ) # callback
    def write_to_buffer(self, fd, condition):
        if condition == glib.IO_IN: #if there's something interesting to read
           char = fd.read(1) # we read one byte per time, to avoid blocking
           buf = self.get_buffer()
           buf.insert_at_cursor(char) # When running don't touch the TextView!!
           return True # FUNDAMENTAL, otherwise the callback isn't recalled
        else:
           return False # Raised an error: exit and I don't want to see you anymore

Every time there is something to read the glib.io_add_watch recall the callback,
and in the callback we write to the textview, nice and simple.

For the complete example, you can download at this link:

command-textview.py

Todo list: Remember the Milk and pyrtm

25/07/2009

I’m a really forgetful person and for a long time I’ve done a lot of  gaffes with colleagues and customers … really frustrating.

A todo list can solve this kind of problems. Now, what todo list? For Linux there are about ten projects, I found Tasque a good solution but there isn’t an active developement actually.

The strenght of tasque is the integration with Remember The Milk,  a web-based todo list with a comfortable API for programmers.  So you can have different applications (in different operating systems and in different devices.. such as cell-phones) that interfaces with the same web database.

Personally I think that this is a really good idea, I simply love web services  expecially for a thing like a task-list: I can’t and I don’t want to bring my laptop here and there just because I don’t remember what I have to do in my life (poor me :().

You can use remember the milk from the nice web-interface. It”s straightforward to learn and has a lot of killer-features such as:

  • contacts : to remember your girlfriend that she have to prepare a delicious dinner for you.
  • ready to print tasks: you can print and check your list.
  • locations: ok nice but not so killer.
  • date:  you type “today” or “tomorrow” and the program understand what do you mean!
  • priority, url notes ecc ecc..

The applications that interfaces them with remember the mik (rtm for friends) doesn’t provide all of this features, and I don’t know why… But at least you are every moment well syncronized and you can access when you want to your tasks.

In addition, stay tuned,  this smart guy and me will develop a nice task list rtm-based! (Ok ok, it isn’t exciting for you but it is for me…)

We want to develop it in python using the almost resurrected pyrtm project. Of course, everybody is invited to partecipate! Just tell me 🙂

Bye Bye!

Curve fitting, my little project :)

23/07/2009

I want to inform you all that some time ago I’ve done a little project.

This is for scientists in general that have to analyze experimental data. If you don’t know what’s a fit almost surely you  don’t need this software…

It’s for Linux and it has some nice features, I’m just waiting someone that uses it to get feedbacks and feature-requests…

I post the link to the project website:

http://hartree.altervista.org/fitta/

Bye Bye, I hope you enjoy!