Shiboken

Last week PySide was launched, the team was glad to see the project finally go public and receive the community feedback, be it positive, negative or both. Many questions arose, like “Why duplicate efforts?” Well, I can’t say much more than what is already answered on PySide FAQ. For us (the team) the fact is that we had a task to accomplish and must perform it the best we can. That said, allow me to remind you that this is my personal blog and many of the views here written are my very own cherished opinion.

The other question that we’re waiting for, and my personal favorite, was “Why Boost.Python?”. Though one. First of all, Boost.Python eases very much the creation of C++ libraries bindings for Python. How to infer which method signature to call based on the Python arguments passed to the method wrapper? Boost.Python will take care of it. Inheritance? Type conversion (in opposition to type wrapping)? You bet: Boost.Python will take care of all this for you. The feature full Boost.Python gave us a great kick start and at first we progressed very fast. Occasionally some strange bug appeared and took some time to figure out the problem through the jungle of template error messages. Part of the job anyhow, and after that: fast pace again.

At some point somebody checked the size of the produced binary modules. “Hey guys, is that correct?”, “Ah, just strip the file.”, “Still huge.”, “Holy cow…”. Next task: size reduction. Some redesigns reduced a good deal of megabytes, g++ flags were also helpful, but these things weren’t enough. Then a new idea: “Let’s try it with the Intel C++ compiler and see what gives.” It gave binary modules with feasible sizes. Good, but the test just proved that it was possible to achieve the reductions. Besides, there were still other new ideas to try, and the fact that as soon as the project was launched the community would step in and say “I had this size problem with Boost.Python before. Here is how I solved it…”. (Which reminds me how limited, communication wise, a project is in its non-open phase. And don’t point your finger, mister — for every open source project has it’s non-open phase, even in your head!)

Part of the team was growing skeptical about the size reduction problem. Why not to try CPython code generation right now? Well, some say you can’t change the plane’s motor while flying, and this is true. Feature wise we were almost there and the reduction was possible. Also some of us had mixed feelings about CPython. In a past project a comparison was made about writing bindings with different technologies, including CPython, to check for speed, size and the burden imposed on the developer. At the end the guy with CPython had good numbers (not stunningly better than the others, at least for the case that mattered back then), but his personal impression was that he was suffering the Stockholm syndrome: he knew CPython abused him, but he developed a bond with his kidnapper.

Still, almost every one started personal (and voluntary, aka “made at home”) experimentation with different CPython generators (even a ctypes one!), and in the end all the ideas (including the ones from the Boost.Python generator) were merged into a single CPython generator, called Shiboken.

Shiboken

Before going on with this, allow me to explain that Shiboken means absolutely nothing. Not buddhist void, I just mean that the word Shiboken has no meaning attached to it. Except, of course, “generator of CPython based binding code for C/C++ libraries”.

Disclaimer: I don’t know a thing about Japanese language and the above kanjis are just something that I found at wikitionary to match the sounds of Shiboken. Forgive me, Lauro. :)

The conspirators’ plan was to develop the alternative generator to a point that could generate PySide bindings that pass all our unit tests, run the apps, etc, thus beign able to replace the Boost.Python front-end. For PySide users, i.e. Python programmers, the replacement would bring no impact, since the API should remain the same. The Shiboken generator is based on the same principles of the Boost.Python one: built using the API Extractor library, how the C++ library should be exported to Python is described on a Type System file, and so on.

Shiboken Generator

The power of Lego-fu!

When Shiboken reached a point that we’d think was good enough to start working with it at work, we presented it to our bosses and the green light was given. The Boost.Python generator will continue as the tool to generate the official PySide bindings, but with the parallel efforts we hope that Shiboken takes its place, the size reduction is achieved, and the Occam’s razor cut off the unnecessary entities.

noboost

Occam's razor demands that Boost.Python go

C/C++ Bindings

Another fix that we aim to achieve with Shiboken is to allow non-Qt C++ and C libraries to be wrapped with the generator scheme. The problem with the current Boost.Python generator is that even a non-Qt library wrapped with it will depend on Qt. Of course this is not the best we can do, but the fixing task had low priority since the PySide bindings are the main target of the work. For Shiboken we make it library agnostic from the start, specially because we do not get our chances trying to wrap the whole Qt from start: a test library with all the problems that could arise was made and is the source of all Shiboken unit tests.

Shiboken Features Worth Noting

Abandoning Boost.Python means abandoning some features already provided by it. One that is worth mentioning is the decision of which signature of a method should be called depending on the Python arguments passed on the call. For this to work we have to write a decisor that progressively checks the argument types until the correct signature is found. Just this? Of course not, the binding developer can use the Type System description to remove method signatures, remove/change the type of its arguments, even remove/change/set its default values! The method call decisor must take everything into account.

Overloaded Method

Debugging a multiple signature method call decisor is easier with the "dumpGraph" method.

No Boost.Python also means that would be harder to convert types and containers back and forth between C++ and Python. The template specialization technique was used to solve this one.

template <> struct Converter<bool> {
  static PyObject* toPython(ValueHolder<bool> holder) {
    return PyBool_FromLong(holder.value);
  }
  static bool toCpp(PyObject* pyobj) {
    return pyobj == Py_True;
  }
};

Why not generating “pyobj == Py_True” directly, you say? The above code scales better, since it will be the same for types that are composed of some primitive types inside containers inside containers, etc. Besides, the compiler could be counted on to inline short methods.

“They have a plan”

Right now Hugo (from PySide fame) started working on generating bare QtCore bindings without QObject and signals, then go to QObject. After that we should solve the signals problem, re-write the pieces of custom code and so on. I think after QtCore is completely done we can make a comparison with the one produced by the Boost.Python generator and see if the whole Shiboken idea can stand to its promise. The best should win for the honor and glory of open source.

We encourage everyone interested in the creation of Python bindings for C++ libraries to test Shiboken, report problems (we now have a component on OpenBossa bugzilla!), and tell us if something is missing for your library to work. Patches are always welcome as usual. :)

Shiboken on gitorious: http://qt.gitorious.org/pyside/shiboken

About these ads

16 thoughts on “Shiboken

  1. Pingback: Shiboken explained – PySide – Python for Qt

  2. It’s nice that you are working on improving PySide but I still wonder why you don’t want to use SMOKE, which is mature and much better than the generator you used for PySide, the generator developed for QtJambi, and probably Shiboken. Ashley Winters, Richard Dale and Arno Rehn have put a lot of effort on SMOKE and right now it is, by far, the best generator for Qt (and KDE) bindings. Why don’t you work with them?

  3. @Pau, in the very early stages of the project we tried some many alternatives, including SMOKE, which we liked a lot in many points. We did some demos with SMOKE but we found it lacking in documentation. Unfortunately the closed nature of our project at the time prevented us to make some needed extensive questions on kde-bindings mailing list without breaking the needed secrecy. I know that this is not the best scenario, but sometimes one must work with unpleasant constraints.
    For now we must continue on the chosen path, but this doesn’t discard future work with SMOKE.

  4. Marcelo: you could have attended a kde-bindings sprint, you could have hired Richard (I think Arno is too young to be legally hired :-) for a week to help you with SMOKE, etc There were options, even more given that a lot of Trolls are also KDE developers.

    Given that the Boost::Python is not an option, and that you have already disclosed what you are working on, make yourself a favor and use SMOKE and, in the way, help us make SMOKE even better.

    Writing a good bindings-generator is difficult and time-consuming, and SMOKE has just been enhanced even more (Arno has spent his GSoC working on smokegen, a kalyptus replacement)

  5. It does suck to work on a project you know is going to be open sourced, but isn’t yet, so you can’t take advantage of the broader open source community really. Open source is much more then using other people’s code after all. I’ve been there, its no good. :)

    But now you’re free, you should look at Smoke. I’ve always found Richard Dale to be very helpful.

    Or at least address Smoke when you write blogs such as this. I guess it just frustrating when folks make their own competing technology not because they think its better, but because they haven’t really looked closely at whats out there already. So I was relieved to hear that you only created PySide after talking to Riverbank. And there might be good technical reasons to favor the CPython solution over Smoke, but you should certainly have one before you build on the CPython solution (given that Smoke has the very real advantage of already existing!).

  6. @Ian: thanks for the understanding on the open-source-to-be matter. :)
    About Smoke, I never get tired of saying how me and the other guys think of it as an awesome idea. On the other hand, the CPython generator even seeming recent and experimental from the way it was launched, has some time of development and we cherish very much. One of the reasons to go on with it is that we will provide a Python binding generator that will benefit creation of bindings for any kind of C/C++ library, and will not impose any dependency on binding developers apart from the basics: his library and Python.
    And yes, is very nice to talk to Richard Dale. :)

  7. @Markus: the short answer is that we didn’t know about the project at the very beginning of the project conception and the first prototypes. I came to know of it “by accident” while following discussions on the Boost mailing list. I started playing with it, and actually tried to wrap the beginnings of a QObject, but it turned out that I’d have to write a lot of custom Python code for that.

    One of shiboken’s (and PySide’s) strong focus is to automate the most laborious part of binding generation as much as possible. Pybindgen has integration with gccxml to help on that, but unfortunately the gccxml parser generates too much low level information for Qt headers (because of the various macros Qt uses). The parser used by PySide/shiboken OTOH was designed to handle Qt headers properly from the start, given its origins from the QtJambi and QtScript generator.

    But one important point where we might share with pybindgen is the generic “code patterns” using to map certain C++ constructs to Python, such as virtual methods, attributes, getters/setters etc. It would be nice if someone could come up with some template specification so we could share the same code snippets for the generic C++ support. But that’s just an idea at the moment :)

  8. @Marcelo…really nice information…I am a novice python and ruby user. I am very interested and excited about Pyside..(Python for Qt).
    I would want to learn python for qt as soon as possible but have not found any basic starter books or tutorials online..

    I have one ebook Rapid GUI Programming with Python and Qt by Mark Summerfield which is for PyQt4. But i am a bit skeptical to follow the book as it might be diffirent then Pyside. Since i am relatively novice user, it would be very nice to have your advice…and also some info where and when i could possible find some starters guide in any form for qt with Python..

    Looking forward…
    Nobin from Finland..

    • @Nobin The book you have is really good, and our released version of PySide aims to follow closely the PyQt4 API, so it should be perfect for you.
      To be precise, there are some exceptions on the compatibility, like not exporting some methods marked as deprecated, but this shouldn’t be a problem since any error can be checked in the Qt4 C++ documentation, or you could ask on PySide mailing list or IRC channel.

  9. PySide installed size on Archlinux ~64M while PyQt ~15M.
    Also, PyQt’s build process is less complex and has fewer project-specific downloads and dependencies.

    I agree that the license alone is good enough argument to start a project like PySide, however, you guys did not spend enough time considering all options in generating the bindings.
    Making bad decisions based on the internal politics of Nokia won’t help getting acceptance in the community either. In fact, the reason most of us enjoy our free software so much is because it is free from the folly of corporate suits.

    Right now, PyQt + SIP is the much better alternative. Unless you’re making your home-brew generator/extractor/whatever build systems easier/less complex than SIP, I don’t see anybody using it. I don’t care much for Boost either.

Deixe um comentário

Preencha os seus dados abaixo ou clique em um ícone para log in:

Logotipo do WordPress.com

Você está comentando utilizando sua conta WordPress.com. Sair / Alterar )

Imagem do Twitter

Você está comentando utilizando sua conta Twitter. Sair / Alterar )

Foto do Facebook

Você está comentando utilizando sua conta Facebook. Sair / Alterar )

Foto do Google+

Você está comentando utilizando sua conta Google+. Sair / Alterar )

Conectando a %s