In my previous post I've explained how to generate a SOAP service from a WSDL file, and how to implement the required methods properly. A solution to run such a service is to use the Twisted Daemon, aka twistd. Though not needed, strictly speaking, it provides a nice set of features aimed at easing complex applications development.

In particular, it allows to write so-called plugins that make it possible to just launch services in the following way:

$ twistd echo

To achieve that, one has to create a file twisted/plugins/echo.py accessible somewhere from the python path (that is, the twisted directory should be a subdir of an element of the path).

Twisted offers a nice service, that can be used to provide an inspection/debug interface for your main application. This is called manhole and provides a python interpreter interface. This way you can just telnet/ssh the manhole, and execute appropriate commands to introspect the current state of the application, or put it in the desired state.

A nice example of a really useful function to run from the manhole service is the rebuild one. It tries very hard to provide in-place reloading of code (and corresponding objects). Don't expect any wonder here, it will fail in some corner cases. Still, it proves really useful most of the time, and helps a lot not restarting the server every now and then.

Here is a fully working example of a twistd plugin for the Echo service, with a functional manhole service in addition.

from zope.interface import implements
from twisted.python import usage
from twisted.python.rebuild import rebuild
from twisted.plugin import IPlugin
from twisted.application.service import IServiceMaker, MultiService
from twisted.application.internet import TCPServer
from twisted.web.server import Site
from twisted.web.resource import Resource
from twisted.conch.manhole_tap import makeService as makeConsoleService
 
from EchoImpl import EchoService
 
class EchoOptions(usage.Options):
    optParameters = [["port", "p", 8080, "main port", int],
                     ["service-port", "s", 8081, "service port", int],
                     ["service-users", "u", "/etc/passwd",
                      "The path to a passwd-like authentication file"]]
 
class EchoServiceMaker(object):
    implements(IServiceMaker, IPlugin)
    tapname = "echo"
    description = "Sample Echo service"
    options = EchoOptions
 
    def makeService(self, options):
        # Create Service
        srv = EchoService()
 
        root = Resource()
        # we'll serve Echo at http://<ip>:<port>/echo
        root.putChild('echo', srv)
        siteFactory = Site(root)
 
        echo_service = TCPServer(options['port'], siteFactory)
        console_service = makeConsoleService(
            {"telnetPort": str(options['service-port']),
             "sshPort": None,
             "namespace": {"service": srv,
                           "rebuild": rebuild},
             "passwd": options['service-users']})
 
        svc = MultiService()
        echo_service.setServiceParent(svc)
        console_service.setServiceParent(svc)
 
        return svc
 
serviceMaker = EchoServiceMaker()

The code is hopefully rather straightforward. It should be noted that the manhole service (created by the makeConsoleService() call) needs to know about a set of objects (the “namespace” key). Those objects are the roots for everything accessible from the service. Here, the main service object (the srv object) is available to the manhole under the name service. Here is how it can be used:

$ telnet localhost 8081
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
Username: yann
Password: x
 
>>> print service
<EchoImpl.EchoService object at 0x2fc8c10>
>>>

Hint: the password is really 'x', as I'm using /etc/passwd by default, which contains 'x' for every “password” field, since the real passwords are in /etc/shadow. Convenient for a dev deployment ;)

As mentioned previously I've been spending some time recently working on a SOAP server mock. For this I've used Python Twisted and ZSI. Since the interaction between the 2 seems rather poorly documented (even Google doesn't know much about it), I thought I'd share what I've investigated and done.

Disclaimer: you should be familiar with SOAP and Python Twisted. I'll not cover anything in these areas.

For the sake of simplicity, I'll take a classical minimal server: an Echo service.

Let's start from the definition of such a service, as a WSDL. This one was randomly picked from there Anyway the implementation details are not so important here.

<?xml version="1.0" encoding="UTF-8"?>
<definitions
    xmlns="http://schemas.xmlsoap.org/wsdl/"
    xmlns:soap="http://schemas.xmlsoap.org/wsdl/soap/"
    xmlns:soapenc="http://schemas.xmlsoap.org/soap/encoding/"
    xmlns:http="http://schemas.xmlsoap.org/wsdl/http/"
    xmlns:xsd="http://www.w3.org/2001/XMLSchema"
    xmlns:tns="urn:ZSI"
    targetNamespace="urn:ZSI" >
  <types>
    <xsd:schema elementFormDefault="qualified" targetNamespace="urn:ZSI">
      <xsd:element name="Echo">
        <xsd:complexType>
          <xsd:sequence>
            <xsd:element name="value" type="xsd:anyType"/>
          </xsd:sequence>
        </xsd:complexType>
      </xsd:element>
    </xsd:schema>
  </types>
 
  <message name="EchoRequest">
    <part name="parameters" element="tns:Echo" />
  </message>
 
  <message name="EchoResponse">
    <part name="parameters" element="tns:Echo"/>
  </message>
  <portType name="EchoServer">
    <operation name="Echo">
      <input message="tns:EchoRequest"/>
      <output message="tns:EchoResponse"/>
    </operation>
  </portType>
 
  <binding name="EchoServer" type="tns:EchoServer">
    <soap:binding style="document"
                  transport="http://schemas.xmlsoap.org/soap/http"/>
    <operation name="Echo">
      <soap:operation soapAction="Echo"/>
      <input>
        <soap:body use="literal"/>
      </input>
      <output>
        <soap:body use="literal"/>
      </output>
    </operation>
  </binding>
 
  <service name="EchoServer">
    <port name="EchoServer" binding="tns:EchoServer">
      <soap:address location="http://localhost:8080/echo"/>
    </port>
  </service>
 
</definitions>

So really nothing fancy here. Just the definitions and shape of the messages/operations (only trivial ones).

ZSI offers the wsdl2py tool, that can process such WSDL files, and produce client bindings and server stub. In this article, the server part is what we want. (note the -w flag will trigger Twisted code generation)

$ wsdl2py -w echo.wsdl

EchoServer_server.py now contains the code for the service stub. Not much in there, the main part being the following method:

def soap_Echo(self, ps, **kw):
    request = ps.Parse(EchoRequest.typecode)
    return request,EchoResponse()

This is the one we should implement to make the service concrete. Here is a first attempt:

class EchoService(EchoServer):
 
    def __init__(self):
        EchoServer.__init__(self)
 
    def soap_Echo(self, ps, **kw):
        request, _ = EchoServer.soap_Echo(self, ps, **kw)
        return request, request

That will work. If you register this Service as a Twisted Resource, it will provide a functional SOAP Echo service, on which you can execute the following.

$ POST http://localhost:8080/echo
Please enter content (application/x-www-form-urlencoded) to be POSTed:
<soapenv:Envelope 
    xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" 
    xmlns:xsd="http://www.w3.org/2001/XMLSchema" 
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <soapenv:Body xmlns:ns1="urn:ZSI">
    <ns1:Echo>
      <ns1:value xsi:type="xsd:string">plop</ns1:value>
    </ns1:Echo>
  </soapenv:Body>
</soapenv:Envelope>
^D
<soapenv:Envelope xmlns:ZSI="http://www.zolera.com/schemas/ZSI/" xmlns:soapenc="http://schemas.xmlsoap.org/soap/encoding/" xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<soapenv:Header></soapenv:Header><soapenv:Body xmlns:ns1="urn:ZSI"><ns1:Echo><ns1:value xsi:type="xsd:string">plop</ns1:value></ns1:Echo></soapenv:Body></soapenv:Envelope>

Still, this is not very Twisted… I mean we don't take advantage of the asynchronous framework here, as the only method we wrote is definitely synchronous and we treat each call after one another. Would the soap_Echo() take time for whatever reason, that would introduce unacceptable delays for the user. Of course, there is no magic in the Deferred objects, which means than simply returning a Deferred in soap_Echo() will absolutely not work.

Digging a bit more in the ZSI Twisted glue, one can find a promising piece of code, appropriately named DeferHandlerChain. This is an alternative to the DefaultHandlerChain that's used (by default) by WSresource objects (hence EchoServer ones). The nice property of this handler chain is to put in place a callback chain from processRequest() to processResponse(), actually supporting Deferred objects returned by soap methods. In a word, what we need…

The code can become something like:

class EchoDeferHandlerChainFactory:
    protocol = DeferHandlerChain
 
    @classmethod
    def newInstance(cls):
        return cls.protocol(DefaultCallbackHandler, DataHandler)
 
class EchoService(EchoServer):
 
    def __init__(self):
        EchoServer.__init__(self)
        EchoServer.factory = EchoDeferHandlerChainFactory
 
    def soap_Echo(self, ps, **kw):
        request, _ = EchoServer.soap_Echo(self, ps, **kw)
 
        def _answer():
            return request
 
        # yeah, we have a very busy echo service... 5 seconds delay !
        d = deferLater(reactor, 5, _answer)
        return request, d

You can now try triggering several parallel Echo calls, and see that they will not be treated in sequence !

Magit is a great emacs tool to help doing everyday's operations on Git repositories. In particular, I really like the contextual approach, that splits the global status in sections, meaning that the same high-level command can have subtle semantics differences when applied to different objects. As an example, the high-level `magit-discard-item' function obviously takes different steps when applied to an untracked file or to a staged hunk, or even to an entire branch.

The only disadvantage I see in the current implementation is the lack of modularity in it: a heavy use of (really nice, btw) macros makes it mandatory to have the complete set of features available when magit is loaded, so that the generated code is complete.

Still, Git has lots of contributed helpers, which means the only stable way of using them with Magit is to have them integrated upstream, which is not always possible (take the example of custom helpers that cannot be distributed) or does not even make sense (those helpers could be entirely bound to some specificities of a project).

I've been working on an attempt to make Magit more modular, by allowing the registration of so-called “extensions”. See code here: http://github.com/sigma/magit/tree/extensions

As a POC, I've extracted 2 features from the mainstream Magit, and converted them to extensions: those are magit-svn and magit-topgit. The way I see things, those 2 features are exactly the kind I'd like to add to magit for my daily work but that would not necessarily make sense as part of the core project. As far as I know, the current behavior when those 2 extensions are loaded is exactly the same as the regular upstream version (including placement of the various sections)

The current code allows you to:

• define new kind of sections (for example a “topic” kind)
• define how and where to display them (for example, right after the core “stashes” section)
• hook new behavior for existing commands for those new sections (for example a way to discard such an item)
• define new top-level commands, and associate menus

I tried to stay very close from the Magit way of doing things, and in particular the way you express things in extensions is exactly the same as what you do in the core code. Once again, tricky macros are at play to make the code look the same.

What I'd like to see in the future (though I don't have much time to do it right now :():

• new extensions, to ease my daily life.
  • a magit-p4 (perforce integration),
  • maybe a magit-stgit (stgit in place of topgit for topic branches)
  • something to integrate with VMware's ReviewBoard
• code stabilization, and official proposal upstream (if there is an interest in this)
• a way to activate those extensions on a per-project basis (I use both git-svn and git-p4 but never at the same time. same for topgit vs stgit)

An example being worth a thousand words, here is how to code the magit-topic extension. If you know the Magit code base, you should feel nearly at home.

Feel free to tell me what you think, any comment or advice is highly welcome !

It's been a long time I wanted to give a try to Google's GWT. Also, lately I needed more and more a robust and straightforward implementation of David Allen's Getting Things Done to help me organize my work. There it is, let's combine those two whishes and produce a working tool.

This toolkit gives you the opportunity to code your web application entirely in Java while deploying it under its compiled form (HTML + JavaScript / Java servlet). The advantages are obvious : during the development, you have access to all the facilities of a Java environment (nice enough language when compared to JavaScript , debugger), to its robustness, and so on.

I've always been attracted by compilers from one high-level language to another lower-level ones. The GWT compiler is clearly a success in this area. It makes it really easy to introduce a layer of AJAX in your web application without even thinking about it.

GWT is composed mainly of the following components:

• a graphical toolkit following roughly the way of AWT, QT, … so you feel like home really fast
• a minimal implementation of the most useful java.lang and java.util classes for client use (translated to JavaScript after compilation)
• a services API that deals with AJAX (client-side) and Java servlets (server-side)

There are also other interesting aspects (JUnit integration for example), but I'll leave them for another day. All this is integrated in a very clever and usable way, which makes it easy to develop your application. During development, all you're going to use is provably “hosted mode”, which is a pack provided by Google, that includes an embedded Tomcat server (for servlets), an embedded browser (based on mozilla), so that you can have a sample of your application running in one click without needing to deploy anything. Plus, this “hosted mode” really runs your Java code, so that you may as well debug it in the meantime. Brilliant !

A small issue that is worth noting : it looks like when you use “hosted mode”, some informations are transmitted to Google. I have to admit I'm a bit disappointed about that.

GTD seems to be very popular these days. The number of articles (in blogs mostly… mine is no exception now ;)) is constantly increasing, and the number of related tools grows in proportion. Still I did not find a single tool that really satisfying.

Here are my expectations for a perfect GTD tools:

• it should be mostly online : I can't rely on a single device (especially not a laptop, and a PDA is not reactive enough)
• there should be an offline client/cache/whatever : I'm not always online, even if I am most of the time
• it should be fast, because most of the time you just add really small tasks to your list
• it should be nice to look at, because you'll use it a lot

With my friend Alexandre, we started to write the tool of our dreams : GWTD. Well, the name is nearly obvious I guess… still we will have to give it a meaning for completeness purpose. The two goals are :

• to discover the possibilities of GWT
• to have a working implementation of GTD that fills our needs.

The project is still in very early stage of development (given the small amount of time we can afford on that, it's no surprise), bu we hope to come up with a release soon enough. The code (distributed under Apache licence 2.0) is available at code.google.com, and a Google group has also been created. Stay tuned !

I just finished upgrading my Wiki to latest development version of Dokuwiki. This was not an easy task because I've made quite a large number of modifications that needed to be ported or incorporated. I've decided to maintain a Darcs branch of both my Dokuwiki and Arctic theme.

For now this is no complete success since I access the filesystem via sshfs, and it looks like it lacks a proper lock implementation. I'll try to find a better way

By the way, I also upgraded several plugins to their latest version, so that you (and I) may experience some weird and uncontrolled border effects. That's the price for living on the bleeding edge.