Yann Hodique blog:2010

Python Webservice, part 2 : a twistd integration

Yann Hodique — 2010-02-19T00:01:46+01:00

Twistd

In my previous post I've explained how to generate a SOAP service from a WSDL file, and how to implement the required methods properly. A solution to run such a service is to use the Twisted Daemon, aka twistd. Though not needed, strictly speaking, it provides a nice set of features aimed at easing complex applications development.

In particular, it allows to write so-called plugins that make it possible to just launch services in the following way:

$ twistd echo

To achieve that, one has to create a file twisted/plugins/echo.py accessible somewhere from the python path (that is, the twisted directory should be a subdir of an element of the path).

Manhole

Twisted offers a nice service, that can be used to provide an inspection/debug interface for your main application. This is called manhole and provides a python interpreter interface. This way you can just telnet/ssh the manhole, and execute appropriate commands to introspect the current state of the application, or put it in the desired state.

A nice example of a really useful function to run from the manhole service is the rebuild one. It tries very hard to provide in-place reloading of code (and corresponding objects). Don't expect any wonder here, it will fail in some corner cases. Still, it proves really useful most of the time, and helps a lot not restarting the server every now and then.

Example

Here is a fully working example of a twistd plugin for the Echo service, with a functional manhole service in addition.

from zope.interface import implements
from twisted.python import usage
from twisted.python.rebuild import rebuild
from twisted.plugin import IPlugin
from twisted.application.service import IServiceMaker, MultiService
from twisted.application.internet import TCPServer
from twisted.web.server import Site
from twisted.web.resource import Resource
from twisted.conch.manhole_tap import makeService as makeConsoleService
 
from EchoImpl import EchoService
 
class EchoOptions(usage.Options):
    optParameters = [["port", "p", 8080, "main port", int],
                     ["service-port", "s", 8081, "service port", int],
                     ["service-users", "u", "/etc/passwd",
                      "The path to a passwd-like authentication file"]]
 
class EchoServiceMaker(object):
    implements(IServiceMaker, IPlugin)
    tapname = "echo"
    description = "Sample Echo service"
    options = EchoOptions
 
    def makeService(self, options):
        # Create Service
        srv = EchoService()
 
        root = Resource()
        # we'll serve Echo at http://<ip>:<port>/echo
        root.putChild('echo', srv)
        siteFactory = Site(root)
 
        echo_service = TCPServer(options['port'], siteFactory)
        console_service = makeConsoleService(
            {"telnetPort": str(options['service-port']),
             "sshPort": None,
             "namespace": {"service": srv,
                           "rebuild": rebuild},
             "passwd": options['service-users']})
 
        svc = MultiService()
        echo_service.setServiceParent(svc)
        console_service.setServiceParent(svc)
 
        return svc
 
serviceMaker = EchoServiceMaker()

The code is hopefully rather straightforward. It should be noted that the manhole service (created by the makeConsoleService() call) needs to know about a set of objects (the “namespace” key). Those objects are the roots for everything accessible from the service. Here, the main service object (the srv object) is available to the manhole under the name service. Here is how it can be used:

$ telnet localhost 8081
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
Username: yann
Password: x
 
>>> print service
<EchoImpl.EchoService object at 0x2fc8c10>
>>>

Hint: the password is really 'x', as I'm using /etc/passwd by default, which contains 'x' for every “password” field, since the real passwords are in /etc/shadow. Convenient for a dev deployment ;)

python, code, twisted

Read or add comments to this article

Python Webservice, part 1 : a Twisted solution

Yann Hodique — 2010-02-17T20:20:16+01:00

As mentioned previously I've been spending some time recently working on a SOAP server mock. For this I've used Python Twisted and ZSI. Since the interaction between the 2 seems rather poorly documented (even Google doesn't know much about it), I thought I'd share what I've investigated and done.

Disclaimer: you should be familiar with SOAP and Python Twisted. I'll not cover anything in these areas.

For the sake of simplicity, I'll take a classical minimal server: an Echo service.

Let's start from the definition of such a service, as a WSDL. This one was randomly picked from there Anyway the implementation details are not so important here.

<?xml version="1.0" encoding="UTF-8"?>
<definitions
    xmlns="http://schemas.xmlsoap.org/wsdl/"
    xmlns:soap="http://schemas.xmlsoap.org/wsdl/soap/"
    xmlns:soapenc="http://schemas.xmlsoap.org/soap/encoding/"
    xmlns:http="http://schemas.xmlsoap.org/wsdl/http/"
    xmlns:xsd="http://www.w3.org/2001/XMLSchema"
    xmlns:tns="urn:ZSI"
    targetNamespace="urn:ZSI" >
  <types>
    <xsd:schema elementFormDefault="qualified" targetNamespace="urn:ZSI">
      <xsd:element name="Echo">
        <xsd:complexType>
          <xsd:sequence>
            <xsd:element name="value" type="xsd:anyType"/>
          </xsd:sequence>
        </xsd:complexType>
      </xsd:element>
    </xsd:schema>
  </types>
 
  <message name="EchoRequest">
    <part name="parameters" element="tns:Echo" />
  </message>
 
  <message name="EchoResponse">
    <part name="parameters" element="tns:Echo"/>
  </message>
  <portType name="EchoServer">
    <operation name="Echo">
      <input message="tns:EchoRequest"/>
      <output message="tns:EchoResponse"/>
    </operation>
  </portType>
 
  <binding name="EchoServer" type="tns:EchoServer">
    <soap:binding style="document"
                  transport="http://schemas.xmlsoap.org/soap/http"/>
    <operation name="Echo">
      <soap:operation soapAction="Echo"/>
      <input>
        <soap:body use="literal"/>
      </input>
      <output>
        <soap:body use="literal"/>
      </output>
    </operation>
  </binding>
 
  <service name="EchoServer">
    <port name="EchoServer" binding="tns:EchoServer">
      <soap:address location="http://localhost:8080/echo"/>
    </port>
  </service>
 
</definitions>

So really nothing fancy here. Just the definitions and shape of the messages/operations (only trivial ones).

ZSI offers the wsdl2py tool, that can process such WSDL files, and produce client bindings and server stub. In this article, the server part is what we want. (note the -w flag will trigger Twisted code generation)

$ wsdl2py -w echo.wsdl

EchoServer_server.py now contains the code for the service stub. Not much in there, the main part being the following method:

def soap_Echo(self, ps, **kw):
    request = ps.Parse(EchoRequest.typecode)
    return request,EchoResponse()

This is the one we should implement to make the service concrete. Here is a first attempt:

class EchoService(EchoServer):
 
    def __init__(self):
        EchoServer.__init__(self)
 
    def soap_Echo(self, ps, **kw):
        request, _ = EchoServer.soap_Echo(self, ps, **kw)
        return request, request

That will work. If you register this Service as a Twisted Resource, it will provide a functional SOAP Echo service, on which you can execute the following.

$ POST http://localhost:8080/echo
Please enter content (application/x-www-form-urlencoded) to be POSTed:
<soapenv:Envelope 
    xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" 
    xmlns:xsd="http://www.w3.org/2001/XMLSchema" 
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <soapenv:Body xmlns:ns1="urn:ZSI">
    <ns1:Echo>
      <ns1:value xsi:type="xsd:string">plop</ns1:value>
    </ns1:Echo>
  </soapenv:Body>
</soapenv:Envelope>
^D
<soapenv:Envelope xmlns:ZSI="http://www.zolera.com/schemas/ZSI/" xmlns:soapenc="http://schemas.xmlsoap.org/soap/encoding/" xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<soapenv:Header></soapenv:Header><soapenv:Body xmlns:ns1="urn:ZSI"><ns1:Echo><ns1:value xsi:type="xsd:string">plop</ns1:value></ns1:Echo></soapenv:Body></soapenv:Envelope>

Still, this is not very Twisted… I mean we don't take advantage of the asynchronous framework here, as the only method we wrote is definitely synchronous and we treat each call after one another. Would the soap_Echo() take time for whatever reason, that would introduce unacceptable delays for the user. Of course, there is no magic in the Deferred objects, which means than simply returning a Deferred in soap_Echo() will absolutely not work.

Digging a bit more in the ZSI Twisted glue, one can find a promising piece of code, appropriately named DeferHandlerChain. This is an alternative to the DefaultHandlerChain that's used (by default) by WSresource objects (hence EchoServer ones). The nice property of this handler chain is to put in place a callback chain from processRequest() to processResponse(), actually supporting Deferred objects returned by soap methods. In a word, what we need…

The code can become something like:

class EchoDeferHandlerChainFactory:
    protocol = DeferHandlerChain
 
    @classmethod
    def newInstance(cls):
        return cls.protocol(DefaultCallbackHandler, DataHandler)
 
class EchoService(EchoServer):
 
    def __init__(self):
        EchoServer.__init__(self)
        EchoServer.factory = EchoDeferHandlerChainFactory
 
    def soap_Echo(self, ps, **kw):
        request, _ = EchoServer.soap_Echo(self, ps, **kw)
 
        def _answer():
            return request
 
        # yeah, we have a very busy echo service... 5 seconds delay !
        d = deferLater(reactor, 5, _answer)
        return request, d

You can now try triggering several parallel Echo calls, and see that they will not be treated in sequence !

python, code, zsi, twisted

Read or add comments to this article

Magit extensions

Yann Hodique — 2010-02-14T13:05:08+01:00

Magit is a great emacs tool to help doing everyday's operations on Git repositories. In particular, I really like the contextual approach, that splits the global status in sections, meaning that the same high-level command can have subtle semantics differences when applied to different objects. As an example, the high-level `magit-discard-item' function obviously takes different steps when applied to an untracked file or to a staged hunk, or even to an entire branch.

The only disadvantage I see in the current implementation is the lack of modularity in it: a heavy use of (really nice, btw) macros makes it mandatory to have the complete set of features available when magit is loaded, so that the generated code is complete.

Still, Git has lots of contributed helpers, which means the only stable way of using them with Magit is to have them integrated upstream, which is not always possible (take the example of custom helpers that cannot be distributed) or does not even make sense (those helpers could be entirely bound to some specificities of a project).

I've been working on an attempt to make Magit more modular, by allowing the registration of so-called “extensions”. See code here: http://github.com/sigma/magit/tree/extensions

As a POC, I've extracted 2 features from the mainstream Magit, and converted them to extensions: those are magit-svn and magit-topgit. The way I see things, those 2 features are exactly the kind I'd like to add to magit for my daily work but that would not necessarily make sense as part of the core project. As far as I know, the current behavior when those 2 extensions are loaded is exactly the same as the regular upstream version (including placement of the various sections)

The current code allows you to:

define new kind of sections (for example a “topic” kind)
define how and where to display them (for example, right after the core “stashes” section)
hook new behavior for existing commands for those new sections (for example a way to discard such an item)
define new top-level commands, and associate menus

I tried to stay very close from the Magit way of doing things, and in particular the way you express things in extensions is exactly the same as what you do in the core code. Once again, tricky macros are at play to make the code look the same.

What I'd like to see in the future (though I don't have much time to do it right now :():

new extensions, to ease my daily life.
- a magit-p4 (perforce integration),
- maybe a magit-stgit (stgit in place of topgit for topic branches)
- something to integrate with VMware's ReviewBoard
code stabilization, and official proposal upstream (if there is an interest in this)
a way to activate those extensions on a per-project basis (I use both git-svn and git-p4 but never at the same time. same for topgit vs stgit)

An example being worth a thousand words, here is how to code the magit-topic extension. If you know the Magit code base, you should feel nearly at home.

Feel free to tell me what you think, any comment or advice is highly welcome !

emacs, git, code

Read or add comments to this article