Yann Hodique

Python Webservice, part 2 : a twistd integration

Yann Hodique — 2010-02-19T00:01:46+01:00

Twistd

In my previous post I've explained how to generate a SOAP service from a WSDL file, and how to implement the required methods properly. A solution to run such a service is to use the Twisted Daemon, aka twistd. Though not needed, strictly speaking, it provides a nice set of features aimed at easing complex applications development.

In particular, it allows to write so-called plugins that make it possible to just launch services in the following way:

$ twistd echo

To achieve that, one has to create a file twisted/plugins/echo.py accessible somewhere from the python path (that is, the twisted directory should be a subdir of an element of the path).

Manhole

Twisted offers a nice service, that can be used to provide an inspection/debug interface for your main application. This is called manhole and provides a python interpreter interface. This way you can just telnet/ssh the manhole, and execute appropriate commands to introspect the current state of the application, or put it in the desired state.

A nice example of a really useful function to run from the manhole service is the rebuild one. It tries very hard to provide in-place reloading of code (and corresponding objects). Don't expect any wonder here, it will fail in some corner cases. Still, it proves really useful most of the time, and helps a lot not restarting the server every now and then.

Example

Here is a fully working example of a twistd plugin for the Echo service, with a functional manhole service in addition.

from zope.interface import implements
from twisted.python import usage
from twisted.python.rebuild import rebuild
from twisted.plugin import IPlugin
from twisted.application.service import IServiceMaker, MultiService
from twisted.application.internet import TCPServer
from twisted.web.server import Site
from twisted.web.resource import Resource
from twisted.conch.manhole_tap import makeService as makeConsoleService
 
from EchoImpl import EchoService
 
class EchoOptions(usage.Options):
    optParameters = [["port", "p", 8080, "main port", int],
                     ["service-port", "s", 8081, "service port", int],
                     ["service-users", "u", "/etc/passwd",
                      "The path to a passwd-like authentication file"]]
 
class EchoServiceMaker(object):
    implements(IServiceMaker, IPlugin)
    tapname = "echo"
    description = "Sample Echo service"
    options = EchoOptions
 
    def makeService(self, options):
        # Create Service
        srv = EchoService()
 
        root = Resource()
        # we'll serve Echo at http://<ip>:<port>/echo
        root.putChild('echo', srv)
        siteFactory = Site(root)
 
        echo_service = TCPServer(options['port'], siteFactory)
        console_service = makeConsoleService(
            {"telnetPort": str(options['service-port']),
             "sshPort": None,
             "namespace": {"service": srv,
                           "rebuild": rebuild},
             "passwd": options['service-users']})
 
        svc = MultiService()
        echo_service.setServiceParent(svc)
        console_service.setServiceParent(svc)
 
        return svc
 
serviceMaker = EchoServiceMaker()

The code is hopefully rather straightforward. It should be noted that the manhole service (created by the makeConsoleService() call) needs to know about a set of objects (the “namespace” key). Those objects are the roots for everything accessible from the service. Here, the main service object (the srv object) is available to the manhole under the name service. Here is how it can be used:

$ telnet localhost 8081
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
Username: yann
Password: x
 
>>> print service
<EchoImpl.EchoService object at 0x2fc8c10>
>>>

Hint: the password is really 'x', as I'm using /etc/passwd by default, which contains 'x' for every “password” field, since the real passwords are in /etc/shadow. Convenient for a dev deployment ;)

python, code, twisted

Read or add comments to this article

Site Map

Yann Hodique — 2010-02-18T14:37:25+01:00

Professional

Personal

About Me
Blog
Documentation
Projects
- BibTeX plugin
- Emacs contribs
Articles

Books

ICFP 2004

Yann Hodique — 2010-02-18T11:43:15+01:00

Background

I took part in the ICFP code contest in 2004, with a few friends of mine from Melting-Pot.

For those who don't know of ICFP code contest, let's have a few words on the rule. A subject is published on the web site at a given time, and from then you have 72 hours to submit your solution. That's all. The problems are often very funny, and the idea of staying in front of a computer for 72 hours is also (did I mention my geekness?)

For informations about the particular 2004 issue, have a look at the official website

Our solution is based on a compiler written in C++ which takes a small C-like dedicated language as input and generates “ant-assembly” language. The compiler itself is rather nice and involves a very expressive preprocessor. Anyway, it took us about 2/3 of the total time to come up with the compiler, so that the real programs (the ones to be compiled… the ant brains…) were not that efficient.

The result was a 83th place in the general ranking. Not so bad, but not as good as it may have been with more time to think about strategies instead of compilation issues.

Idea

C++ was obviously not the best choice to write a compiler in very limited time. It's efficient, it's clean, but it was much too long to develop and debug.

Later, during my holidays (remember my geekness :p), I felt the need to experiment a better solution. As you may find out on the web or even on this site, I'm a big fan of Lisp languages, and I had the strong feeling that I could come up with a functional solution very quickly.

Of course I had already the scheme of the solution in mind (that is, the general design of the language I would need). We were 4 persons the first time, and the general idea of the language to be compiled took us a few hours. Anyway, in 6 hours, I wrote alone a compiler that is much better than our submission (more expressiveness, opimizations,…) without any prior knowledge of Common Lisp specificities (I like learning a language by programming such things).

Target language

(shamelessly extracted from the rules)

The behavior of ants is defined by a simple, finite state machine. Informally, the instructions of this state machine can be described as follows:

instruction	effect
`Sense sensedir st1 st2 cond`	Go to state `st1` if `cond` holds in `sensedir`; and to state `st2` otherwise.
`Mark i st`	Set mark `i` in current cell and go to `st`.
`Unmark i st`	Clear mark `i` in current cell and go to `st`.
`PickUp st1 st2`	Pick up food from current cell and go to `st1`; go to `st2` if there is no food in the current cell.
`Drop st`	Drop food in current cell and go to `st`.
`Turn lr st`	Turn left or right and go to `st`.
`Move st1 st2`	Move forward and go to `st1`; go to `st2` if the cell ahead is blocked.
`Flip p st1 st2`	Choose a random number x from 0 to `p`-1; go to `st1` if x=0 and `st2` otherwise.

That's all. Now that's possible to write a brain directly with these instructions, but obviously one could benefit from an abstraction that would make the code easier to write… a compiler.

Description of the language

The compiler is able to compile a small lisp-like language (Lisp is very good at using its own representation).

Top level

There are two kinds of functional units in this languages

func: they are some kind of lisp functions, but limitations in the target code prevent from having a notion of return. Thus, they all end with a call to another func

(func function-name
  <statement 1>
  ...
  <statement n>)

macro: as usual, macro allows to inline a block of code. It is essential for the same reason: there is no return. Thus, code replication is unavoidable

(macro macro-name
  <statement 1>
  ...
  <statement n>)

Statements

The body of a top-level form is composed of statements.

progn: a classical sequence in Lisp. It reflects the execution of all its arguments

(progn
  <statement 1>
  ...
  <statement n>)

turn: the primitive action of making the ant turn its head in the given direction ('left or 'right or any variable that contains one of these symbols)

(turn 'left)

mark, unmark: the primitive action of putting/removing a mark at the current position

(mark 0)

if: the choice betwwen two statements, based on a condition

(if <condition>
    <statement 1>
  <statement 2>)

when and unless: trivial variants of if
cond: generalized if

(cond (<condition 1>
       <statement 1>)
      ...
      (<condition n>
       <statement n>))

drop: drop the food in-place

(drop)

call: branch to the beginning of another function

(call 'another-function)

while: classical loop structure

(while <condition>
  <statement>)

dotimes: constant loop

(dotimes 3
  <statement>)

any other symbol is treated as an invocation to a macro

Conditions

Several instructions produce boolean results, to be used within the context of conditional statements.

pickup: try to pick food up there

(if (pickup)
  ...)

flip: basic random action. The highest the argument, the most unlikely the event.

(if (flip 3)
  ...)

move: a movement may fail, due to a rock, another ant, …

(if (move)
  ...)

sense: try to detect something somewhere

(if (sense 'left-ahead 'food)
  ...)

not, or, and: classical composition operators

Examples

The engine makes it possible to express in a lisp-like language strategies like the following

;; this will be inlined for each "invocation"
(macro uturn ()
  (dotimes 3 (turn left)))
 
;; entry point of the ant brain
(func main
  ;; let's randomize a bit the initial direction
  (if (flip 3)
      (turn 'left)
      (when (flip 2)
        (turn 'right)))
  ;; from now, search some food. Note that the "function" will never return
  ;; since it can't be expressed easily with the basic instructions
  ;; (plus it's not that useful)
  (call 'search-food))
 
(func search-food
  (while (!(pickup))
    (cond ((sense 'left-ahead 'food)
           (progn
             (turn 'left)
             (while (not (move)))
             (while (not (pickup)))
             (uturn)
             (while (not (move)))
             (turn 'right)
             (call 'back-to-home)))
          ((sense 'right-ahead 'food)
           (progn
             (turn 'right)
             (while (not (move)))
             (while (not (pickup)))
             (uturn)
             (while (not (move)))
             (turn 'left)
             (call 'back-to-home)))
          ((!(move))
           (turn 'left)
           (when (flip 2)
             (turn 'left)))))
  (uturn)
  (call 'back-to-home))
 
(func back-to-home
  (while (not (sense 'here 'home))
    (unless (move)
      (turn 'right)))
  (drop)
  (call 'go-out-from-home))
 
(func go-out-from-home
  (uturn)
  (while (sense 'here 'home)
    (when (move)))
  (call 'search-food))
 
;; compile everything from "main"
(make-program)
 
;; dump the result in a suitable format
(dump-program "primitive-ant")

Solution

Let's see the code of the compiler

(defpackage icfp
  (:use common-lisp)
  (:export func macro make-program dump-program))
 
(in-package icfp)
 
(defvar *program* nil)
(defvar *functions* nil)
(defvar *macros* nil)
(defvar *optimize* t)
 
(defmacro while (condition &body body) `(do () ((not ,condition) t) ,@body))
 
(defun indexes-of (elem list)
  (let ((counter -1))
    (loop for e in list
       do (incf counter)
       if (equal e elem) collect counter)))
 
(defun uniquify (list)
  (loop for elem in list
     if (not (member elem acc)) collect elem into acc
     finally (return acc)
       ))
 
(defmacro single-instr (&rest instr) `(list (list ,@instr)))
 
(defmacro relative-jump (n) `(cons 'relative ,n))
 
(defun relative-jump-p (elem)
  (and (consp elem) (eq (car elem) 'relative)))
 
(defmacro absolute-jump (n) `(cons 'absolute ,n))
 
(defun absolute-jump-p (elem)
  (and (consp elem) (eq (car elem) 'absolute)))
 
(defmacro extern-function (f) `(cons 'extern ,f))
 
(defun extern-function-p (elem)
  (let ((el elem))
    (and (consp el ) (eq (car el) 'extern))))
 
(defmacro marker-symbol (n) n)
 
;; Instructions
(defun compile-goto (where)
  (single-instr 'flip 1 where where))
 
(defun compile-turn-instr (direction)
  (assert (or (eq direction 'left)
              (eq direction 'right)))
  (single-instr 'turn direction (relative-jump 1)))
 
(defun compile-mark-instr (marker)
  (single-instr 'mark (marker-symbol marker) (relative-jump 1)))
 
(defun compile-unmark-instr (marker)
  (single-instr 'unmark (marker-symbol marker) (relative-jump 1)))
 
(defmacro mapindex (i f seq)
  `(let ((,i 0)) (mapcar #'(lambda (s) (apply ,f (list s)) (incf i)) ,seq)))
 
(defun compile-if-instr (condition &optional then &rest else)
  (let ((compiled-cond (compile-cond condition))
        (compiled-then (compile-instr then))
        (compiled-else (compile-block else)))
 
    (mapindex i #'(lambda (ins)
                    (nsubst (relative-jump (- (length compiled-cond) i))
                            'alt1 ins))
              compiled-cond)
    (mapindex i #'(lambda (ins)
                    (nsubst (relative-jump (- (+ (length compiled-cond)
                                                 (length compiled-then)
                                                 (if else 1 0)) i))
                            'alt2 ins))
              compiled-cond)
    (append compiled-cond
            compiled-then
            (if else (compile-goto (relative-jump (+ (length compiled-else) 1))))
            compiled-else)))
 
(defun compile-when-instr (condition &rest then)
  (compile-if-instr condition (cons 'progn then)))
 
(defun compile-unless-instr (condition &rest else)
  (apply #'compile-if-instr condition nil else))
 
(defun compile-cond-instr (&rest clauses)
  (let* ((compiled-clauses
          (loop for cl in clauses collect (list (compile-cond (car cl))
                                                (compile-block (cdr cl)))))
         (total (+ 1
                   (length clauses) ; jumps
                   (apply #'+ (mapcar #'(lambda (cl)
                                          (+ (length (car cl))
                                             (length (cadr cl)))) compiled-clauses)))))
    (mapcan #'(lambda (cl)
                (let ((condition (car cl))
                      (bl (cadr cl)))
                  (setq total (- total (length condition) (length bl) 1))
 
                  (mapindex i #'(lambda (ins)
                                  (nsubst (relative-jump (- (length condition) i))
                                          'alt1 ins))
                            condition)
                  (mapindex i #'(lambda (ins)
                                  (nsubst (relative-jump (- (+ (length condition)
                                                               (length bl) 1) i))
                                          'alt2 ins))
                            condition)
                  (append condition
                          bl
                          (compile-goto (relative-jump (+ total 1))))))
            compiled-clauses)))
 
(defun compile-while-instr (condition &rest body)
  (let* ((compiled-cond (compile-cond condition))
         (compiled-body (compile-block body))
         (l (+ (length compiled-cond) (length compiled-body))))
 
    (mapindex i #'(lambda (ins)
                    (nsubst (relative-jump (- (length compiled-cond) i))
                            'alt1 ins))
              compiled-cond)
    (mapindex i #'(lambda (ins)
                    (nsubst (relative-jump (- (+ l 1) i))
                            'alt2 ins))
              compiled-cond)
    (append compiled-cond
            compiled-body
            (compile-goto (relative-jump (- l))))))
 
(defun compile-dotimes-instr (n &rest body)
  (assert (numberp n))
  (let ((compiled-body (compile-block body))
        (res nil))
    (loop for x below n do (setq res (append res compiled-body)))
    res))
 
(defun compile-drop-instr ()
  (single-instr 'drop (relative-jump 1)))
 
(defun compile-call-instr (function)
  (single-instr 'flip 1 (extern-function function) (extern-function function)))
 
(defmacro macro-body (m) `(caddr (assoc ,m *macros*)))
 
(defmacro macro-args (m) `(cadr (assoc ,m *macros*)))
 
(defun args-subst (sym-args real-args body)
  (assert (equal (length sym-args) (length real-args)))
  (let ((fbody body))
    (loop for sym in sym-args for real in real-args
         do (nsubst real sym fbody))
    fbody))
 
(defun compile-macro-instr (m &rest args)
  (let ((body (macro-body m))
        (sym-args (macro-args m)))
    (assert body)
    (compile-block (args-subst sym-args args body))))
 
;; Conditions
(defun compile-pickup-cond ()
  (single-instr 'pickup 'alt1 'alt2))
 
(defun compile-flip-cond (value)
  (single-instr 'flip value 'alt1 'alt2))
 
(defun compile-move-cond ()
  (single-instr 'move 'alt1 'alt2))
 
(defun compile-sense-cond (place kind)
  (single-instr 'sense place 'alt1 'alt2 kind))
 
(defun compile-not-cond (condition)
  (let ((compiled-cond (compile-cond condition)))
    (nsubst 'alt3 'alt1 compiled-cond)
    (nsubst 'alt1 'alt2 compiled-cond)
    (nsubst 'alt2 'alt3 compiled-cond)
    compiled-cond))
 
(defun compile-or-cond (&rest conditions)
  (cond ((null conditions) nil)
        ((equal (length conditions) 1)
         (compile-cond (car conditions)))
        (t
         (let ((first (compile-cond (car conditions)))
               (rest (apply #'compile-or-cond (cdr conditions))))
           (nsubst (relative-jump (length first)) 'alt2 first)
           (append first rest)))))
 
(defun compile-and-cond (&rest conditions)
  (cond ((null conditions) nil)
        ((equal (length conditions) 1)
         (compile-cond (car conditions)))
        (t
         (let ((first (compile-cond (car conditions)))
               (rest (apply #'compile-and-cond (cdr conditions))))
           (nsubst (relative-jump (length first)) 'alt1 first)
           (append first rest)))))
 
(defun compile-t-cond ()
  (single-instr 'flip 1 'alt1 'alt1))
 
(defun compile-nil-cond ()
  (single-instr 'flip 1 'alt2 'alt2))
 
(defun expand-args (args)
  (mapcar #'(lambda (x)
              (cond ((and (consp x) (eq (car x) 'quote))
                     (cadr x))
                    ((symbolp x) (symbol-value x))
                    (t x)))
          args))
 
(defun build (f args)
  (apply f (expand-args args)))
 
(defun compile-cond (condition)
  (if (not (consp condition))
      (case condition
        ('nil (compile-nil-cond))
        ('t (compile-t-cond))
        (t (error "No such condition constant")))
      (let ((instr (car condition))
            (args (cdr condition)))
        (case instr
          ('pickup (build 'compile-pickup-cond args))
          ('flip (build 'compile-flip-cond args))
          ('move (build 'compile-move-cond args))
          ('sense (build 'compile-sense-cond args))
          ('not (build 'compile-not-cond args))
          ('! (build 'compile-not-cond args))
          ('or (build 'compile-or-cond args))
          ('and (build 'compile-and-cond args))
          (t (error "No such condition operator"))
          ))))
 
(defun compile-instr (instruction)
  (let ((instr (car instruction))
        (args (cdr instruction)))
    (when instr
      (case instr
        ('progn (mapcan #'compile-instr args))
        ('turn (build 'compile-turn-instr args))
        ('mark (build 'compile-mark-instr args))
        ('unmark (build 'compile-unmark-instr args))
        ('if (build 'compile-if-instr args))
        ('when (build 'compile-when-instr args))
        ('unless (build 'compile-unless-instr args))
        ('cond (build 'compile-cond-instr args))
        ('drop (build 'compile-drop-instr args))
        ('call (build 'compile-call-instr args))
        ('while (build 'compile-while-instr args))
        ('dotimes (build 'compile-dotimes-instr args))
        (t (apply 'compile-macro-instr instr args))
        ))))
 
(defun compile-block (body)
  (compile-instr (cons 'progn body)))
 
(defun compile-function (body)
  (let ((compiled-body (compile-block body)))
    (append compiled-body
            ;; protect from going out a bad designed func (a func should always
            ;; end by a call). This should NEVER be executed. In case it is
            ;; anyway, let's re-execute the function. The useless code will be
            ;; removed at optimization time anyway.
            (compile-goto (relative-jump (- (length compiled-body)))))))
 
(defmacro func (name &body body)
  `(let ((compiled-function (compile-function ',body)))
     (setq *functions* (cons (cons ',name compiled-function)
                             (delete-if #'(lambda (f) (eq ',name (car f))) *functions*)))))
 
(defmacro macro (name args &body body)
  `(setq *macros* (cons (list ',name ',args ',body)
                        (delete-if #'(lambda (f) (eq ',name (car f))) *macros*))))
 
(defmacro function-body (f) `(cdr (assoc ,f *functions*)))
 
(defun function-dependencies (f)
  (mapcan #'identity
          (let ((body (function-body f)))
            (loop for instr in body collect
                 (loop for elem in instr
                    if (and (consp elem ) (eq (car elem) 'extern))
                    collect (cdr elem))))))
 
(defmacro run-on-program-with-counter (counter f)
  `(let ((counter 0))
     (mapcar #'(lambda (instr)
                 (let ((res (apply ,f (list instr))))
                   (setq ,counter (+ 1 ,counter))
                   res))
             *program*)))
 
(defun transform-matching-instr-function (match op)
  #'(lambda (instr)
      (loop for elem in instr
         if (apply match (list elem))
         collect (apply op (list elem)) else collect elem)))
 
(defun resolve-extern (f offset)
  (setq *program*
        (run-on-program-with-counter
         counter
         (transform-matching-instr-function
          #'(lambda (elem)
              (and (extern-function-p elem)
                   (eq (cdr elem) f)))
          #'(lambda (elem)
              (declare (ignore elem))
              (absolute-jump offset))))))
 
(defun resolve-relative ()
  (setq *program*
        (run-on-program-with-counter
         counter
         (transform-matching-instr-function
          #'relative-jump-p
          #'(lambda (elem)
              (absolute-jump (+ (cdr elem) counter)))))))
 
(defun resolve-absolute ()
  (setq *program*
        (run-on-program-with-counter
         counter
         (transform-matching-instr-function
          #'absolute-jump-p
          #'cdr))))
 
(defun goto-alist ()
  (let ((l (delete
            nil
            (run-on-program-with-counter
             counter
             #'(lambda (instr)
                 (if (and (eq (car instr) 'flip)
                          (equal (nth 2 instr) (nth 3 instr)))
                     (cons counter
                           (if (consp (nth 2 instr))
                               ;; called before (resolve-absolute)
                               (cdr (nth 2 instr))
                               ;; called on final version
                               (nth 2 instr)))))))))
    (loop for j in l collect
         (let ((orig (car j))
               (from (car j)))
           (while (assoc from l)
             (setq from (cdr (assoc from l))))
           (cons orig from)))))
 
(defun simplify-jumps ()
  (setq *program*
        (let ((g (goto-alist)))
          (run-on-program-with-counter
           counter
           (transform-matching-instr-function
            #'absolute-jump-p
            #'(lambda (elem)
                (cons (car elem)
                      (or (cdr (assoc (cdr elem) g))
                          (cdr elem)))))))))
 
(defun following (instr)
  (loop for elem in instr
       if (absolute-jump-p elem) collect (cdr elem)))
 
(defun non-dead-code ()
  (let ((to-follow nil)
        (followed nil)
        (current 0))
    (while (setq to-follow
                 (append
                  (delete-if
                   #'(lambda (n) (or (member n followed)
                                     (member n to-follow)))
                   (uniquify (following (nth current *program*))))
                  to-follow))
      (setq followed (append followed (list current)))
      (setq current (car to-follow))
      (setq to-follow (cdr to-follow)))
    (setq followed (append followed (list current)))
    (run-on-program-with-counter
     counter
     #'(lambda (instr)
         (if (member counter followed)
             instr)))))
 
(defun kill-dead-code ()
  (setq *program* (non-dead-code))
  (setq *program*
        (delete
         nil
         (let* ((indexes (indexes-of nil *program*)))
           (run-on-program-with-counter
            counter
            (transform-matching-instr-function
             #'absolute-jump-p
             #'(lambda (elem)
                 (cons (car elem)
                       (- (cdr elem)
                          (length (delete-if
                                   #'(lambda (i) (> i (cdr elem)))
                                   (copy-list indexes))))))))))))
 
(defun optimize-program ()
  (simplify-jumps)
  (kill-dead-code))
 
(defun link-program (functions)
  (mapcar #'(lambda (f) (resolve-extern (car f) (cdr f)))
          functions)
  (resolve-relative)
  (if *optimize*
      (optimize-program))
  (resolve-absolute))
 
(defun make-program ()
  (setq *program* (function-body 'main))
  (let ((active-functions '((main . 0)))
        (deps (function-dependencies 'main)))
    (do () ((not deps) t)
      (let ((f (car deps))
            (l (length *program*)))
        (if (not (assoc f active-functions))
            (progn
              (setq deps (append deps (function-dependencies f)))
              (setq *program* (append *program* (function-body f)))
              (setq active-functions (cons (cons f l) active-functions)))))
      (setq deps (cdr deps)))
    (link-program active-functions)))
 
(defun print-program ()
  (loop for instr in *program* do
       (progn
         (loop for el in instr
            do (progn
                 (princ el)
                 (princ #\Space)))
         (princ  #\NewLine)
         )))
 
(defun dump-program (filename)
  (assert (stringp filename))
  (with-open-file (s filename :direction :output :if-exists :overwrite)
    (loop for instr in *program* do
         (progn
           (loop for el in instr
              do (format s "~S " el))
           (format s "~%")))))

Conclusion

Would we have written all this in Lisp from the beginning, we would have had at least two entire days to think about strategies to implement, instead of less than one :p
Lisp rocks !

lisp, code

tutorial lisp bindings

Yann Hodique — 2010-02-18T11:42:49+01:00

« Blog Tags

ICFP 2004	2010/02/14 08:58		lisp, code
libxosd sous guile	2010/02/14 08:58		lisp

libxosd sous guile

Yann Hodique — 2010-02-18T11:32:35+01:00

Présentation

Bien souvent il arrive au programmeur en quête de réutilisation de vouloir se lier à une bibliothèque partagée existante, mais qui malheureusement se trouve être écrite dans un autre langage que celui qu'il compte employer. Dans le cas le plus commun, il s'agira probablement d'une bibliothèque écrite en C (ce qui correspond aux recommandations du projet GNU). Le programmeur n'a alors d'autre choix que de recourir à une couche de liaison (binding en anglais) pour importer les fonctionnalités de ladite bibliothèque dans son langage favori.

Dans ce tutoriel, nous nous intéresserons à l'exploitation de la libxosd dans un programme écrit (ou extensible) en guile, le langage d'extension officiel du projet GNU, qui est une implémentation de scheme. On pourra notamment envisager l'utilisation de cette liaison pour l'affichage de noms de bureaux virtuels dans Scwm, la génération d'un énorme message d'avertissement lorsque vos comptes (gérés par gnucash) seront dans le rouge, ou simplement l'écriture de scripts guile servant à notifier un événement quelconque.

Pour être complets il convient de noter que les langages considérés ne sont pas neutres: en effet guile exporte une interface native C, ce qui simplifie notablement le portage de la bibliothèque choisie.

De C à guile

Encapsulation de la structure de données

Pour commencer, il est bon de se rappeler (ou d'apprendre) que guile manipule un seul type de données: SCM, aussi les fonctions que nous allons écrire manipuleront ce type en lieu et place des arguments C des fonctions “normales”. Nous aurons donc recours à un certain nombre d'assertions et de conversions pour passer des arguments scheme aux arguments réels dans de bonnes conditions (et vice versa).

La libxosd adopte un modèle objet centré autour d'une structure xosd. Ce type jouant un rôle central, nous devons tout d'abord lui donner un équivalent guile. Guile permet de définir des smobs (small objects), qui nous permettront de faire ce que nous voulons avec les objets xosd. La définition d'un smob se fait simplement en lui donnant un nom et une taille :

xosd_tag = scm_make_smob_type(xosd,sizeof( xosd *));

Remarquons que nous n'avons pas fait de lien entre le smob et la structure de données que nous voulons encapsuler (xosd *). Ce lien ne sera fait qu'à la création d'un nouvel objet du type de ce smob.

Ainsi, SCM_RETURN_NEWSMOB(xosd_tag, w); où w est de type xosd* crée un objet guile auquel l'identifiant xosd_tag est attaché, et dont on pourra contrôler le type ultérieurement. Nous reviendrons sur cette étape lorsque nous nous intéresserons à l'écriture d'un constructeur.

Passons à présent aux fonctions permettant conversion d'un SCM en son véritable type C.

Nous aurons dans la majeure partie des cas à vérifier que le premier argument de la fonction considérée est bien un pointeur vers un xosd. L'interface de programmation des smobs nous fournit une macro pour cela:

SCM_SMOB_PREDICATE(xosd_tag,osd)

qui teste l'appartenance de osd (de type SCM) au type smob d'identifiant xosd_tag. D'autres macros de test sont disponibles pour les types “primitifs”: ainsi on pourra écrire SCM_INUMP(n) pour tester l'appartenance de n aux entiers naturels.

Pour effectuer un vrai test de validité d'appel, nous avons encore besoin de la macro suivante

SCM_ASSERT(condition,argument_effectif,
           argument_symbolique,fonction)

qui se chargera d'interrompre l'exécution et de lever une erreur si condition n'est pas respectée.

Enfin, lorsque toutes le vérifications sont passées, il nous reste encore à exécuter réellement les conversions de types nécessaire. Nous avons vu précédemment la macro SCM_RETURN_NEWSMOB qui nous permettait de convertir un xosd* en son smob associé, il est temps de découvrir la macro SCM_SMOB_DATA(x) qui, appliquée au résultat précédent, retourne le xosd* de départ. Dans la suite, nous utiliserons la macro suivante.

#define XOSD(x) ((xosd *) SCM_SMOB_DATA(x))

Encore une fois, il existe également des macros de conversion pour les types “primitifs”, ainsi SCM_INUM(n) appelé sur un SCM renverra un int, et la conversion est valide sous réserve que SCM_INUMP(n) soit vrai.

Encapsulation d'une fonction

Une fois les données correctement encapsulées, l'encapsulation d'une fonction se fait assez facilement suivant un schéma constant :

vérification des arguments ;
appel du code utile ;
construction de la valeur de retour.

Voyons un exemple complet d'encapsulation de la fonction xosd_set_horizontal_offset(xosd * osd, int n)

static SCM _wrap_xosd_set_horizontal_offset(SCM osd, SCM n) {
    SCM_ASSERT(SCM_SMOB_PREDICATE(xosd_tag,osd), osd,
               SCM_ARG1, "xosd-set-horizontal-offset!");
    SCM_ASSERT(SCM_INUMP(n), n,
               SCM_ARG2, "xosd-set-horizontal-offset!");
 
    xosd_set_horizontal_offset(XOSD(osd),SCM_INUM(n));
 
    return SCM_UNSPECIFIED;
}

La valeur SCM_UNSPECIFIED retournée peut être vue comme l'équivalent d'un void C: une valeur inutilisable.

Adaptation des interfaces

En passant d'un langage à un autre, il apparaît rapidement que certaines constructions, parfaitement adaptées dans l'un, se trouvent tout à fait inadaptées dans l'autre. Ainsi, la fonction suivante semble complexe, et peu transposable en scheme :

int xosd_display (xosd *osd, int line,
                  xosd_command command, ...);
// command appartient à l'ensemble
// {XOSD_percentage, XOSD_string, XOSD_printf, XOSD_slider}

Il est bien sûr possible d'écrire une fonction scheme qui adopte le même comportement, mais la simple présence d'arguments génériques rendrait la solution inélégante. Si des adaptations mineures de l'interface sont envisageables, on pourra faire le choix de séparer les différents comportements (les différentes valeurs de command) en plusieurs fonctions. D'autre part la commande XOSD_printf est clairement écrite comme un substitut à la fonction printf() du C, par conséquent, il est logique de ne pas la transcrire telle quelle, mais d'émuler son comportement à travers la fonction scheme format et grâce à la commande XOSD_string. Ainsi les chaînes de formatage acceptées ne seront pas nécessairement les mêmes, mais elles seront cohérentes avec les habitudes liées au langage.

Nous écrirons donc trois fonctions: xosd-display-percentage, xosd-display-slider, et xosd-display-string. Une fois ceci fait, on pourra sans problème écrire (xosd-display-string (format ”%d” n)) pour simuler une fonction (xosd-display-printf)

// affiche str à la ligne line
int xosd_display_string(xosd *osd, int line, char *str) {
    return xosd_display(osd,line,XOSD_string,str);
}
 
static SCM _wrap_xosd_display_string(SCM osd, SCM line, SCM str) {
    SCM_ASSERT(SCM_SMOB_PREDICATE(xosd_tag,osd), osd,
               SCM_ARG1, "xosd-display-string");
    SCM_ASSERT(SCM_INUMP(line), line,
               SCM_ARG2, "xosd-display-string");
    SCM_ASSERT(SCM_STRINGP(str), str,
               SCM_ARG3, "xosd-display-string");
 
    xosd_display_string(XOSD(osd),SCM_INUM(line),
                        SCM_STRING_CHARS(str));
 
    return SCM_UNSPECIFIED;
}

Dans un registre différent, il arrive qu'en C on émule le retour de plusieurs valeurs par le passage de pointeurs vers des variables locales. Ainsi la fonction xosd_get_colour a le profil suivant:

int xosd_get_colour(xosd *osd, int *red,
                    int *green, int *blue);

Cette façon de faire est tout à fait inappropriée en scheme (puisque l'on n'a pas de pointeurs), où l'usage dans des cas semblables est plutôt de renvoyer une liste de résultats. Ainsi on voudra plutôt :

(xosd-get-colour osd)
  -> (42 42 42)

Constructeurs/destructeurs

libxosd est une bibliothèque objet (bien qu'écrite en C) et fournit des fonctions pour construire et détruire des instances. Le premier réflexe (malheureux, évidemment) sera d'encapsuler ces méthodes comme les autres, et de les utiliser en l'état en lisp. Bien que cela fonctionne au premier abord, il subsiste un problème majeur: supposons que nous ayions écrit une fonction xosd-destroy qui sert de destructeur. Que se passera-t-il si nous réutilisons l'objet détruit? la même chose qu'en C, une segfault. D'autre part, que se passera-t-il si nous perdons un object xosd en faisant par exemple (set! xosd nil) (ceci a pour conséquence d'affecter nil, qui est la valeur nulle de scheme à la variable xosd)? absolument rien, mais il nous est désormais absolument impossible de récupérer la mémoire allouée pour cet objet (memory leak). Ces deux facettes du même problème mettent en exergue le fait que dans un programme scheme, la mémoire doit être gérée automatiquement, il faut donc s'assurer que les objets passent bien par le ramasse-miettes (garbage collector) pour ne pas avoir de problème.

Guile fournit une interface pour associer à un smob une fonction de libération et une fonction de marquage. La première sert à libérer explicitement la mémoire, la seconde sert à marquer d'éventuels objets à passer au ramasse-miettes.

Ces fonctions sont installées lors de la phase d'initialisation du module par les appels suivants :

scm_set_smob_mark(xosd_tag,mark_xosd);
scm_set_smob_free(xosd_tag,free_xosd);

ils permettent au ramasse-miettes de prendre conscience de l'existence d'objets supplémentaires, et surtout de la manière dont il doit les gérer.

Puisque notre smob n'est en fait rien d'autre qu'un xosd* la fonction de marquage est triviale. La fonction de libération, quant à elle, est simplement un enrobage autour de xosd_destroy(). En ce qui concerne le constructeur en revanche, rien de particulier, si ce n'est qu'il ne faut pas oublier que l'on a un smob à créer (cf encapsulation de la structure).

// crée un objet xosd à n lignes
static SCM make_xosd(SCM n) {
    xosd * w;
 
    SCM_ASSERT(SCM_INUMP(n), n, SCM_ARG1, "make-xosd");
    w = xosd_create(SCM_INUM(n));
    SCM_RETURN_NEWSMOB(xosd_tag, w);
}
 
// rien à marquer, le smob est simple
static SCM mark_xosd(SCM xosd_smob) {
    return SCM_BOOL_F;
}
 
// libération du smob
static size_t free_xosd(SCM xosd_smob) {
    xosd * osd = XOSD(xosd_smob);
    xosd_destroy(osd);
    return sizeof( xosd * );
}

Insistons sur le fait que la fonction de libération n'a absolument pas vocation a être appelée par l'utilisateur (d'ailleurs elle renvoie un size_t qui ne signifie rien en scheme). L'appel sera exclusivement interne.

Une fois ceci fait, la désallocation se fera automatiquement lors d'un passage du ramasse-miettes. Ainsi, si nous reprenons notre exemple de (set! xosd nil), au prochain passage du ramasse-miettes, l'ancien contenu de xosd sera détecté comme désormais inutilisable (car non atteignable) et free_xosd() sera appelé.

Définitions et export de module

Une fois que toutes les fonctions de la bibliothèque d'origine ont été encapsulées pour présenter un profil plus lisp, il est encore nécessaire de les enregistrer auprès de l'interpréteur à l'aide de la fonction scm_c_define_gsubr().

scm_c_define_gsubr("make-xosd", 1, 0, 0, make_xosd);

les trois paramètres entiers représentent respectivement le nombre de paramètres obligatoires, le nombre de paramètres optionnels, et une valeur booléenne signalant l'existence ou non d'un “reste” d'arguments, mais ce n'est pas le propos ici. Cette instruction associe au symbole d'identifiant “make-xosd” la fonction C make_xosd().

Le langage scheme possède un système de modules (que l'on peut voir comme une série d'espaces de (re)nommage), chacun exportant une interface à destination de l'utilisateur. Par défaut, les fonctions déclarées dans un module ne sont pas exportées (elles sont privées en quelque sorte), aussi nous devons explicitement exporter chacun des symboles de fonctions définis, et ce grace à scm_c_export() qui accepte une série d'arguments terminée par NULL

scm_c_export("make-xosd", ... , "xosd-display-string", NULL);

Compilation

Une simple compilation en bibliothèque partagée suffit à rendre les fonctions créées accessibles depuis guile, sous réserve qu'il existe une fonction particulière scm_init_<name>_module si la bibliothèque s'appelle lib<name>.so et que cette fonction déclare l'ensemble des symboles et fonctions définis.

SCM scm_init_xosdguile_module (void) {
    SCM module;
    module = scm_c_define_module("xosdguile", init_helper, NULL);
    return SCM_UNSPECIFIED;
}

où init_helper est une fonction chargée de la création du module proprement dit, en d'autres termes, il s'agit de la fonction contenant les instructions de définition et export vues précédemment.

Si notre fichier C s'appelle xosd_wrap.c, la ligne de compilation idoine est la suivante:

gcc -Wall -ansi -pedantic -shared -o libxosdguile.so -lxosd xosd_wrap.c

Exemple

Donnons maintenant un petit exemple d'utilisation. Pour être original, on veut afficher un “Hello World!” en bas de l'écran, centré avec un décalage supplémentaire de 80 pixels vers la droite. En outre, on désire qu'il disparaisse de lui-même au bout de 5 secondes.

(use-modules (xosdguile)) ;; chargement du module
(define osd-size 5) ;; on veut 5 lignes
(define osd-timeout 5) ;; les messages disparaissent en 5 secondes
(define osd (make-xosd osd-size)) ;; création d'un xosd
(define osd-position '(80 . 0)) ;; offset horizontal/vertical
 
(xosd-set-timeout! osd osd-timeout) ;; activation du timeout
(xosd-set-pos! osd 'bottom) ;; placement en bas
(xosd-set-align! osd 'center) ;; et au centre
(xosd-set-horizontal-offset! osd (car osd-position)) ;; décalage horizontal
(xosd-set-vertical-offset! osd (cdr osd-position)) ;; et vertical
 
(xosd-display-string osd 0 "Hello World !") ;; affichage

Voyons une capture d'écran de la bête en fonctionnement. Franchement vous attendiez quoi d'un “Hello World” ? :)

Références

projet (osd)
guile

lisp

zsi

Yann Hodique — 2010-02-18T11:31:30+01:00

« Blog Tags

Python Webservice, part 1 : a Twisted solution

2010/02/17 20:02

Yann Hodique

python, code, zsi, twisted

twisted

Yann Hodique — 2010-02-18T11:31:16+01:00

« Blog Tags

Python Webservice, part 1 : a Twisted solution	2010/02/17 20:02	Yann Hodique	python, code, zsi, twisted
Python Webservice, part 2 : a twistd integration	2010/02/19 00:01	Yann Hodique	python, code, twisted

test

Yann Hodique — 2010-02-18T11:30:56+01:00

« Blog Tags

Playground

2010/02/14 08:58

test

python

Yann Hodique — 2010-02-18T11:30:38+01:00

« Blog Tags

Python Webservice, part 1 : a Twisted solution	2010/02/17 20:02	Yann Hodique	python, code, zsi, twisted
Python Webservice, part 2 : a twistd integration	2010/02/19 00:01	Yann Hodique	python, code, twisted

Python Webservice, part 1 : a Twisted solution

Yann Hodique — 2010-02-17T20:20:16+01:00

As mentioned previously I've been spending some time recently working on a SOAP server mock. For this I've used Python Twisted and ZSI. Since the interaction between the 2 seems rather poorly documented (even Google doesn't know much about it), I thought I'd share what I've investigated and done.

Disclaimer: you should be familiar with SOAP and Python Twisted. I'll not cover anything in these areas.

For the sake of simplicity, I'll take a classical minimal server: an Echo service.

Let's start from the definition of such a service, as a WSDL. This one was randomly picked from there Anyway the implementation details are not so important here.

<?xml version="1.0" encoding="UTF-8"?>
<definitions
    xmlns="http://schemas.xmlsoap.org/wsdl/"
    xmlns:soap="http://schemas.xmlsoap.org/wsdl/soap/"
    xmlns:soapenc="http://schemas.xmlsoap.org/soap/encoding/"
    xmlns:http="http://schemas.xmlsoap.org/wsdl/http/"
    xmlns:xsd="http://www.w3.org/2001/XMLSchema"
    xmlns:tns="urn:ZSI"
    targetNamespace="urn:ZSI" >
  <types>
    <xsd:schema elementFormDefault="qualified" targetNamespace="urn:ZSI">
      <xsd:element name="Echo">
        <xsd:complexType>
          <xsd:sequence>
            <xsd:element name="value" type="xsd:anyType"/>
          </xsd:sequence>
        </xsd:complexType>
      </xsd:element>
    </xsd:schema>
  </types>
 
  <message name="EchoRequest">
    <part name="parameters" element="tns:Echo" />
  </message>
 
  <message name="EchoResponse">
    <part name="parameters" element="tns:Echo"/>
  </message>
  <portType name="EchoServer">
    <operation name="Echo">
      <input message="tns:EchoRequest"/>
      <output message="tns:EchoResponse"/>
    </operation>
  </portType>
 
  <binding name="EchoServer" type="tns:EchoServer">
    <soap:binding style="document"
                  transport="http://schemas.xmlsoap.org/soap/http"/>
    <operation name="Echo">
      <soap:operation soapAction="Echo"/>
      <input>
        <soap:body use="literal"/>
      </input>
      <output>
        <soap:body use="literal"/>
      </output>
    </operation>
  </binding>
 
  <service name="EchoServer">
    <port name="EchoServer" binding="tns:EchoServer">
      <soap:address location="http://localhost:8080/echo"/>
    </port>
  </service>
 
</definitions>

So really nothing fancy here. Just the definitions and shape of the messages/operations (only trivial ones).

ZSI offers the wsdl2py tool, that can process such WSDL files, and produce client bindings and server stub. In this article, the server part is what we want. (note the -w flag will trigger Twisted code generation)

$ wsdl2py -w echo.wsdl

EchoServer_server.py now contains the code for the service stub. Not much in there, the main part being the following method:

def soap_Echo(self, ps, **kw):
    request = ps.Parse(EchoRequest.typecode)
    return request,EchoResponse()

This is the one we should implement to make the service concrete. Here is a first attempt:

class EchoService(EchoServer):
 
    def __init__(self):
        EchoServer.__init__(self)
 
    def soap_Echo(self, ps, **kw):
        request, _ = EchoServer.soap_Echo(self, ps, **kw)
        return request, request

That will work. If you register this Service as a Twisted Resource, it will provide a functional SOAP Echo service, on which you can execute the following.

$ POST http://localhost:8080/echo
Please enter content (application/x-www-form-urlencoded) to be POSTed:
<soapenv:Envelope 
    xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" 
    xmlns:xsd="http://www.w3.org/2001/XMLSchema" 
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <soapenv:Body xmlns:ns1="urn:ZSI">
    <ns1:Echo>
      <ns1:value xsi:type="xsd:string">plop</ns1:value>
    </ns1:Echo>
  </soapenv:Body>
</soapenv:Envelope>
^D
<soapenv:Envelope xmlns:ZSI="http://www.zolera.com/schemas/ZSI/" xmlns:soapenc="http://schemas.xmlsoap.org/soap/encoding/" xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<soapenv:Header></soapenv:Header><soapenv:Body xmlns:ns1="urn:ZSI"><ns1:Echo><ns1:value xsi:type="xsd:string">plop</ns1:value></ns1:Echo></soapenv:Body></soapenv:Envelope>

Still, this is not very Twisted… I mean we don't take advantage of the asynchronous framework here, as the only method we wrote is definitely synchronous and we treat each call after one another. Would the soap_Echo() take time for whatever reason, that would introduce unacceptable delays for the user. Of course, there is no magic in the Deferred objects, which means than simply returning a Deferred in soap_Echo() will absolutely not work.

Digging a bit more in the ZSI Twisted glue, one can find a promising piece of code, appropriately named DeferHandlerChain. This is an alternative to the DefaultHandlerChain that's used (by default) by WSresource objects (hence EchoServer ones). The nice property of this handler chain is to put in place a callback chain from processRequest() to processResponse(), actually supporting Deferred objects returned by soap methods. In a word, what we need…

The code can become something like:

class EchoDeferHandlerChainFactory:
    protocol = DeferHandlerChain
 
    @classmethod
    def newInstance(cls):
        return cls.protocol(DefaultCallbackHandler, DataHandler)
 
class EchoService(EchoServer):
 
    def __init__(self):
        EchoServer.__init__(self)
        EchoServer.factory = EchoDeferHandlerChainFactory
 
    def soap_Echo(self, ps, **kw):
        request, _ = EchoServer.soap_Echo(self, ps, **kw)
 
        def _answer():
            return request
 
        # yeah, we have a very busy echo service... 5 seconds delay !
        d = deferLater(reactor, 5, _answer)
        return request, d

You can now try triggering several parallel Echo calls, and see that they will not be treated in sequence !

python, code, zsi, twisted

Read or add comments to this article

Magit extensions

Yann Hodique — 2010-02-14T13:05:08+01:00

Magit is a great emacs tool to help doing everyday's operations on Git repositories. In particular, I really like the contextual approach, that splits the global status in sections, meaning that the same high-level command can have subtle semantics differences when applied to different objects. As an example, the high-level `magit-discard-item' function obviously takes different steps when applied to an untracked file or to a staged hunk, or even to an entire branch.

The only disadvantage I see in the current implementation is the lack of modularity in it: a heavy use of (really nice, btw) macros makes it mandatory to have the complete set of features available when magit is loaded, so that the generated code is complete.

Still, Git has lots of contributed helpers, which means the only stable way of using them with Magit is to have them integrated upstream, which is not always possible (take the example of custom helpers that cannot be distributed) or does not even make sense (those helpers could be entirely bound to some specificities of a project).

I've been working on an attempt to make Magit more modular, by allowing the registration of so-called “extensions”. See code here: http://github.com/sigma/magit/tree/extensions

As a POC, I've extracted 2 features from the mainstream Magit, and converted them to extensions: those are magit-svn and magit-topgit. The way I see things, those 2 features are exactly the kind I'd like to add to magit for my daily work but that would not necessarily make sense as part of the core project. As far as I know, the current behavior when those 2 extensions are loaded is exactly the same as the regular upstream version (including placement of the various sections)

The current code allows you to:

define new kind of sections (for example a “topic” kind)
define how and where to display them (for example, right after the core “stashes” section)
hook new behavior for existing commands for those new sections (for example a way to discard such an item)
define new top-level commands, and associate menus

I tried to stay very close from the Magit way of doing things, and in particular the way you express things in extensions is exactly the same as what you do in the core code. Once again, tricky macros are at play to make the code look the same.

What I'd like to see in the future (though I don't have much time to do it right now :():

new extensions, to ease my daily life.
- a magit-p4 (perforce integration),
- maybe a magit-stgit (stgit in place of topgit for topic branches)
- something to integrate with VMware's ReviewBoard
code stabilization, and official proposal upstream (if there is an interest in this)
a way to activate those extensions on a per-project basis (I use both git-svn and git-p4 but never at the same time. same for topgit vs stgit)

An example being worth a thousand words, here is how to code the magit-topic extension. If you know the Magit code base, you should feel nearly at home.

Feel free to tell me what you think, any comment or advice is highly welcome !

emacs, git, code

Read or add comments to this article

Yann Hodique — 2010-02-14T12:16:25+01:00

Python Webservice, part 2 : a twistd integration

Twistd

In particular, it allows to write so-called plugins that make it possible to just launch services in the following way:

$ twistd echo

To achieve that, one has to create a file twisted/plugins/echo.py accessible somewhere from the python path (that is, the twisted directory should be a subdir of an element of the path).

Manhole

Example

Here is a fully working example of a twistd plugin for the Echo service, with a functional manhole service in addition.

from zope.interface import implements
from twisted.python import usage
from twisted.python.rebuild import rebuild
from twisted.plugin import IPlugin
from twisted.application.service import IServiceMaker, MultiService
from twisted.application.internet import TCPServer
from twisted.web.server import Site
from twisted.web.resource import Resource
from twisted.conch.manhole_tap import makeService as makeConsoleService
 
from EchoImpl import EchoService
 
class EchoOptions(usage.Options):
    optParameters = [["port", "p", 8080, "main port", int],
                     ["service-port", "s", 8081, "service port", int],
                     ["service-users", "u", "/etc/passwd",
                      "The path to a passwd-like authentication file"]]
 
class EchoServiceMaker(object):
    implements(IServiceMaker, IPlugin)
    tapname = "echo"
    description = "Sample Echo service"
    options = EchoOptions
 
    def makeService(self, options):
        # Create Service
        srv = EchoService()
 
        root = Resource()
        # we'll serve Echo at http://<ip>:<port>/echo
        root.putChild('echo', srv)
        siteFactory = Site(root)
 
        echo_service = TCPServer(options['port'], siteFactory)
        console_service = makeConsoleService(
            {"telnetPort": str(options['service-port']),
             "sshPort": None,
             "namespace": {"service": srv,
                           "rebuild": rebuild},
             "passwd": options['service-users']})
 
        svc = MultiService()
        echo_service.setServiceParent(svc)
        console_service.setServiceParent(svc)
 
        return svc
 
serviceMaker = EchoServiceMaker()

$ telnet localhost 8081
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
Username: yann
Password: x
 
>>> print service
<EchoImpl.EchoService object at 0x2fc8c10>
>>>

2010/02/19 00:01 · Yann Hodique · 0 Comments

python, code, twisted

Python Webservice, part 1 : a Twisted solution

Disclaimer: you should be familiar with SOAP and Python Twisted. I'll not cover anything in these areas.

For the sake of simplicity, I'll take a classical minimal server: an Echo service.

Let's start from the definition of such a service, as a WSDL. This one was randomly picked from there Anyway the implementation details are not so important here.

<?xml version="1.0" encoding="UTF-8"?>
<definitions
    xmlns="http://schemas.xmlsoap.org/wsdl/"
    xmlns:soap="http://schemas.xmlsoap.org/wsdl/soap/"
    xmlns:soapenc="http://schemas.xmlsoap.org/soap/encoding/"
    xmlns:http="http://schemas.xmlsoap.org/wsdl/http/"
    xmlns:xsd="http://www.w3.org/2001/XMLSchema"
    xmlns:tns="urn:ZSI"
    targetNamespace="urn:ZSI" >
  <types>
    <xsd:schema elementFormDefault="qualified" targetNamespace="urn:ZSI">
      <xsd:element name="Echo">
        <xsd:complexType>
          <xsd:sequence>
            <xsd:element name="value" type="xsd:anyType"/>
          </xsd:sequence>
        </xsd:complexType>
      </xsd:element>
    </xsd:schema>
  </types>
 
  <message name="EchoRequest">
    <part name="parameters" element="tns:Echo" />
  </message>
 
  <message name="EchoResponse">
    <part name="parameters" element="tns:Echo"/>
  </message>
  <portType name="EchoServer">
    <operation name="Echo">
      <input message="tns:EchoRequest"/>
      <output message="tns:EchoResponse"/>
    </operation>
  </portType>
 
  <binding name="EchoServer" type="tns:EchoServer">
    <soap:binding style="document"
                  transport="http://schemas.xmlsoap.org/soap/http"/>
    <operation name="Echo">
      <soap:operation soapAction="Echo"/>
      <input>
        <soap:body use="literal"/>
      </input>
      <output>
        <soap:body use="literal"/>
      </output>
    </operation>
  </binding>
 
  <service name="EchoServer">
    <port name="EchoServer" binding="tns:EchoServer">
      <soap:address location="http://localhost:8080/echo"/>
    </port>
  </service>
 
</definitions>

So really nothing fancy here. Just the definitions and shape of the messages/operations (only trivial ones).

$ wsdl2py -w echo.wsdl

EchoServer_server.py now contains the code for the service stub. Not much in there, the main part being the following method:

def soap_Echo(self, ps, **kw):
    request = ps.Parse(EchoRequest.typecode)
    return request,EchoResponse()

This is the one we should implement to make the service concrete. Here is a first attempt:

class EchoService(EchoServer):
 
    def __init__(self):
        EchoServer.__init__(self)
 
    def soap_Echo(self, ps, **kw):
        request, _ = EchoServer.soap_Echo(self, ps, **kw)
        return request, request

That will work. If you register this Service as a Twisted Resource, it will provide a functional SOAP Echo service, on which you can execute the following.

$ POST http://localhost:8080/echo
Please enter content (application/x-www-form-urlencoded) to be POSTed:
<soapenv:Envelope 
    xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" 
    xmlns:xsd="http://www.w3.org/2001/XMLSchema" 
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <soapenv:Body xmlns:ns1="urn:ZSI">
    <ns1:Echo>
      <ns1:value xsi:type="xsd:string">plop</ns1:value>
    </ns1:Echo>
  </soapenv:Body>
</soapenv:Envelope>
^D
<soapenv:Envelope xmlns:ZSI="http://www.zolera.com/schemas/ZSI/" xmlns:soapenc="http://schemas.xmlsoap.org/soap/encoding/" xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<soapenv:Header></soapenv:Header><soapenv:Body xmlns:ns1="urn:ZSI"><ns1:Echo><ns1:value xsi:type="xsd:string">plop</ns1:value></ns1:Echo></soapenv:Body></soapenv:Envelope>

The code can become something like:

class EchoDeferHandlerChainFactory:
    protocol = DeferHandlerChain
 
    @classmethod
    def newInstance(cls):
        return cls.protocol(DefaultCallbackHandler, DataHandler)
 
class EchoService(EchoServer):
 
    def __init__(self):
        EchoServer.__init__(self)
        EchoServer.factory = EchoDeferHandlerChainFactory
 
    def soap_Echo(self, ps, **kw):
        request, _ = EchoServer.soap_Echo(self, ps, **kw)
 
        def _answer():
            return request
 
        # yeah, we have a very busy echo service... 5 seconds delay !
        d = deferLater(reactor, 5, _answer)
        return request, d

You can now try triggering several parallel Echo calls, and see that they will not be treated in sequence !

2010/02/17 20:02 · Yann Hodique · 0 Comments

python, code, zsi, twisted

Magit extensions

I've been working on an attempt to make Magit more modular, by allowing the registration of so-called “extensions”. See code here: http://github.com/sigma/magit/tree/extensions

The current code allows you to:

define new kind of sections (for example a “topic” kind)
define how and where to display them (for example, right after the core “stashes” section)
hook new behavior for existing commands for those new sections (for example a way to discard such an item)
define new top-level commands, and associate menus

What I'd like to see in the future (though I don't have much time to do it right now :():

new extensions, to ease my daily life.
- a magit-p4 (perforce integration),
- maybe a magit-stgit (stgit in place of topgit for topic branches)
- something to integrate with VMware's ReviewBoard
code stabilization, and official proposal upstream (if there is an interest in this)
a way to activate those extensions on a per-project basis (I use both git-svn and git-p4 but never at the same time. same for topgit vs stgit)

An example being worth a thousand words, here is how to code the magit-topic extension. If you know the Magit code base, you should feel nearly at home.

Feel free to tell me what you think, any comment or advice is highly welcome !

2010/02/14 13:05 · Yann Hodique · 0 Comments

emacs, git, code

Getting things done with GWT

It's been a long time I wanted to give a try to Google's GWT. Also, lately I needed more and more a robust and straightforward implementation of David Allen's Getting Things Done to help me organize my work. There it is, let's combine those two whishes and produce a working tool.

GWT in brief

This toolkit gives you the opportunity to code your web application entirely in Java while deploying it under its compiled form (HTML + JavaScript / Java servlet). The advantages are obvious : during the development, you have access to all the facilities of a Java environment (nice enough language when compared to JavaScript , debugger), to its robustness, and so on.

I've always been attracted by compilers from one high-level language to another lower-level ones. The GWT compiler is clearly a success in this area. It makes it really easy to introduce a layer of AJAX in your web application without even thinking about it.

GWT is composed mainly of the following components:

a graphical toolkit following roughly the way of AWT, QT, … so you feel like home really fast
a minimal implementation of the most useful java.lang and java.util classes for client use (translated to JavaScript after compilation)
a services API that deals with AJAX (client-side) and Java servlets (server-side)

There are also other interesting aspects (JUnit integration for example), but I'll leave them for another day. All this is integrated in a very clever and usable way, which makes it easy to develop your application. During development, all you're going to use is provably “hosted mode”, which is a pack provided by Google, that includes an embedded Tomcat server (for servlets), an embedded browser (based on mozilla), so that you can have a sample of your application running in one click without needing to deploy anything. Plus, this “hosted mode” really runs your Java code, so that you may as well debug it in the meantime. Brilliant !

A small issue that is worth noting : it looks like when you use “hosted mode”, some informations are transmitted to Google. I have to admit I'm a bit disappointed about that.

GTD methodology

GTD seems to be very popular these days. The number of articles (in blogs mostly… mine is no exception now ;)) is constantly increasing, and the number of related tools grows in proportion. Still I did not find a single tool that really satisfying.

Here are my expectations for a perfect GTD tools:

it should be mostly online : I can't rely on a single device (especially not a laptop, and a PDA is not reactive enough)
there should be an offline client/cache/whatever : I'm not always online, even if I am most of the time
it should be fast, because most of the time you just add really small tasks to your list
it should be nice to look at, because you'll use it a lot

GWTD started

With my friend Alexandre, we started to write the tool of our dreams : GWTD. Well, the name is nearly obvious I guess… still we will have to give it a meaning for completeness purpose. The two goals are :

to discover the possibilities of GWT
to have a working implementation of GTD that fills our needs.

The project is still in very early stage of development (given the small amount of time we can afford on that, it's no surprise), bu we hope to come up with a release soon enough. The code (distributed under Apache licence 2.0) is available at code.google.com, and a Google group has also been created. Stay tuned !

2010/02/14 08:58

code, google, gwt, gtd

Upgrading Dokuwiki

I just finished upgrading my Wiki to latest development version of Dokuwiki. This was not an easy task because I've made quite a large number of modifications that needed to be ported or incorporated. I've decided to maintain a Darcs branch of both my Dokuwiki and Arctic theme.

For now this is no complete success since I access the filesystem via sshfs, and it looks like it lacks a proper lock implementation. I'll try to find a better way

By the way, I also upgraded several plugins to their latest version, so that you (and I) may experience some weird and uncontrolled border effects. That's the price for living on the bleeding edge.

2010/02/14 08:58

wiki, darcs

Older entries >>

Read or add comments to this article