Adventures in Buildout: Zope 2.12 meets SOAP

After 3 years of hanging around in Plone 2.5, I am finally getting around to upgrading our infrastructure to Plone 4. Part of that upgrade includes pulling out all of our services and api calls into a separate buildout, a solution that does not actually need the Plone package. A big problem with this migration was that we still use SOAP to talk to external vendors, most python SOAP modules suck, and none of them are fully integrated with a database/framework.

As a satisfied user of SOAPSupport for many years, my first inclination was just to continue with an old version of Zope and continue to use that.  However, the project hasn't been supported/updated in a long time and I was excited to take advantage of memory improvements in Python 2.6, a fully eggified Zope, and Blob storage among other things. There is a great article on configuring SOAP in Plone, which targets Zope 2.11 and custom products, however its still very mind bending for those of us with simpler needs.

While updating the z3c.soap package for Zope 2.12 compatability, I was also busy trying to grok "new" buildout concepts (I know - I've been in the closet for a while). I couldn't find one really solid simple tutorial on getting started with buildout. After a roller coaster ride of emotions and awesome help from the Plone community, I *think* I've figured out the simplest way to get any python SOAP server up and running with (or without) buildout. I have not addressed the WSDL or complex types questions yet, since I don't need them. I also said simplest, which means I had little regard for proper classing and blah blah blah... yell at me in the comments if it's grossly off-key or if I missed something - it's bloody likely.

Below is an introduction to configuring SOAP in Zope as well as an extra detailed buildout process for old-schoolers like myself who are used to Products based install. I hope it helps someone get up and running with a python soap server fast, in a repeatable, buildout style.

FYI, this can also easily be customized to support Plone and other zope based apps since its buildout based.  My goal was to make this like MySite, where you can start and build off of it. In the end I don't think I have even come close the greatness that was MySite but you can still download the finished package if you don't care about the how and just want the now, or if you want to follow along easier.

NOTE: At the time of writing virtualenv has a bug with this buildout so if you get an ImportError for shutil, just use regular old python. This is also why this tutorial is not using virtualenv, although its perfect for this scenario otherwise.

Summary

For those who don't care about buildout or the details, here is the recipe for simple zope + soap. Customize away!

The Details: Creating the Buildout

I won't go into the philosophy of buildout since its all over so we can get to implementing. I keep all of my buildouts in a directory in my home folder, called 'buildouts', so I always know where to start. Also, before you get to the end and kick yourself, I'm using Python 2.6 here. I have not tested it with any other version. Caveat Emptor.

First, make sure that you have the buildout package installed:

> sudo easy_install zc.buildout

Create a buildout environment, with all the scripts we need to get started

# create a new directory for our new buildout
> cd ~/buildouts
> mkdir zsoap
> cd zsoap
# initialize the buildout environment
> buildout init
Creating '/Users/eleddy/buildouts/zsoap/buildout.cfg'.
Creating directory '/Users/eleddy/buildouts/zsoap/bin'.
Creating directory '/Users/eleddy/buildouts/zsoap/parts'.
Creating directory '/Users/eleddy/buildouts/zsoap/eggs'.
Creating directory '/Users/eleddy/buildouts/zsoap/develop-eggs'.
Generated script '/Users/eleddy/buildouts/zsoap/bin/buildout'.
# optional: include bootstrap.py
> wget http://svn.zope.org/*checkout*/zc.buildout/trunk/bootstrap/bootstrap.py

You don't have to include the bootstrap.py file, but it makes things hella easier and I'll assume that you have it for the rest of this. After initializing the buildout, you will have a bunch of extra directories in your zsoap folder, including bin, buildout.cfg, develop-eggs, eggs, and parts. Don't worry about them for now - just make sure they are there.

Next, edit buildout.cfg so it looks like this. I'll explain it all while we wait for the buildout to run.

Finally run the buildout:

> python bootstrap.py
> ./bin/buildout

Note that whatever version of python you run bootstrap with is the version that zope will be running with. This recipe was created in Python 2.6, since Zope 2.12 runs swimmingly on it. If your system python is not 2.6, make sure to alter your bootstrap command to reflect that. 

While its running, you may see errors like "SyntaxError: ("'return' outside function",...". Don't worry about those. It's simply trying to compile scripts, which you can't do.

The Breakdown

Once the buildout starts going, go ahead and take a shower, walk the dog, or read the line by line description of what is happening:

1. [buildout]

2. parts = scripts

3.             instance

4.             test

5. 

6. extends = http://svn.zope.org/*checkout*/Zope/tags/2.12.5/versions.cfg

7. versions = versions

8. eggs = z3c.soap

9. extensions = buildout.threatlevel

Lines 1-4 say that this buildout has three parts that need to be run. When you run this, you will see that the "parts" directory that was automagically created from initializing the environment now has 3 folders with the exact same names. Coincidence? You decide. 

Line 6 is a link to download an automatically configured buildout for installing zope 2 in a delicious eggy format. To change versions of zope, just modify this url. Go ahead. Try it.

Line 7 Indicates that we are going to force some packages to get a certain version, instead of the latest. This is called "pinning" a version. Some packages have conflicting versions, and you may need to pin the version which satisfies everything. This line simply tells buildout that there is a section, called versions, which will do just that.

Line 8 indicates which additional eggs we need to get the project going. It will ask PyPi for the latest version of this eggs (unless we pinned it) and install them. Notice that we didn't have to include Zope2 because it's handled with the scripts section with a special recipe that you'll see later.

10. [versions]

11. Zope2 = 2.12.5

12.    

10-11 show how to pin a version of a package. This is not necessary for this case since the Zope2 recipe does this for us, but other packages probably won't have this luxury. Maybe you want to pin to a minor version, for example. Furthermore I already numbered these lines and I'm too lazy to go through and renumber them.

13. [scripts]

14. recipe = zc.recipe.egg:scripts

15. eggs = Zope2

Lines 13-15 might as well be magic. They install Zope2, the inner workings of which I have no idea.

16. [instance]

17. recipe = plone.recipe.zope2instance

18. user = admin:admin

19. http-address = 8081

20. products = ${buildout:directory}/products

21. debug-mode=on

22. zcml = z3c.soap

23. eggs = ${buildout:eggs}

Lines 16-23 create a new zope2 instance. It's pretty self-explanatory except for a couple lines. Line 22 males sure that we include the z3c.soap package, and run the zcml slug that initializes it. This patches the publisher to make it accept soap requests. Line 23 makes sure that all the eggs created in other places of this buildout are put in the path of the instance. Without this, it won't be able to import anything.

24. [test]

25. recipe = zc.recipe.testrunner

26. eggs = z3c.soap

And finally, we must not forget to include test cases to make sure everything is running smashingly. 

Up and Running

Assuming everything runs correctly, we can start testing things out. First we want to validate that everything looks like its running ok.

> ./bin/test

It's that easy. This should run fine and pass all tests. Yeehaw! Let's start the new zope 2 instance in foreground mode:

> ./bin/instance fg

Still easy. Woohoo! If the slug was installed correctly, z3c.soap will have a line in the trackback that says something like "INFO Zope z3c.soap: modified ZPublisher.HTTPRequest.processInputs". Congratulations, you are ready to serve SOAP requests.

Taking it a Step Awesomer

Now what? Well, you can follow the original tutorial and add wsdls and custom type and all that fancy stuff. Since I'm lazy I prefer to do everything through scripts. In the ZMI, I like to add a folder called 'services' and then just pile up python scripts in there. Err... I mean Script (Python)'s. Let's verify it's all SOAPed up like we want. Add a script called 'test' in a folder called 'scripts' that simply returns a string 'hello world!'. Then you can use the SOAPpy package to see what's really going on. From the python prompt:

>>> from SOAPpy import SOAPProxy, URLopener
  >>> url = "http://localhost:8081/services"
  >>> namespace = "http://plone.org"
  >>> server = SOAPProxy(url)
  >>> server.config.dumpSOAPOut = 1
  >>> server.config.dumpSOAPIn = 1
  >>> print server._ns(namespace).test()[0]
  hello world!

Every script you add will have a SOAP interface built in. How cool is that? Also, a tip from the trenches, use dictionaries to pass back values for more complex values. In my experience, it's more compatible with Java stacks, which I assume you have to deal with, because otherwise you wouldn't be using SOAP. I digress...

So I know you're thinking, what is the awesomer part of "taking it a step awesomer"? Well, if you want a repeatable buildout you don't want to store your scripts in the ZMI. I know I didn't, since we have an SVN repository that is way better for such things (again, I'm behind, I'm not cool enough to Git or Hg yet). We can do is use the CMF Filesystem Directory view and store all of our scripts on the filesystem. While they are not editable in the ZMI, but the can be executed and customized.

I tried my hardest to find a way to do this without making a product, but I can't find a way to register a CMF directory view without being in the context of a product. So let's walk through making one - you probably need it anyways. It's slightly annoying, but at least this way it won't get wiped by re-running buildout. If you already have a product then skip the paster part and just add the necessary slugs.

First make sure that you have access to paster. If you can't use paster from the command line,

> sudo easy_install -U ZopeSkel

Then decide a name for the product, and run paster. I will use z.soap for this purpose.

> cd develop-eggs
> paster create -t basic_zope
> ...blah blah blah

Next we need to add our new egg to the buildout.cfg, mark it as a development package, and include it in our zope instance. Update your buildout.cfg to look like this, with the bold lines indicating the changes. Hopefully by now the buildout thing is starting to make sense. But we aren't done yet.

Now we need to actually register a services directory. Let's make the directories first

> mkdir ~/buildouts/zsoap/develop-eggs/z.soap/z/soap/skins
> mkdir ~/buildouts/zsoap/develop-eggs/z.soap/z/soap/skins/services

All of the Script (Python)'s that we want to be version controlled can go in the services directory. You can version control the whole buildout, or of course register your new egg with PyPi and replace that whole mess with eggs = my.egg.

Tell zope about the directory that you created by editing your new eggs configure.zcml, which will be in ~buildouts/zsoap/develop-eggs/z.soap/z/soap/configure.zcml. Add the registerDirectory directive to point to the folder you just created, adding the cmf namespace if needed. The final configure.zcml should look like this, with the changes highlighted.

Phew! Are we done yet? No. Re-run buildout and restart your instance

> ./bin/buildout
> ./bin/instance fg

At the root of the ZMI, add a Filesystem Directory View and you will see that your directory is listed. Let's put the same test script that was in the zodb the first time in the filesystem now. In your new skins, services directory, add a file, test.py with the contents below and run the SOAPpy test above. Viola! Now you can add scripts until you are sick. They can compute, forward info, act as an API, and even interact with your products. No restart needed to add/change scripts since its a filesystem view.

##Script (Python) "test"
##bind container=container
##bind context=context
##bind namespace=
##bind script=script
##bind subpath=traverse_subpath
##parameters=
##title=
##
return "hello world!"

But wait, you don't want to do that last step manually? You want that automated? I did too! Last but not least, let's add the directory view on startup if it's not already there. There is a special initialize function that we can call on startup. Open up develop-eggs/z.soap/z/soap/__init__.py and make it look like this. Now instead of calling zope2 initialization, we need to call our own package initializer by modifying the configure.zcml to look like this. Now start up the zope instance and it will come alive!

So, if there is anything wrong here or it could be done easier, please comment and I will update. The last part was definitely not as easy to setup as I would have hoped. And because I'm so nice, you can download this package, untar, bootstrap, build, and go crazy with customizing.