Valentine Blog

There is no straightforward way to run plone, repoze and buildout yet, atleast not if you want to use a plone/zope version other than the one repoze.plone ships with. Chris McDonough has this ticket assigned to him, so there will hopefully be an official and easy way soon. Until then this is how I got it to work.

Buildout.cfg:

[buildout]
parts = plone repoze zope2 instance addpath
eggs =
find-links = http://dist.repoze.org

[plone]
recipe = plone.recipe.plone

[zope2]
recipe = plone.recipe.zope2install
url = ${plone:zope2-url}

[repoze]
recipe = zc.recipe.egg
eggs =
     PasteScript
     WSGIUtils
     repoze.zope2
     deliverance
     ${plone:eggs}

[instance]
recipe = plone.recipe.zope2instance
zope2-location = ${zope2:location}
user = admin:admin
http-port = 8080
debug-mode = on
verbose-security = on
eggs =
    ${buildout:eggs}
    ${plone:eggs}
products =
    ${plone:products}

[addpath]
recipe = z3c.recipe.runscript
update-script = addpath.py:main
install-script = addpath.py:main

This is not that different from a normal plone buildout. What's new is [repoze] which installs paste, deliverance and repoze eggs and scripts to enable wsgi.

When you want to start the instance you can either start it the old way without enabling repoze or wsgi::

  $ bin/instance fg

  $ bin/paster serve paste.ini

You need do add the paste.ini file to your buildout. The file says where it can find zope.conf, what server should run and what middleware should be in the wsgi pipeline.

Last in buildout.cfg there's an 'addpath' section, which is a little hack to come around the fact that repoze has a dependency on zopelib which is an egg with all of zope2. But here I want to use the version of zope that is officially supported for the current plone version instead of the version that comes with repoze. So addpath runs a script that adds 'parts/instance/lib/python' to the top of the egg listing in bin/paster. The result is that the compiled zope will be used instead of repoze's zopelib.

You can find a buildout with all necessary files in collective trac or collective svn.

To make repoze/wsgi easier with buildout I think we would need to make a repozpe.zope2-like package without the plone and zope dependencies that repoze.zope2 currently has. Then you could pass 'extra_paths = ${zope2:location}/lib/python' to zc.recipe.egg and the compiled zope2 would be found. That would make [addpath] obsolete. This would be a first step to ease the use of repoze in buildout.

I had a great time at the snow sprint. I paired with Martijn Faasen and Robert Marianski and we implemented viewlets for Grok. It's nearly finished. It is working, but it needs a few more tests before it will be merged to trunk.

This is what your viewlet code could look like:

class PortalHeader(grok.ViewletManager):
    grok.name('header')

class Logo(grok.Viewlet):
    grok.viewletmanager(PortalHeader)
    grok.template('logo')

    def update(self):
        self.willBeAvailableInTemplate = 8

A template that wants to render the 'header' viewlet would include this:

<div tal:content="structure provider:header" />

I wrote a sample grok application, grokremoteinclude , to make sure the grok viewlets were compatible with other zope3 packages dealing with viewlets. It's a buildout, running it will install the grok app, nginx and varnish. Read the README.txt for instructions how to get going. The application uses SSI which allows viewlets to be either cached or retrieved from another server. The same technique can be used in Plone which was recently reported by Sasha Vincic.

Snowsprint 2008 is over for this year and many thanks to Lovely Systems and all participants.

Since Lovely Systems has deployed some fast sites I was eager to see if I could reuse some of it with Plone. One of the products was lovely.remoteinclude which enables viewlets to be accessed through own unique urls and therefore cached separately. With nginx you can then use SSI (server side include) to assemble the page instead of doing it all in Plone. The lovely.remoteinclude hooks up adapters for all IIncludableViews if they send the event zope.contentprovider.interfaces.IBeforeUpdateEvent. The adapter then checks if the view is called from within a page or directly. If it is called within a page the render method is changed so it renders the url to the view, and when called directly, no changes are made.

My task was to try to make Plone portlets accessible through urls. This was little bit harder then just marking the PortletManager and PortletRender with IIncludableView since they where not normal viewlets or viewlet mangers and there was no acquisition in portlets. Anyway after some pair programming with Tom Gross we managed to create a package valentine.remoteinclude (we didn't dare to name it plone.remoteinclude). This package is a proof of concept that lovely.remoteinclude can be used on Zope2 and Plone. It will provide urls for portlets that are available on the plone root. While we were learning the plone portlet machinery we found the namespace ++contextportlets++ which we reused to generate urls for our portlets. We also added a namespace for portlets since the ++viewlet++ didn't work for plone portlets. An url to a default calendar portlet will look like:

<!--#include virtual="/++contextportlets++plone.rightcolumn/++portlet++calendar" -->

Together with Whit Morriss and help from Tim Terlegård(who made lovely.remoteinclude work with Grok viewlets) we set up a buildout that will build a nginx and a varnish prepared for SSI and lovely.remoteinclude. The templates were made from a configuration provided by Lovely Systems. With this buildout and the valentine.remoteinclude enabled on a plone trunk builout we where enable to render a normal plone page where nginx assembled the calendar portlet with the rest of the page.

Tom and I couldn't figure out how to get the information from a portlet from which context the "mapping" (portlet storage) and then the portlet was fetched we where not able to generate urls for context specific portlets other then the root. This information seems to be hidden by the PortletFetcher in Plone. Ideas on how this could be solved are welcome.

With your help I think we can create a good SSI solution for Plone which would enable us to include structure and content in the page from different servers and have it assembled before sent to the user. This would be one step in the direction of using the right tool for the job and not put everything in Plone.

Technorati Tags: plone, zope, nginx, ssi, snowsprint

First day of Perfomance sprint in Copenhagen has passed and it really feels like we all have gotten productive from the start. 10 of us are seating at Symbion where Headnet is located and we have two remote sprinters.

Me and and Matt Hamilton started to work on improving the speed of cataloging while you move/rename a tree of objects. Matt had already an idea about using md5 hash on ZCTextIndex which in a quick test gave 30% speed improvement when you edit a document but don't change the body. But when we renamed objects we didn't get this improvement because the object got unindexed and then indexed on the new path so we tried do some magic in ObjectWillBeMovedEvent which would move the index in catalog for this object to the new path instead of unindexing and then get reindexed on ObjectMovedEvent instead of clean index. This way we could benefit from our speed improvement with md5 since the body isn't changed.

This was quickly done and the profiling after this indicated that object_provides indexing is taking way too long. We found out that the calls to zope.component.interface.interfaceToName was to be blamed for this. We tried some simple caching on it with plone.memoize and yeah! We slashed the indexing time by half on rename on a site with 300 random content objects! We let Martijn Pieters take a look at the method and why it was so slow just for an interface name. He found out that Plone really shouldn't use this for object_provides since it was there for a specific use case (I hope Martijn can comment on which) .

Martijn commited the fix, so current plone trunk has an indexing that is twice as fast as yesterday. This commit does not include the magic on ObjectWillBeMovedEvent and md5 so expect more speedups :)

NB! This fix is for trunk (plone 3.x) but me and others have the object_provides index in older Plones so please check the diff (see also) and fix your indexes to get some speed.

Thanks to Nate at Jazkarta for sponsoring, we got a nice sushi delux meal for dinner yesterday.

lovely.remotetask is a zope3 package for running asynchronous tasks. A service is running as a separate thread (not using any of zope's own threads). You define a class that performs a task. The task can be executed by adding it to the service (which adds it to its queue). It can look like this:

   >>> service = getUtility(ITaskService, name='LongRunningJob')
   >>> service.add(u'exampletask', 'input to the task')

The service will execute the task as soon as the other tasks in the queue are completed. The service and the task are registered as named utilities in zcml.

lovely.remotetask is ported to Zope 2.9 and works on 2.10 as well. It depends on zc.queue which runs on 2.9/2.10 without modification. The code is found in zope svn

The zmi (managing services and tasks) is not ported, but there is an example in svn that tells how to do it programmatically. Removing tasks can already be done in zmi as the utilities are added to the utilities folder in the site root.

Headnet sponsored this work.

Time for the annual Plone Conference and I must say I am very happy to see how the preparations are done. Looks like we will have a great conference and probably the largest one. Two days before the trip I looked through the agenda and as always it's hard to choose. First day we have Joel Burton's Building a humane CMS for Plone: updated tutorial and Martin Aspeli's Extending and Customising Plone 3 that are colliding. I am really interested to hear what Alexander Limi has learned at Google, so Simple and effective techniques for better usability is booked. If I choose Joel's talk I'll probably visit Geir Bækholt's Portlets in Plone 3. On the second day I can recommend the talk about the foundation by Paul Everitt but I am also very interested in Duco Dokter's Plone for the enterprise market: technical musing on caching, Clustering and Single Sign-On since this is the knowledge we need for big sites with Plone.

I'll probably end up changing my mind several times during the days and one or two regrets missing some talks but I hope it will be a nice time like all other conferences and a chance to meet friends.

Oh one more thing don't forget board nominations if you know anyone that should be on the board, even your self. Last day tomorrow Monday.

See you at the conference.

/Sasha