Zope Projects

Zope is an open source, flexible, platform independent web application server framework especially useful to build dynamic websites. You find more information about Zope via Zope's home page.

This page contains my contributions to the Zope software fund. They are either completely free, or covered by an open source or open content license. Note however, that I consider whether I will expressly forbid the use of my software by institutions of the European Union (Councel of Ministers, Commission, Parliament, ...) and member governments supporting software patents. This is in protest against a recent decision of the Councel to make software patentable. This decision ignores a Parliament vote that pleaded to keep logic (algorithms, business rules) un-patentable. Of course, only new versions will be affected by such a change -- should it come.

Newer versions of my software are now maintained on the Python package index ("PyPI"). In general, they can be used for Zope 2.11+. This site maintains versions that can still be used for earlier Zope versions. Any reworked version will move over to "PyPI".

1 No software patents please

If you live in the European Union, please consider signing the petition to forbid software patents in this area. Software patents threaten free/open source software development by individuals and small companies because they introduce the danger to accidentally infringe one of the trivial patents with the risk of immense legal costs and dammage claims. Only large multi-national companies can afford the costs of a legal department that continuously checks against the huge number of patents and defends against (potentially) unjustified patent infringement claims.

2 Zope3 in Zope2

Zope3 components can be used in Zope2 applications.

3 Reviews

4 Book: Building dynamic WebSites with Zope

This section is devoted to what is already available of my Zope book Building dynamic WebSites with Zope -- Concepts, Components, Programming. These are 3 chapters of about 8. The available chapters, too, are likely to change considerably before final publication, because Zope is a fast developing system: what is true today may no longer hold tomorrow. The chapters currently describe Zope 2.3 with some additions for Zope 2.4. I would say, they are in a beta, i.e. already quite usable state.

5 Products

Products are one of Zope's extension facilities. Usually, they provide new object classes that can be used to build the Website.

5.1 DigestAuth

DigestAuth provides HTTP Digest Authentication (RFC 2617) for Zope. More precisely, it provides a Digest Authentication Crumbler which works similar to Shane's CookieCrumbler: it turns HTTP digest authentication info provided by the client into HTTP basic authentication info and thereby allows to use user folders expecting basic authentication with the much more secure digest authentication.

Digest authentication (at least the implemented crumbler) requires that the password is available in cleartext. It must not be encrypted.

I plan to provide also a PAS (PluggableAuthService) module supporting digest authentication in a future version.

Download version 0.1 7 kb TGZ archive.

5.2 CompiledExecutables

CompiledExecutables defines a set of executable objects that are precompiled and the compilation stored either in the ZODB or in the file system. It currently defines CompiledPageTemplate, FSCompiledPageTemplate and FSCompiledPythonScript (PythonScript is already compiled; although in a somewhat stupid manner (in __setstate__ rather than when used)).

Storing the compilation result avoids expensive recompilation. This can drastically speed up access to the first pages after startup -- especially for large CMF based sites.

The product does not register anything. It is up to you to register the filesystem based objects with Products.CMFCore.DirectoryView.registerFileExtension or the ZODB based objects by calling their initialize function.

The product works by making instance methods and code objects picklable. This might appear as a security risk. However, unpickling untrusted pickles is always a security risk (because the pickler decides what is picklable and not the unpickler).

Version 0.5 customized a 'CompiledFSPageTemplate' into a caching policy manager aware version of PageTemplate.

Version 0.4 is now tested against a standard Zope 2.8.1 (without our local modifications).

Version 0.3 fixes an import bug in FSCompiledPythonScript and defines a _compile method for CompiledPageTemplate to make it more consistent with scripts.

Version 0.2 handles a read only storage correctly, improves a work around for a PythonScript bug and fixes an initialization bug in FSCompiledPythonScript.getBindingAssignments.

Download version 0.5 4 kb TGZ archive.

5.3 TrustedExecutables

TrustedExecutables is a set of executable objects unrestricted by Zope's security. Currently, it contains TrustedPageTemplateFile, TrustedFSPageTemplate and TrustedFSPythonScript. TrustedFSPageTemplate and TrustedFSPythonScript are registered with the filename extensions xpt and xpy, respectively.

As Zope's security checks are expensive, avoiding them can drastically speed things up. On the other hand, these objects must make their own security checks at places where access control is required.

Use with extreme care!

ATTENTION: It is not unlikely that this product breaks between Zope releases, as it uses undocumented implementation details. It is probably not very difficult to fix things again but you will need programming skills to do so.

5.3.1 Version 1.0

With Zope 2.10, Zope2 switched to the Zope3 page template implementation breaking the TrustedExecutables 0.x versions' trusted PageTemplate variants. From version 1.0 on, TrustedExecutables is adapted to the Zope3 page template implementation (and no longer supports the old Zope2 implementation). The new versions are maintained as Products.TrustedExecutables on PyPI and can be installed via easy_install Products.TrustedExecutables (provided that "setuptools/easy_install" is installed). The easy installed versions require Zope 2.11 or above. Repackaging (moving TrustedExecutables into your "$INSTANCE_HOME/Products") might allow you to use it with Zope 2.10, but I have not tested this.

5.3.2 Version 0.x

These versions only work for Zope upto version 2.9. From Zope 2.10 on, Zope2 uses Zope3's page template implementation breaking the trusted PageTemplate variants. If you are using Zope 2.11 (or above), you can use the Products.TrustedExecutables on PyPI.

Version 0.5.2: apparently, I have problems to get the 0.5 version right. This is another fix for the pdb debuggability.

Version 0.5.1 provides the features promissed for 0.5 (they went accidentally into the variant maintained for my employee).

Version 0.5 makes 'TrustedFSPythonScript' debuggable via 'pdb/zdb' (with correct line numbers) using code from Chris Wither's 'zdb'.

Version 0.4 copies utilities (such as 'test', 'reorder') from TemplateDict (the builtin namespace for untrusted code) to Python's __builtin__ module (used by trusted code). This way, trusted PageTemplates and trusted DTMLDocuments can use the same variables as untrusted ones. Note, that this might have some ramifications for other file based code, too.

Version 0.3 became necessary to adapt to the cosmetic changes introduced in Zope 2.7.1

New in version 0.2:

Download version 0.5.2 (for Zope 2.7 upto 2.9 (including) 6 kb TGZ archive. From version 1.0 on, you can download TrustedExecutables as Products.TrustedExecutables from PyPI. These versions support Zope 2.11 (or above).

5.4 AdvancedQuery

AdvancedQuery is a Zope product aimed to overcome several limitations of ZCatalog's native search function.

Like ZCatalog's search, it supports elementary index searches. While ZCatalog can combine such elementary searches only by "and", AdvancedQuery allows them to be combined arbitrary with & (and), | (or) and ~ (not).

While ZCatalog supports an efficient sorting via an index on one level, AdvancedQuery supports sorting via any level of (field) indexes. Moreover, it sorts the result incrementally -- only as far as you access your result. This can drastically speed up the time required for sorting. It uses Python's generators for this (and thus requires Python 2.2 or better).

From version 1.1 on, AdvancedQuery supports incremental filtering. See the documentation. Note that incremental filtering is only effective when the index supports it (which is the case for most ManagableIndex types). Furthermore, you must use IncrementalSearch or IncrementalSearch2.

Note: AdvancedQuery's test suite requires that ManagableIndex is installed.

Version 2.2: fixed a slicing bug reported by Santi Camps.

Version 2.1: fixed: AdvancedQuery was only usable with IncrementalSearch[2] installed. Thank's to "-r" for reporting this problem (he did not tell his real name and I do not want to expose his email).

Version 2.0: support incremental ranking.

Version 1.1: support for incremental filtering.

Version 1.0 uses IncrementalSearch2 if it is installed.

New in version 0.6: uses the new incremental search engine, if installed.

New in version 0.5: new query types MatchGlob and MatchRegexp to provide a nice interface to the new matching facility of ManagableIndex.

New in version 0.4: fixes for large Or queries (resulted in a TypeError) and wrong data type (again resulting in a TypeError); added missing security declaration for addSubquery.

New in version 0.3: lets CatalogTool.evalAdvancedQuery use an index ValidityRange (assumed to be a Managable RangeIndex) in preference of CMF's effective and expires indexes which can significantly improve query speed.

New in version 0.2: uses ManagableIndexes ReverseOrder option to speed up descending sorting.

More information

Download version 2.2 16 kb TGZ archive.

5.5 ManagableIndex

ManagableIndex can mean different things for different people. For a content manager, it brings flexible field, keyword and efficient range indexes managable via the ZMI to

For a developer, ManagableIndex provides a framework for index definition, improving on PluginIndexes. It provides for managability, automatically and intelligently handles unindexing when an object is reindexed and implements and, or and range queries (for not too complex indexes).

For Zope 2.11, ManagableIndex can be downloaded as Products.ManagableIndex from PyPI. A version for older Zopes can be downloaded below.

ManagableIndex now requires Zope 2.7+ and Python 2.3.3+.

ManagableIndex requires OFolder. You can download OFolder from this same page.

Version 1.7.3 fixes a rounding bug for indexes with a DateInteger term type. Note, that a reindexing of such indexes is required after you upgraded to the new version.

Version 1.7.1 and 1.7.2 contain some bug fixes for SimpleTextIndex.

Version 1.7 comes with SimpleTextIndex -- a very fast text index for word and phrase queries but without a query parser and without ranking.

Version 1.6.2 fixes a problem with conversion to 'DateTime' (and related conversions): trying to convert a mx.DateTime.DateTime instance resulted in an infinite loop in mx.DateTime C code. mx.DateTime and Python's datetime.datetime and datetime.date instances should now be handled correctly.

Version 1.6.1 checks for legal property names in the configuration during programmatic creation.

Version 1.6 supports configuration during programmatic creation via the extra argument for the ZCatalog's addIndex.

Version 1.5 fixes a 'PathIndex' bug concerning paths paths ending in '/'.

Version 1.4 adds 'KeywordIndex_scalable', a 'KeywordIndex' which can efficiently handle huge keyword sets for each object.

Version 1.3.1 fixes an incompatibility between 'ManagableIndex' and Zope 2.8+/Python 2.4+. This incompatibilty causes "len" to always return 1 and thereby misleading 'AdvancedQuery' (and other similar query execution frameworks) which can cause massively inefficient query execution.

Version 1.3 supports incremental filtering for most index types. See the AdvancedQuery documentation for details.

Version 1.2 defines the method indexSize newly required by Zope 2.8

Version 1.1 fixes the test suite such that it passes on Zope 2.8.x.
Warning: The test suite failure revealed an essential change in Zope 2.8.1: Until Zope 2.7, TALES expressions have been trusted when used outside of Zope; from Zope 2.8 on, TALES expression evaluation is always subject to Zope's security restrictions even when used outside of Zope. This may have strange consequences when you perform index management (e.g. mass reindexing) in an external script (run outside of Zope). In such cases, you should probably let the script login as a Zope user with sufficient priviledges.

Version 1.0 uses IncrementalSearch2 when it is installed.

Version 0.13 no longer uses 'IncrementalSearch' for 'or' unless explicitely called for because multiunion is much faster.

Version 0.12 fixes a typo in "Managable PathIndex".

Version 0.11 fixes a bug reported by Santi Camps (thank you, Santi!): "or"s with more than 5 involved sets returned all results, when IncrementalSearch has not been installed.

New in version 0.10:

New in version 0.9:

New in version 0.8:

New in version 0.7:

New in version 0.6:

New in version 0.5:

New in version 0.4:

More information

Download version 1.7.3 37 kb TGZ archive.

5.6 OFolder

A tiny layer above Folder providing ordering control.

Download 4 kb TGZ archive.

5.7 SkinnedFolder

SkinnedFolder is a tiny integration layer on top of Zope's Folder and the CMF's SkinsTool. It provides different skins for a Zope Folder.

A skin is the third element in the bones-flesh-skin view on web applications. The bones correspond to the basic infrastructure with object classes and services, the flesh is the content and the skin provides for presentation and business logic. Separating skins (presentation) from flesh (content) provides for easier personalization, greater flexibility and simpler maintenance.

The CMF SkinsTool allows to define a set of skins as a sequence of name spaces (usually folder like objects). It supports the notion of current skin. This skin is used as name space to extend the attribute resolution on the SkinsTool's parent. This parent appears to have as attributes all names defined and mapped to objects by the current skin. Switching the current skin allows for easy personalization of presentation and business logic by changing the attribute name to object map..

CMFs SkinsTool requires a request object in order to set up a skin. This makes it difficult to be used in scripts outside of Zope. The problem becomes much worse because the skin information is maintained in a volatile attribute which can be automatically deleted at transaction boundaries. Skinned Folder, therefore, provides for the property useSkinWithoutRequest. If set, it defines the skin to be used when there is no request to determine it.

This product integrates the SkinsTool with the standard Zope Folder. You must have CMFCore installed which you can get from the CMF distribution.

Almost surely, you will only be interested in this product, when you want to separate presentation from content. You may want to use this product together with FileSystemSite. It is the variant of the CMF's FSDirectoryView for use outside the CMF.

Changes in version 0.3
Changes in version 0.2

Download version 0.3 (1.4 kB TGZ archive)

5.8 ZopeProfiler

ZopeProfiler provides profiling support for the development of Zope applications. It can derive both high and low level timing statistics (Zope object call and Python function call level, respectively).

Unlike with the standard Zope profiling support, Zope runs normally in multi-threaded mode. Statistics gathering can be enabled/disabled dynamically via the ZopeProfiler object in the Control_Panel. This objects supports most features of Python's pstats.Stats timing analysis class and is much smarter than the standard Zope profiler. Should the features be insufficient, then statistics data can be saved to files and later analysed with Python's Stats class.

The product installs itself via "monkey patching". It overrides ZServer.PubCore.ZServerPublisher.publish_module. Since version 0.2, it carefully tries to coexist with other applications that override the same resource (WingIDE's Zope debugging support is one such application). There is no guarantee, however, that it succeeds.

Version 1.7.2: Work around a FIVE bug (view classes have a broken getPhysicalPath); the new PersistentState property controls whether the profiling state is persistent.

Version 1.7.1: Replaces filtering based on function name by pstats filtering on stdname. Fixes a bug for regular expressions containing < or &.

Version 1.7: Supports analysis of functions matched by a regular expression. This drastically facilitates caller and callee analysis.

Version 1.6: fixes a c_exception dispatch bug (the same that will be fixed for Python 2.4.2)

Version 1.5: made compatible with Python 2.4.1 (thanks to Yoshinori Okuji) and work around an "xmlrpclib" peculiarity (reported by Pascal Peregrina)

Version 1.4 fixes first time installation (which could cause a ConflictError during startup)

New in version 1.3: fixed a callers format bug (resulting in an exception in Python's pstats code).

New in version 1.2: made compatible with Zope 2.7.3

New in version 1.1:

Some UI improvements and Zope 2.8 compatibility.

New in version 1.0:

New in version 0.2:

Download version 1.7.2 (12.5 kB TGZ archive)

5.9 References

Sometimes, you want to refer from one place in the Zope Web site hierarchy to an object in another place. The reference should behave almost as the refered to object. I.e., you want something like a symbolic link in Unix.

Shane's Symlink product allows you to do this. However, Symlinks are too good in behaving like the refered to object. Link and original are almost not distinguishable. Therefore, it is very easy to modify the link and accidentally change the original, too. That's why I consider Symlink a bit unsafe and recommend to use it only under very restrictive circumstances.

References is very similar to Symlink. However, a Reference has its own id, title, icon and management facilities. Therefore, it can be quite easily distinguished from the target. I consider it safer. However, there are still some security weaknesses.

See the README file in the distribution, for details.

Download version 0.10 (5.5 kB TGZ archive)

5.10 Mirroring Folder

Mirroring Folder is a folder able to mirror one of its subfolders, the mirrored subfolder, into its other subfolders during traversal. Due to this mirroring the objects along the traversal subpath in the mirrored folder appear to be contained in the corresponding actually traversed object. Mirroring Folder uses acquisition to obtain this effect. Using, e.g., absolute_url will reveal the illusion.

Mirroring Folder can facilitate the management of complex products in many variants that share a considerable amount of objects. An example would be a complex mulitilingual Website, where the mirrored folder contains all language independent staff while the other subfolders contain the language specific material.

More information and download.

5.11 Wizzard (no longer supported)

Wizzard is a small tool to maintain the state of a set of related forms used in a complex task.

You would use a Wizzard when the input required for a complex task is too complicated to fit on a single form.

More information and download.

5.12 Form Dispatcher (no longer supported)

A Form Dispatcher manages the values of a single form. It provides for automatic initialization of the form fields and automatic saving of the current form values in a session object, when the form is left.

You would use a Form Dispatcher, when you need to leave a partially filled form to obtain additional information for meaningful or comfortable input for one of its fields. Another use case is to work around limitations in HTML form processing, e.g. to provide arguments for submit buttons.

More information and download.

5.13 DocFinder

Zope is a complex system. And often it is not so easy to find documentation, although things are improving. But Zope is build upon Python, a progamming language with excellent inspection facilities. DocFinder is a product to use Python's inspection facilities to derive the available documentation immediately from Zope's source. Of course, this may contain irrevant and loosely structured information, but on the other hand, it is accurate and complete.

You would use DocFinder if either

More information and download.

When you use DocFinder, you may also be interested in Stefan H. Holek's DocFinderEverywhere. It uses (so called) monkey patching, to give most Zope objects a "Doc" tab that displays the DocFinder information.

5.14 Cache Controlled SQL Methods

Z SQL methods, Zope's abstraction of SQL queries or commands, support caching of results. They use non-persistent attributes to maintain the cache. However, such attributes are thread local. Therefore, it is very difficult to flush the cache when the application knows about the invalidation of potentially cached information. Cache Controlled SQL Methods use a cache shared by all threads that can be managed by any of the threads.

You would use Cache Controlled SQL Methods when you maintain data in SQL databases that usually rarely changes (and therefore allows for long caching time) but that nevertheless needs to be kept up-to date. A prominent example is user management.

More information and download.

6 External Methods

External Methods are a light weight extension mechanism of Zope. Usually, they are used to provide utilities. Examples are conversions, access to otherwise protected operating system resources (e.g. files), filtering, interfacing with external systems. Like (Python based) products, they are not restricted by Zope's security subsystem (as they are not editable through the Web (TTW)). They live in the Extensions subfolder of the Zope distribution and are instantiated inside the Web site hierarchy through the management interface.

6.1 analyseObjects

With ExtensionClass becoming a Python new style class (i.e. with Zope 2.8), Zope's main tool to analyse memory leaks (the "Refcount"s display in "Control_Panel --> Debug information) lost most of its value as it now only reports refcounts for old style classes and acquistion wrappers. Therefore, I developped analyseObjects. It can be used as an "External Method".

analyseObjects analyses the objects known to the garbage collector. Usually, these include only objects that can contain links to other objects (such as tuples, lists, dicts, class instances, ...). Therefore, it can not detect all memory leaks (e.g. an old leak losing string objects in the session machinery could not have been detected). Nevertheless, it is a valuable aid.

Download analyseObjects and the small page template viewObjects for presentation of the results.

6.2 copyPropertySheet

Zope's support for PropertySheets is very weak. They cannot be renamed or copied. They can be exported but only imported if they are empty. The reason for these restrictions: The property values are not stored in the sheet but in the associated v_self (value self). The persistence mechanism, on with copying, renaming and export is based, cannot find them in the current implementation.

copyPropertySheet uses the specific PropertySheet interface to handle property values correctly. You can use it to copy a PropertySheet sheet into a container container and optionally give it a different id. sheet and container can be either strings or objects. Strings are resolved into objects with restrictedTraverse. container defaults to the sheets container.


6.3 emulateRedirect

emulateRedirect emulates HTTP redirects. This is more efficient than true redirects and it avoids weeknesses in the HTTP 1.1 specification.

You would use emulateRedirect when you need to redirect POST requests or want easy control for transaction handling across redirects.

When I wrote the current implementation, I was unaware that absolute_url discards context information. Because of this, emulateRedirect is difficult to use in a site that makes heavy use of acquisition. Therefore, there is now a compagnion method emulateRedirectURL. It takes (at least) two parameters self and url and redirects to url returning the resulting object. url needs to be a host relative URL, such as the URLPATHn request variables.

Warning: emulateRedirect uses undocumented internal Zope code. It was developed for Zope 2.3 and may not work for other Zope versions. On the other hand, any proficient Python programmer should easily be able to fix a potential problem.

Note: Redirection may not work in virtual hosting environments that use external URL magic, such as e.g. Apache rewrite rules. The reason, of course, is the missing magic. You can use emulateRedirectURL in such a situations provided the magic is (manually) applied to the passed in URL


6.4 Localizer

Localizer searches objects during traversal at different places. Usually, it is used as a SiteAccess AccessRule.

It was developed to facilitate the built up of a multi-lingual Web site and supports:


6.5 decodeQueryString

HTTP requests can take parameters. These parameters can be passed either as so called Query String (GET method) or in the request body (POST method). Zope's ZPublisher usually fetches the parameters whereever they are, decodes them and makes them accessible via the REQUEST object. Thereby, ZPublisher interprets name suffixes to determine converters, packagers and controlers for the decoding process (details).

Sometimes, you may want to decode the query string yourself. That's the task of decodeQueryString. It decodes the passed in query string in the same way ZPublisher would do and returns the result as a dictionary.

You may use this utility when you need to pass all parameters of one request to a later request. In this case, passing the query string (a single object) is much easier than passing each single parameter.


6.6 Catalog Support

Zope comes with an embedded search engine, the ZCatalog. Search support is build around the concepts of indexes and meta data. ZCatalog currently supports field indexes, keyword indexes and text indexes (details). Text indexes split the value associated with a document into words and index the document under each word. However, the built in index uses the raw DTML source for indexing. HTML and DTML tags as well as entity references disturb the splitting process. This External Method renders DTML objects, filters HTML tags and decodes HTML entities before ZCatalog indexing.

You would use it when either

Note: The code itself is obsolete. Modern text indexes provide direct means to filter out HTML tags. Chris Wither's stripogram module does a better job than the trivial parser in this module. The code may, however, still serve as an example how to control indexing via a script.

More information and download

7 Packages/Modules

Packages and modules are an extension facility not directly used by Zope but indirectly through either External Methods or Python based products. Almost the complete Zope framework is implemented through such packages and modules. External contributions, i.e. those not part of the Zope core, should usually live in a subpackage of Shared. My modules live in Shared/DM.

7.1 DMHistorical

A set of tools to access the historical state of a ZODB.

Such tools may be helpful to analyse why unwanted or unexpected changes happened and/or to selectively go back to a previous state.

This implementation resolves not only the main object but also all (direct or indirect) persistent references from this object at the given time. Zope's 'History' facility on the other hand can show the state of an object at a given time but would resolve persistent references with respect to the current time.

This implementation will not work for ZODB 3.2 or earlier. It has been tested with ZODB 3.4. It may (or may not) work for ZODB 3.6 or above.

Version 0.2 fixed a spelling error for the time parameter.

DMHistorical is now maintained on PyPi as dm.historical.

7.2 DMpdb

is a small package providing a "pdb" extension that understands Zope's additional traceback infos __traceback_info__ and __traceback_supplement__.

More information and download

7.3 dm.incrementalsearch

dm.incrementalsearch is the new version of IncrementalSearch2. It is now maintained on PyPI. It requires ZODB 3.8 (or above) (part of Zope 2.11). The new PyPI maintained versions Products.ManagableIndex and Products.AdvancedQuery (to be released soon) will use dm.incrementalsearch (if installed).

7.4 IncrementalSearch2

IncrementalSearch2 is a C implementation of IncrementalSearch (see below). While already IncrementalSearch could gain orders of magnitude in speed for specific "and" queries, its "or" query implementation was much slower (by a factor of about 100) than that of modern indexes using "multiunion" (it was by a factor of 10 faster than the old (still widely used) indexes using "union", e.g. FieldIndex and KeywordIndex). Moving to a C implementation allows to take advantage of the fact that document ids used in indexes are not general Python objects but known to be integers -- the same fact that makes "multiunion" so fast. IncrementalSearch2's "or" implementation is still slightly slower than "multiunion" but only about 5 per cent.

If a query contains specific "and" components, then IncrementalSearch2 can significantly reduce querying time. Our experiments with a query consisting of a large "or" component and 2 medium specific "and" components (the query had the form created >= DateTime(2004,1,1) & in_reply_to = '' & belongs_to = portal_id) against a dataset with about 30.000 objects showed gains between 20 and 35 % over traditional search (with modern indexes). With an empty ZODB cache, search time varied between 1.6 and 2.0 s (traditional search: 2.35 and 2.6s), and with a completely filled cache between 13 and 15ms (traditional search: 18 and 24 ms). The gain for the empty ZODB cache case came from the fact that incremental search touched much fewer objects: between 490 and 650 (traditional search: 740 and 830). This means also that it is a lower burden for the ZODB cache than traditional search.

IncrementalSearch2 works directly with the BTrees C level data structures. Because these structure definitions are not prepared to be used from outside the BTrees package, I had to copy the relevant parts for IncrementalSearch2. This introduces a strong dependancy: should the BTrees package change its data structures in a significant way, IncrementalSearch2 is likely to crash. A new copy of the relevant definitions will (at least) be necessary to let it work again. The current version was tested against ZODB 3.2/3.4 (the ones distributed together with Zope 2.7/2.8, respectively).

The current version was built and tested under Linux only (I do not have other development environments). Please let me know when you use IncrementalSearch2 on other platforms.

7.4.1 Installation and Generation

7.4.2 Documentation and Tests

You can find significant technical documentation about IncrementalSearch2 as source documentation of the package (i.e. in its __init__.py). If you use IncrementalSearch2 via ManagableIndex and AdvancedQuery (recommended), then this documentation is not essential. It may be relevant however for index or search engine developpers integrating IncrementalSearch2.

IncrementalSearch2 comes with an extensive set of tests that can be run with Zope's testrunner in the normal way. I recommend to run the test suite after you generated IncrementalSearch2 -- at least for non Linux platforms.

7.4.3 Version history

Version 1.3 fixes a bug on 64 bit architectures.

Version 1.2 fixes a memory leak.

Version 1.1 supports also types derived from BTrees (and not only true BTrees) as apparently some index authors prefer derived types despite some performance loss.

Version 1.0 supports incremental filtering. See the AdvancedQuery documentation for details.

Version 0.3 fixes a bug that can cause (with low probability) the loss of individual hits of a not query used as subquery of an or query. It also adds some minor optimizations.

Version 0.2 provides ZODB3.4/Zope 2.8.x compatibility.

7.4.4 Download

Download Version 1.3, tested on Linux with ZODB 3.2 (Zope 2.7.x) and ZODB 3.4 (Zope 2.8.x). Attention: will not work for 64 bit BTrees (introduced in ZODB 3.7).

7.5 IncrementalSearch

IncrementalSearch is an efficient incremental search engine supporting 'and', 'or' and 'not' queries. Unlike Zope's standard search engine, it determines hits incrementally, one at a time. It interleaves index lookup and thereby prevents the creation of large intermediate result sets and touches (almost) as few index blocks as possible. This can significantly reduce load time.

Note that IncrementalSearch can be slower than standard search if the search containes large "or" subqueries (against modern indexes using "multiunion" to implement "or" queries). This is because the Python implementation of IncrementalSearch cannot take advantage of the fact that document ids are in fact integers; "multiunion", implemented in C, can which makes it faster by a factor of about 100. The C implementation IncrementalSearch2 removes this drawback.

The table below demonstrates the effect of an incremental search with a PathIndex lookup. Inc-PI represents a Managable PathIndex with incremental search, Zope-PI a standard Zope PathIndex.

index           path              hits    time   loads       lsize       ltime

Inc-PI          I               262233   7.075    3140     1563942       5.516
Zope-PI         I               262233   7.048    3141     1562231       5.483

Inc-PI          I/I              32909  10.748    3542     1872776       6.646
Zope-PI         I/I              32909   8.160    3538     1770542       6.336

Inc-PI          I/I/B             5397   9.733    3472     1847008       7.081
Zope-PI         I/I/B             5397   8.453    3604     1812966       6.599

Inc-PI          I/I/B/A            319   2.989    1002      645967       2.068
Zope-PI         I/I/B/A            319   9.802    3693     1858580       7.890

Inc-PI          I/I/B/A/A           83   1.188     373      332506       0.832
Zope-PI         I/I/B/A/A           83   8.909    3696     1860707       7.033

Inc-PI          I/I/B/A/A/3          1   0.632     206      176983       0.444
Zope-PI         I/I/B/A/A/3          1   9.799    4000     2031824       7.737

The effect is even more drastic when the path query is anded with a quite specifc additional subquery.

index           path              hits    time   loads       lsize       ltime
Inc-PI          I                  529   1.095     372      313970       0.743
Zope-PI         I                  529   7.184    3156     1573317       5.553

Inc-PI          I/I                  0   0.621     222      157895       0.443
Zope-PI         I/I                  0   8.220    3553     1781628       6.396

Inc-PI          I/I/B                0   0.301      90       94055       0.205
Zope-PI         I/I/B                0   8.504    3619     1824052       6.635

Inc-PI          I/I/B/A              0   0.322      94       99164       0.212
Zope-PI         I/I/B/A              0   8.868    3708     1869666       6.962

Inc-PI          I/I/B/A/A            0   0.198      37       71296       0.102
Zope-PI         I/I/B/A/A            0   9.000    3711     1871793       7.078

Inc-PI          I/I/B/A/A/3          0   0.213      42       98903       0.127
Zope-PI         I/I/B/A/A/3          0   9.869    4015     2042910       7.806

Install IncrementalSearch somewhere on your PYTHONPATH, e.g. in INSTANCE_HOME/lib/python.

Version 1.0 supports incremental filtering. See the AdvancedQuery documentation for details.

Version 0.5 fixes a bug that can cause (with low probability) the loss of individual hits of a not query used as subquery of an or query.

New in version 0.4: optimizations for large 'or' searches (with hundreds and thousands of subsearches). 'or' searches run faster when the heapq C implementation from Python 2.4 (and up) is installed.

New in version 0.3: fixed bug in 'Not' handling.

New in version 0.2: fixed bug in empty tree handling.

Download Version 1.0

7.6 HCFileStorage: History controllable FileStorage (works only for (old) ZODB 3.1)

ZODB's FileStorage supports interesting information about modifications of the objects it maintains. The information includes when objects were changed, how they were changed and by whom they were changed. It is possible to revert to an old state.

However, FileStorage provides only limited control for the management of this historical information. Packing deletes all non-current object versions before the specified pack time and effectively eliminates historical information before the pack time.

HCFileStorage provides better control in that it allows the definition of history categories. A history category has a name, consists of a set of base classes and has an associated default retension time. When an object version is about to be deleted by packing, it is checked whether the object belongs to one of the defined history categories. If it does, it is only deleted when it is older than the retension time. An object belongs to the first history category such that it is an instance of one of the categories classes.

The only differences between HCFileStorage and FileStorage lie in the constructor and packing. Otherwise, HCFileStorage is identical to FileStorage.

download (6.5k TGZ archive)

7.7 Interactive debugging support

Sometimes, difficult problems require an interactive analysis of ZODB content in an environment very similar to that inside Zope. Especially, when you need to analyse the effect of traversal hooks, you need a request object. This module provides the functions getRequest() and getAuthRequest(user,password). They return a request object similar to that created by ZPublisher. You can use it to interactively traverse to an object along an URL in the same way ZPublisher would do. Note, that you can use a request object for a single traversal only. Create a new one when you need to traverse to a different object.


7.8 CVSSupport

Steve Spicklemire has implemented ZCVSFolder, a product that provides restricted CVS support for Zope objects. CVS operations are performed via an intermediate working directory inside the file system. ZCVSFolder uses Zope's import/export facility to synchronize the Zope objects with the corresponding files in the file system and normal CVS commands to synchronize working directory and repository.

While ZCVSFolder is already very helpful, there are currently two severe restrictions:

  1. The XML export format contains lots of ids consecutively assigned. They provide the necessary references in case that the same subobject is referenced twice. However, these ids make merging a nightmare, as already small changes of an object lead to wide spread changes in these artificial ids.
  2. The recursive features of ZCVSFolder are enabled only for standard Zope Folders. Other types of ObjectManagers are not supported. Even for Zope Folders, properties assigned to the folders are not CVS maintained. The reason comes from the fact, that Zope Folders are mapped to file system directories and they do not have place to store the properties.

The module below provides a significant improvement towards the first restriction. First, an id is only generated, when the corresponding object is indeed referenced again. Second, references to scalar types are replaced by their value to restrict sharing to non-trivial cases. While this may lead to drastical memory consumption, it is very rare that large elementary objects are shared in typical Zope environments. These two measures ensure that there are few, if any, ids in the XML representation of Zope objects.

The module defines exportXML as a replacement for OFS.XMLExportImport.exportXML. Be careful, the code is not yet thouroughly tested.


I plan to work also towards a solution for the second restriction.

7.9 Shared Resource

Zope is a multi-threaded application with built in transaction control. As always with multi-threaded applications, synchronization becomes an essential part of the application as otherwise non-deterministic chaos threatens. Synchronization can be very difficult and error prone and can make application development a nightmare. Therefore, the Zope framework takes over responsibility for the synchronization and combines it with transaction control.

In Zope, each thread has its own independent view of the Web site; it uses private copies of its objects. When a request modifies an object, the executing thread modifies its own local copy of the object only. This way, the application code does not need to worry about synchronization because the modifications do not affect other threads. When the request terminates, the corresponding transaction is either aborted or commited. If aborted, the modified copies are simply invalidated (fresh copies will be loaded, when they are accessed). If commited, the changes are stored into the corresponding storages. At this stage, Zope checks for conflicts with other threads, tries object specific conflict resolution, if necessary, and raises a ConflictException, if it fails. It would then abort the transaction (the commit is a standard two phase commit) and usually try to reexecute the request.

As a consequence, it is not easy to implement non-persistent resources shared between threads in Zope applications. Such resources usually require explicit synchronization. SharedResource is a module that implements such shared resources. In fact, it provides a framework for the creation and use of such resources.

You would use it when you need non-persistent resources that all threads should share and see. Registries of any kind are the most prominent example. If your registries can be implemented through Python dictionaries and other elementary Python datatypes and your registry algorithm is somewhat resilient against asynchrony, you might not need to use SharedResource. Otherwise, you will need such a facility.

More information and download.

8 Patches

Patches are (usually) small instructions to modify source files in order to fix problems or provide enhancements. All patches below are unified or context diffs and can be applied with the GNU patch utility.

8.1 ZSyncer via Pickle (obsolete)

Note that current ZSyncer versions contain this patch already. Thus, this patch is obsolete.

Usually ZSyncer uses XML-RPC to exchange commands and data with a remote ZSyncer. Unfortunately, XML-RPC is very sensitive with respect to non-ASCII characters. This patch lets ZSyncer use Python's serialization, called Pickling. This is much more robust than XML-RPC communication.

Warning: Pickle is not safe against malicious use; it is easy to create pickles that can do almost everything when they are unpickled (including execution of arbitrary (operating system level) commands as the user running Zope). Therefore, the patch below requires the View management screens permission on the target ZSyncer instance before unpickling is performed. By very careful about whom you allow to use this service.

Note: it might be necessary to remove the DOS/Windows style line endings in the patch file before you can apply it.


8.2 Post authentication hook

In some case, you want to take special actions once the user has been determined at the end of traversal. You may, e.g., want to deny Manager role to any non HTTPS request.

This patch adds support for a post authentication hook. It adds code to the validated_hook, called when the user is successfully authenticated. The code tries to acquire an object post_authentication_hook from request.PARENTS[0]. If this is successful, it calls this object with the arguments request and user. Usually, this call will raise Unauthorized when it has objections against the request.

As it is very easy (making a mistake in your post authentication hook code) to shut yourself out of the management interface, the hook can be disabled with the environment variable SUPPRESS_PAH.


8.3 No more ReadConflictErrors (for (old) ZODB 3.1; not safe!) (obsolete)

The ZODB usually guarantees that a transaction does not read inconsistent data by issuing a ReadConflictError when it tries to read an object from the ZODB that has been modified since transaction start. Zope usually restarts the transaction in this case.

When transactions are long or writes quite frequent, ReadConflictError can become a plague. Storages with UndoSupport can do better by providing the state when the transaction startet. This is the Snapshot transaction isolation level: a transaction sees the state as it was when the transaction started, independent from other tasks' activities. When several transactions try to concurrently change an object, all transactions but the one that commits first, receive a ConflictError, as with Zope's normal transaction policy.

This patch against Zope 2.6.1 provides the Snapshot isolation level as default. Up to now, it has been tested with FileStorage and ClientStorage. Usually, it needs a storage that supports undo, at least in a restricted form. Sessions in a TemporaryStorage will work, too, as they do not use the modified code. When a storage without undo support deliveres an object state that is too young, a ReadConflictError is raised. Jeremy thinks he will be able to provide support for Berkeley storage, as well (although is does not support undo).

This patch was formerly known as ReadCommitted patch. Version 0.2 fixes several bugs (bug in transaction handling, bad call to 'ReadConflictError') and makes the patch simpler (it now modifies only 3 files rather than 6).

Note, that there are reports that the patch does not work unmodified with ZODB 3.2 because "setLocalTransaction" changes transaction handling significantly.

Download version 0.2.

Attention: One of the ZODB (3.1.2) invalidation tests fails on high speed servers, when this patch is applied. This reveals that the patch is not able to garantee consistent views of the data and can lead to inconsistencies.
This can only be fixed for ZODB 3.2, not for ZODB 3.1.

8.4 TALES expressions for DCWorkflow worklists (educational only)

Note that this patch can no longer be applied to modern DCWorkflow versions because incompatible but less flexible changes have been made for DCWorkflow worklists.

The standard DCWorkflow does not support TALES expressions in the definition of worklists. This patch interprets the search terms in worklist definitions as TALES expressions if they contain a :. It also provides a search function, in order not to duplicate the code in the worklist definition and the actual search for the work items overview.


8.5 VirtualHostMonster - RemoteAddr patch (obsolete)

Note that modern Apache versions support the ProxyVia directive which lets Apache pass remote address information in an X_FORWARDED_FOR header. Zope can use this information (at least since Zope 2.7.x) provided the Apache host is listed as trusted-proxy. Therefore, this patch is not obsolete.

VirtualHostMonster, or short VHM, is usually used together with Apache's rewrite rules and proxy mode to implement virtual hosting. Unfortunately, when used in this way, Zope's REMOTE_ADDR is the (usually uninteresting) IP address of the Apache server and not the one of the browser/its proxy. There is an Apache patch that passes this IP address via an VIA header, but this is against the HTTP 1.1 spec.

This VHM patch enables VHM to take the remote address from the URL. If the URL segments following the VirtualHostBase/protocol/host are VirtualHostRemoteAddr/remote_addr, then VHM sets Zope's REMOTE_ADDR to remote_addr. This feature is used together with Apache rewrite rules of the form: RewriteRule pattern http://localhost:8080/VirtualHostBase/proto/host/VirtualHostRemoteAddr/%{REMOTE_ADDR}/ something [P].


8.6 ZSyncer - CMF patch (obsolete)

Note that modern ZSyncer versions already contain this patch.

ZSyncer is a utility (from Andy) which facilitates the synchronization of several production sites with a central development site. It works by automating Zope's export/import mechanism through XML-RPC.

This patch to ZSyncer 0.4.4 allows ZSyncer to synchronize CMF sites more easily. For CMF sites, you usually want to synchronize the portal configuration but not the portal content from development to production. Therefore, you can not synchronize the complete portal. On the other hand, the portal tools are not registered with Zope's meta type registry. Therefore, ZSyncer's standard configuration to control syncable meta types fails for CMF portal tools which contain most of the configuration. The tiny patch adds an additional textarea to the configuration form which allows to add the meta types of unregistered object classes in free form (as lines, to be precise).

Note: The patch is integrated in ZSyncer 0.5.


8.7 TemporaryStorage - Version patch (obsolete)

Note that modern Zope versions already contain this patch.

Zope 2.5 comes with builtin session support. An undoable storage is quite inefficient to store large amounts of rapidly changing data such as typical session data. Therefore, a RAM based storage, the so called TemporaryStorage, was integrated as container mainly for session data. Unfortunately, this storage does not implement versions. There is a high chance that using versions leads to a TemporaryStorage does not support versions exception, even if you are unaware to use sessions. At least, I made this experience using CMF.

The patch below lets TemporaryStorage ignore versions, when the environment variable ZOPE_TEMPSTORE_IGNORE_VERSIONS is set (to a non-empty value). This seams adequate for session data, as sessions seem orthogonal to versions. Be aware, however, that this may hit you. As some objects in the version are usually different from the corresponding objects outside the version, session information may be compatible with just one and not both of them.


8.8 Formulator 1.0 patch - don't write help pages on startup

Formalator 1.0 installs its help pages on startup even if they are already installed. This causes unnecessary writes to the ZODB and makes it impossible to use the ZODB in read only mode. The patch fixes this problems. The help pages are only installed, if something changed. The patch requires Python 2.1 or above.


8.9 ZPT - parse restarts after incomplete entity/char reference (obsolete)

Note that modern ZPT versions use Python's HTML parser (which hopefully does not have this bug).

When ZPT's (version 1.4) HTML parser encounters an incomplete entity or character reference, it restarts parsing from the beginning. Usually, it will raise an HTMLParseError when it encounters the incomplete reference for the second time. If, however, a macro definition preceeds the incomplete reference, then an "TAL.TALDefs.METALError: duplicate macro definition" exception is raised which is very difficult to understand.

This patch fixes the HTML parser to not restart from the beginning but continue at the current position.


8.10 ZPT - show traceback (obsolete)

Modern ZPT versions provide this functionality without the patch.

The great ZPT (Zope Page Template) product is currently in its beta phase. It already works quite well but sometimes it hides usefull error information for template problems due to errors in ZPT itself. The patch (against ZPT 1-3-1/2) adds traceback information to the problem report at the top of faulty templates.

You might want to apply this patch, if you are willing to fix ZPT yourself in the case that error information is hidden by a secondary problem in ZPT itself.


8.11 ZPT - delay cooking (obsolete)

Note that modern ZPT versions delay cooking without the patch.

Before a ZPT can be used, it must be cooked. Cooking essentially means parsing the template and transforming it into an internal object.

The current ZPT implementation performs cooking immediately when the ZPT is loaded from the ZODB. This may become a problem, as cooking may take a long time. For example, if you have a folder with lots of ZPT's and the folder get displays in the left frame of the ZMI for the first time, then all contained ZPT's are loaded. It can take minutes before the tree display is finally build.

The following patch delays cooking until the ZPT is actually used for something. This speeds up initial operations considerably.


8.12 ObjectManager(Zope) - FTP support

Zope's FTP support tries hard not to expose acquired objects. The current implementation checks that the acquisition chain of an accessed object corresponds to its physical path. This is much stronger than an acquired check. It prevents FTP support for any object below one that uses acquisition in a non-trivial way. A prominent victim is Mirroring Folder and its content.

The patch removes the acquisition check altogether. I expect, it might have been introduced to avoid either of the following dangers:

Anyway, I am interested to learn about any problem that may result from the installation of this FTP patch.


8.13 ZPublisher(Zope) - Image Button Support; ":last" suffix

ZPublisher recognizes ":method" (now aliased to ":action") parameter name suffixes to call the specified method. This is a very nice feature for forms with several submit buttons as it allows to give each button its own action and avoids tedious dispatching in the single form action. Unfortunately, it breaks down for HTML 3.2 image buttons because, for them, the user agent adds .x and .y suffixes to the form control name and ZPublisher no longer recognizes the suffix.

The patch lets ZPublisher recognize :method suffixes (and its alias :action) even for image buttons. For image buttons, ZPublisher defines a records variable imgclickpos with the two fields x and y.
The patch also defines an additional suffix :last that lets ZPublisher only retain the last value of a sequence of name-value pairs for the given name. This is handy for batching to avoid the start parameter to accumulate in the request.

You may want to use this patch if you use image buttons and do not want to use a JavaScript solution as an alternative to the :method feature.

Download for Zope 2.5.

Download for Zope 2.3/4.

9 Misc

This section contains contributions that can not easily be classified into one of the above classes.

9.1 Pydoc support

Python 2.1 comes with a great documentation facility pydoc. pydoc uses Python's excellent inspection mechanisms to derive all available documentation from Python sources. It presents them (among other) via an HTTP server, e.g. via the intranet. Therefore, it is very easy to find out about the available packages, modules, classes and their methods and get accurate information about their signature and documentation strings. If more information is needed, the source code itself can be listed.

This support is now provided by dm.zdoc, available via PyPI.

Dieter Maurer
Last modified: Sat Jan 16 10:09:43 CET 2010