ReadConflictError
s (for (old) ZODB 3.1;
not safe!) (obsolete)Zope is an open source, flexible, platform independent web application server framework especially useful to build dynamic websites. You find more information about Zope via Zope's home page.
This page contains my contributions to the Zope software fund. They are either completely free, or covered by an open source or open content license. Note however, that I consider whether I will expressly forbid the use of my software by institutions of the European Union (Councel of Ministers, Commission, Parliament, ...) and member governments supporting software patents. This is in protest against a recent decision of the Councel to make software patentable. This decision ignores a Parliament vote that pleaded to keep logic (algorithms, business rules) un-patentable. Of course, only new versions will be affected by such a change -- should it come.
Newer versions of my software are now maintained on the Python package index ("PyPI"). In general, they can be used for Zope 2.11+. This site maintains versions that can still be used for earlier Zope versions. Any reworked version will move over to "PyPI".
If you live in the European Union, please consider signing the petition to forbid software patents in this area. Software patents threaten free/open source software development by individuals and small companies because they introduce the danger to accidentally infringe one of the trivial patents with the risk of immense legal costs and dammage claims. Only large multi-national companies can afford the costs of a legal department that continuously checks against the huge number of patents and defends against (potentially) unjustified patent infringement claims.
This section is devoted to what is already available of my Zope book Building dynamic WebSites with Zope -- Concepts, Components, Programming. These are 3 chapters of about 8. The available chapters, too, are likely to change considerably before final publication, because Zope is a fast developing system: what is true today may no longer hold tomorrow. The chapters currently describe Zope 2.3 with some additions for Zope 2.4. I would say, they are in a beta, i.e. already quite usable state.
Products are one of Zope's extension facilities. Usually, they provide new object classes that can be used to build the Website.
DigestAuth
provides HTTP Digest Authentication (RFC
2617) for Zope. More precisely, it provides a Digest
Authentication Crumbler
which works similar to Shane's
CookieCrumbler
: it turns HTTP digest authentication
info provided by the client into HTTP basic authentication info and
thereby allows to use user folders expecting basic authentication
with the much more secure digest authentication.
Digest authentication (at least the implemented crumbler) requires that the password is available in cleartext. It must not be encrypted.
I plan to provide also a PAS (PluggableAuthService) module supporting digest authentication in a future version.
Download version 0.1 7 kb TGZ archive.
CompiledExecutables
defines a set of executable
objects that are precompiled and the compilation stored either in
the ZODB or in the file system. It currently defines
CompiledPageTemplate
,
FSCompiledPageTemplate
and
FSCompiledPythonScript
(PythonScript
is
already compiled; although in a somewhat stupid manner (in
__setstate__
rather than when used)).
Storing the compilation result avoids expensive recompilation. This can drastically speed up access to the first pages after startup -- especially for large CMF based sites.
The product does not register anything. It is up to you to
register the filesystem based objects with
Products.CMFCore.DirectoryView.registerFileExtension
or the ZODB based objects by calling their initialize
function.
The product works by making instance methods and code objects picklable. This might appear as a security risk. However, unpickling untrusted pickles is always a security risk (because the pickler decides what is picklable and not the unpickler).
Version 0.5 customized a 'CompiledFSPageTemplate' into a caching policy manager aware version of PageTemplate.
Version 0.4 is now tested against a standard Zope 2.8.1 (without our local modifications).
Version 0.3 fixes an import bug in
FSCompiledPythonScript
and defines a
_compile
method for CompiledPageTemplate
to make it more consistent with scripts.
Version 0.2 handles a read only storage correctly, improves a
work around for a PythonScript
bug and fixes an
initialization bug in
FSCompiledPythonScript.getBindingAssignments
.
Download version 0.5 4 kb TGZ archive.
TrustedExecutables
is a set of executable objects
unrestricted by Zope's security. Currently, it contains
TrustedPageTemplateFile
,
TrustedFSPageTemplate
and
TrustedFSPythonScript
.
TrustedFSPageTemplate
and
TrustedFSPythonScript
are registered with the filename
extensions xpt
and xpy
, respectively.
As Zope's security checks are expensive, avoiding them can drastically speed things up. On the other hand, these objects must make their own security checks at places where access control is required.
Use with extreme care!
ATTENTION: It is not unlikely that this product breaks between Zope releases, as it uses undocumented implementation details. It is probably not very difficult to fix things again but you will need programming skills to do so.
TrustedExecutables
0.x
versions' trusted PageTemplate variants. From version 1.0 on,
TrustedExecutables
is adapted to the Zope3 page
template implementation (and no longer supports the old Zope2
implementation). The new versions are maintained as
Products.TrustedExecutables
on PyPI and can be
installed via easy_install Products.TrustedExecutables
(provided that "setuptools/easy_install" is installed). The easy
installed versions require Zope 2.11 or above. Repackaging (moving
TrustedExecutables
into your
"$INSTANCE_HOME/Products") might allow you to use it with Zope
2.10, but I have not tested this.
Products.TrustedExecutables
on
PyPI.
Version 0.5.2: apparently, I have problems to get the 0.5 version right. This is another fix for the pdb debuggability.
Version 0.5.1 provides the features promissed for 0.5 (they went accidentally into the variant maintained for my employee).
Version 0.5 makes 'TrustedFSPythonScript' debuggable via 'pdb/zdb' (with correct line numbers) using code from Chris Wither's 'zdb'.
Version 0.4 copies utilities (such as 'test', 'reorder') from
TemplateDict
(the builtin namespace for untrusted
code) to Python's __builtin__
module (used by trusted
code). This way, trusted PageTemplates and trusted DTMLDocuments
can use the same variables as untrusted ones. Note, that this might
have some ramifications for other file based code, too.
Version 0.3 became necessary to adapt to the cosmetic changes introduced in Zope 2.7.1
New in version 0.2:
TrustedFS*
objects via
customization template. Customization itself will raise a
TypeError
, though (as there are not ZODB based trusted
objects).print
>>target
.Download version 0.5.2 (for
Zope 2.7 upto 2.9 (including) 6 kb TGZ archive. From version
1.0 on, you can download TrustedExecutables
as
Products.TrustedExecutables
from PyPI. These versions
support Zope 2.11 (or above).
AdvancedQuery
is a Zope product aimed to overcome
several limitations of ZCatalog
's native search
function.
Like ZCatalog
's search, it supports elementary
index searches. While ZCatalog
can combine such
elementary searches only by "and", AdvancedQuery
allows them to be combined arbitrary with &
(and),
|
(or) and ~
(not).
While ZCatalog
supports an efficient sorting via an
index on one level, AdvancedQuery
supports sorting via
any level of (field) indexes. Moreover, it sorts the result
incrementally -- only as far as you access your result. This can
drastically speed up the time required for sorting. It uses
Python's generators for this (and thus requires Python 2.2 or
better).
From version 1.1 on, AdvancedQuery
supports
incremental filtering. See the
documentation. Note that incremental filtering is only
effective when the index supports it (which is the case for most
ManagableIndex
types). Furthermore, you must use
IncrementalSearch
or
IncrementalSearch2
.
Note: AdvancedQuery
's test suite
requires that ManagableIndex
is installed.
Version 2.2: fixed a slicing bug reported by Santi Camps.
Version 2.1: fixed: AdvancedQuery
was only usable
with IncrementalSearch[2]
installed. Thank's to "-r"
for reporting this problem (he did not tell his real name and I do
not want to expose his email).
Version 2.0: support incremental ranking.
Version 1.1: support for incremental filtering.
Version 1.0 uses IncrementalSearch2
if it is
installed.
New in version 0.6: uses the new incremental search engine, if installed.
New in version 0.5: new query types MatchGlob
and
MatchRegexp
to provide a nice interface to the new
matching facility of ManagableIndex
.
New in version 0.4: fixes for large Or
queries
(resulted in a TypeError
) and wrong data type (again
resulting in a TypeError
); added missing security
declaration for addSubquery
.
New in version 0.3: lets
CatalogTool.evalAdvancedQuery
use an index
ValidityRange
(assumed to be a Managable RangeIndex
) in
preference of CMF's effective
and expires
indexes which can significantly improve query speed.
New in version 0.2: uses ManagableIndex
es
ReverseOrder
option to speed up descending
sorting.
Download version 2.2 16 kb TGZ archive.
ManagableIndex
can mean different things for
different people. For a content manager, it brings flexible field,
keyword and efficient range indexes managable via the ZMI to
portal_type
to the Subject
index transforming it into a sequence (as required by a keyword
index).For a developer, ManagableIndex
provides a
framework for index definition, improving on
PluginIndexes
. It provides for managability,
automatically and intelligently handles unindexing when an object
is reindexed and implements and, or and
range queries (for not too complex indexes).
For Zope 2.11, ManagableIndex
can be downloaded as
Products.ManagableIndex
from PyPI. A version for older
Zopes can be downloaded below.
ManagableIndex
now requires Zope 2.7+ and Python
2.3.3+.
ManagableIndex
requires OFolder
. You
can download OFolder
from this same page.
Version 1.7.3 fixes a rounding bug for indexes with a
DateInteger
term type. Note, that a reindexing of such
indexes is required after you upgraded to the new version.
Version 1.7.1 and 1.7.2 contain some bug fixes for
SimpleTextIndex
.
Version 1.7 comes with SimpleTextIndex
-- a very
fast text index for word and phrase queries but without a query
parser and without ranking.
Version 1.6.2 fixes a problem with conversion to 'DateTime' (and
related conversions): trying to convert a
mx.DateTime.DateTime
instance resulted in an infinite
loop in mx.DateTime
C code. mx.DateTime
and Python's datetime.datetime
and
datetime.date
instances should now be handled
correctly.
Version 1.6.1 checks for legal property names in the configuration during programmatic creation.
Version 1.6 supports configuration during programmatic creation
via the extra
argument for the ZCatalog
's
addIndex
.
Version 1.5 fixes a 'PathIndex' bug concerning paths paths ending in '/'.
Version 1.4 adds 'KeywordIndex_scalable', a 'KeywordIndex' which can efficiently handle huge keyword sets for each object.
Version 1.3.1 fixes an incompatibility between 'ManagableIndex' and Zope 2.8+/Python 2.4+. This incompatibilty causes "len" to always return 1 and thereby misleading 'AdvancedQuery' (and other similar query execution frameworks) which can cause massively inefficient query execution.
Version 1.3 supports incremental filtering for most index types. See the AdvancedQuery documentation for details.
Version 1.2 defines the method indexSize
newly
required by Zope 2.8
Version 1.1 fixes the test suite such that it passes on Zope
2.8.x.
Warning: The test suite failure
revealed an essential change in Zope 2.8.1: Until Zope 2.7, TALES
expressions have been trusted when used outside of Zope; from Zope
2.8 on, TALES expression evaluation is always subject to Zope's
security restrictions even when used outside of Zope. This may have
strange consequences when you perform index management (e.g. mass
reindexing) in an external script (run outside of Zope). In such
cases, you should probably let the script login as a Zope user with
sufficient priviledges.
Version 1.0 uses IncrementalSearch2
when it is
installed.
Version 0.13 no longer uses 'IncrementalSearch' for 'or' unless
explicitely called for because multiunion
is
much faster.
Version 0.12 fixes a typo in "Managable PathIndex".
Version 0.11 fixes a bug reported by Santi Camps (thank you,
Santi!): "or"s with more than 5 involved sets returned all results,
when IncrementalSearch
has not been installed.
New in version 0.10:
New in version 0.9:
RangeIndex
).WordIndex
for minimal transaction and load
sizes.New in version 0.8:
Cannot pickle
code object
exceptions could result from this bug.Orginisation 'high-then-low'
. When true, the
high
index is primary and the low
one
subordinate. Check this for normally used date range indexes.New in version 0.7:
New in version 0.6:
New in version 0.5:
RangeIndex
now efficiently supports improper
ranges, i.e. ranges with at least one unrestricted boundary,RangeIndex
can now be used as a (partial) plugin
replacement for CMF's effective
and
expires
indexes.New in version 0.4:
DateTime
and Date
indexes,RangeIndex
es,AdvancedQuery
s descending
sorting.Download version 1.7.3 37 kb TGZ archive.
A tiny layer above Folder
providing ordering
control.
Download 4 kb TGZ archive.
SkinnedFolder is a tiny integration layer on top of Zope's
Folder
and the CMF's SkinsTool
. It
provides different skins for a Zope Folder
.
A skin is the third element in the bones-flesh-skin view on web applications. The bones correspond to the basic infrastructure with object classes and services, the flesh is the content and the skin provides for presentation and business logic. Separating skins (presentation) from flesh (content) provides for easier personalization, greater flexibility and simpler maintenance.
The CMF SkinsTool
allows to define a set of skins
as a sequence of name spaces (usually folder like objects). It
supports the notion of current skin. This skin is used as
name space to extend the attribute resolution on the
SkinsTool
's parent. This parent appears to have as
attributes all names defined and mapped to objects by the current
skin. Switching the current skin allows for easy personalization of
presentation and business logic by changing the attribute name to
object map..
CMF
s SkinsTool
requires a request
object in order to set up a skin. This makes it difficult to be
used in scripts outside of Zope. The problem becomes much worse
because the skin information is maintained in a volatile attribute
which can be automatically deleted at transaction boundaries.
Skinned Folder
, therefore, provides for the property
useSkinWithoutRequest
. If set, it defines the skin to
be used when there is no request to determine it.
This product integrates the SkinsTool
with the
standard Zope Folder. You must have CMFCore
installed
which you can get from the CMF distribution.
Almost surely, you will only be interested in this product, when
you want to separate presentation from content. You may want to use
this product together with
FileSystemSite. It is the variant of the CMF's
FSDirectoryView
for use outside the CMF.
useSkinWithoutRequest
addedmanage_options
fixed.Download version 0.3 (1.4 kB TGZ archive)
ZopeProfiler provides profiling support for the development of Zope applications. It can derive both high and low level timing statistics (Zope object call and Python function call level, respectively).
Unlike with the standard Zope profiling support, Zope runs
normally in multi-threaded mode. Statistics gathering can be
enabled/disabled dynamically via the ZopeProfiler
object in the Control_Panel
. This objects supports
most features of Python's pstats.Stats
timing analysis
class and is much smarter than the standard Zope profiler. Should
the features be insufficient, then statistics data can be saved to
files and later analysed with Python's Stats
class.
The product installs itself via "monkey patching". It overrides
ZServer.PubCore.ZServerPublisher.publish_module
. Since
version 0.2, it carefully tries to coexist with other applications
that override the same resource (WingIDE's Zope debugging support
is one such application). There is no guarantee, however, that it
succeeds.
Version 1.7.2: Work around a FIVE bug (view classes have a
broken getPhysicalPath
); the new
PersistentState
property controls whether the
profiling state is persistent.
Version 1.7.1: Replaces filtering based on function name by
pstats
filtering on stdname. Fixes a bug for regular
expressions containing < or &.
Version 1.7: Supports analysis of functions matched by a regular expression. This drastically facilitates caller and callee analysis.
Version 1.6: fixes a c_exception
dispatch bug (the
same that will be fixed for Python 2.4.2)
Version 1.5: made compatible with Python 2.4.1 (thanks to Yoshinori Okuji) and work around an "xmlrpclib" peculiarity (reported by Pascal Peregrina)
Version 1.4 fixes first time installation (which could cause a
ConflictError
during startup)
New in version 1.3: fixed a callers
format bug
(resulting in an exception in Python's pstats
code).
New in version 1.2: made compatible with Zope 2.7.3
New in version 1.1:
Some UI improvements and Zope 2.8 compatibility.
New in version 1.0:
Documentation
and
showDocumentation
(to Info
and
showInfo
, respectively) to more easily coexist with
Stefan's DocFinderEverywhere
.New in version 0.2:
Download version 1.7.2 (12.5 kB TGZ archive)
Sometimes, you want to refer from one place in the Zope Web site hierarchy to an object in another place. The reference should behave almost as the refered to object. I.e., you want something like a symbolic link in Unix.
Shane's Symlink
product allows you to do this.
However, Symlink
s are too good in behaving like the
refered to object. Link and original are almost not
distinguishable. Therefore, it is very easy to modify the link and
accidentally change the original, too. That's why I consider
Symlink
a bit unsafe and recommend to use it only
under very restrictive circumstances.
References
is very similar to Symlink
.
However, a Reference
has its own id, title, icon and
management facilities. Therefore, it can be quite easily
distinguished from the target. I consider it safer. However, there
are still some security weaknesses.
See the README
file in the distribution, for
details.
Download version 0.10 (5.5 kB TGZ archive)
Mirroring Folder
is a folder able to mirror one of
its subfolders, the mirrored subfolder, into its other subfolders
during traversal. Due to this mirroring the objects along the
traversal subpath in the mirrored folder appear to be contained in
the corresponding actually traversed object. Mirroring
Folder
uses acquisition to obtain this effect. Using, e.g.,
absolute_url
will reveal the illusion.
Mirroring Folder
can facilitate the management of
complex products in many variants that share a considerable amount
of objects. An example would be a complex mulitilingual Website,
where the mirrored folder contains all language independent staff
while the other subfolders contain the language specific
material.
More information and download.
Wizzard is a small tool to maintain the state of a set of related forms used in a complex task.
You would use a Wizzard
when the input required for
a complex task is too complicated to fit on a single form.
More information and download.
A Form Dispatcher
manages the values of a single
form. It provides for automatic initialization of the form fields
and automatic saving of the current form values in a session
object, when the form is left.
You would use a Form Dispatcher
, when you need to
leave a partially filled form to obtain additional information for
meaningful or comfortable input for one of its fields. Another use
case is to work around limitations in HTML form processing, e.g. to
provide arguments for submit
buttons.
More information and download.
Zope is a complex system. And often it is not so easy to find
documentation, although things are improving. But Zope is build
upon Python, a progamming language with excellent inspection
facilities. DocFinder
is a product to use Python's
inspection facilities to derive the available documentation
immediately from Zope's source. Of course, this may contain
irrevant and loosely structured information, but on the other hand,
it is accurate and complete.
You would use DocFinder
if either
More information and download.
When you use DocFinder
, you may also be interested
in Stefan H. Holek's DocFinderEverywhere.
It uses (so called) monkey patching, to give most Zope objects a
"Doc" tab that displays the DocFinder
information.
Z SQL methods, Zope's abstraction of SQL queries or commands,
support caching of results. They use non-persistent attributes to
maintain the cache. However, such attributes are thread local.
Therefore, it is very difficult to flush the cache when the
application knows about the invalidation of potentially cached
information. Cache Controlled SQL Methods
use a cache
shared by all threads that can be managed by any of the
threads.
You would use Cache Controlled SQL Methods
when you
maintain data in SQL databases that usually rarely changes (and
therefore allows for long caching time) but that nevertheless needs
to be kept up-to date. A prominent example is user management.
More information and download.
External Methods
are a light weight extension
mechanism of Zope. Usually, they are used to provide utilities.
Examples are conversions, access to otherwise protected operating
system resources (e.g. files), filtering, interfacing with external
systems. Like (Python based) products, they are not restricted by
Zope's security subsystem (as they are not editable through the Web
(TTW)). They live in the Extensions
subfolder of the
Zope distribution and are instantiated inside the Web site
hierarchy through the management interface.
analyseObjects
With ExtensionClass
becoming a Python new style
class (i.e. with Zope 2.8), Zope's main tool to analyse memory
leaks (the "Refcount"s display in "Control_Panel --> Debug
information) lost most of its value as it now only reports
refcounts for old style classes and acquistion wrappers. Therefore,
I developped analyseObjects
. It can be used as an
"External Method".
analyseObjects
analyses the objects known to the
garbage collector. Usually, these include only objects that can
contain links to other objects (such as tuples, lists, dicts, class
instances, ...). Therefore, it can not detect all memory leaks
(e.g. an old leak losing string objects in the session machinery
could not have been detected). Nevertheless, it is a valuable
aid.
Download analyseObjects and the small page template viewObjects for presentation of the results.
copyPropertySheet
Zope's support for PropertySheets is very weak. They cannot be
renamed or copied. They can be exported but only imported if they
are empty. The reason for these restrictions: The property values
are not stored in the sheet but in the associated
v_self
(value self). The persistence mechanism, on
with copying, renaming and export is based, cannot find them in the
current implementation.
copyPropertySheet
uses the specific PropertySheet
interface to handle property values correctly. You can use it to
copy a PropertySheet sheet into a container
container and optionally give it a different
id. sheet and container can be
either strings or objects. Strings are resolved into objects with
restrictedTraverse
. container defaults to
the sheets container.
emulateRedirect
emulateRedirect
emulates HTTP redirects. This is
more efficient than true redirects and it avoids weeknesses in the
HTTP 1.1 specification.
You would use emulateRedirect
when you need to
redirect POST
requests or want easy control for
transaction handling across redirects.
When I wrote the current implementation, I was unaware that
absolute_url
discards context information. Because of
this, emulateRedirect
is difficult to use in a site
that makes heavy use of acquisition. Therefore, there is now a
compagnion method emulateRedirectURL
. It takes (at
least) two parameters self and url and
redirects to url returning the resulting object.
url needs to be a host relative URL, such as the
URLPATHn
request variables.
Warning:
emulateRedirect
uses undocumented internal Zope code.
It was developed for Zope 2.3 and may not work for other Zope
versions. On the other hand, any proficient Python programmer
should easily be able to fix a potential problem.
Note: Redirection may not work in
virtual hosting environments that use external URL magic, such as
e.g. Apache rewrite rules. The reason, of course, is the missing
magic. You can use emulateRedirectURL
in such a
situations provided the magic is (manually) applied to the passed
in URL
Localizer
searches objects during traversal at
different places. Usually, it is used as a SiteAccess
AccessRule
.
It was developed to facilitate the built up of a multi-lingual Web site and supports:
HTTP requests can take parameters. These parameters can be
passed either as so called Query String
(GET
method) or in the request body (POST
method). Zope's ZPublisher usually fetches the parameters whereever
they are, decodes them and makes them accessible via the
REQUEST
object. Thereby, ZPublisher interprets name
suffixes to determine converters, packagers and controlers for the
decoding process (details).
Sometimes, you may want to decode the query string yourself.
That's the task of decodeQueryString
. It decodes the
passed in query string in the same way ZPublisher would do and
returns the result as a dictionary.
You may use this utility when you need to pass all parameters of one request to a later request. In this case, passing the query string (a single object) is much easier than passing each single parameter.
Zope comes with an embedded search engine, the
ZCatalog
. Search support is build around the concepts
of indexes and meta data. ZCatalog currently supports field
indexes, keyword indexes and text indexes (details). Text indexes split the value
associated with a document into words and index the document under
each word. However, the built in index uses the raw DTML source for
indexing. HTML and DTML tags as well as entity references disturb
the splitting process. This External Method renders DTML objects,
filters HTML tags and decodes HTML entities before ZCatalog
indexing.
You would use it when either
Note: The code itself is obsolete. Modern text indexes
provide direct means to filter out HTML tags. Chris Wither's
stripogram
module does a better job than the trivial
parser in this module. The code may, however, still serve as an
example how to control indexing via a script.
Packages and modules are an extension facility not directly used by Zope but indirectly through either External Methods or Python based products. Almost the complete Zope framework is implemented through such packages and modules. External contributions, i.e. those not part of the Zope core, should usually live in a subpackage of Shared. My modules live in Shared/DM.
A set of tools to access the historical state of a ZODB.
Such tools may be helpful to analyse why unwanted or unexpected changes happened and/or to selectively go back to a previous state.
This implementation resolves not only the main object but also all (direct or indirect) persistent references from this object at the given time. Zope's 'History' facility on the other hand can show the state of an object at a given time but would resolve persistent references with respect to the current time.
This implementation will not work for ZODB 3.2 or earlier. It has been tested with ZODB 3.4. It may (or may not) work for ZODB 3.6 or above.
Version 0.2 fixed a spelling error for the time parameter.
DMHistorical
is now maintained on PyPi as
dm.historical
.
is a small package providing a "pdb" extension that understands
Zope's additional traceback infos __traceback_info__
and __traceback_supplement__
.
dm.incrementalsearch
is the new version of IncrementalSearch2
. It
is now maintained on PyPI. It requires ZODB 3.8 (or above) (part of
Zope 2.11). The new PyPI maintained versions
Products.ManagableIndex
and
Products.AdvancedQuery
(to be released soon) will use
dm.incrementalsearch
(if installed).
IncrementalSearch2
is a C implementation of IncrementalSearch
(see
below). While already IncrementalSearch
could gain
orders of magnitude in speed for specific "and" queries, its "or"
query implementation was much slower (by a factor of about 100)
than that of modern indexes using "multiunion" (it was by a factor
of 10 faster than the old (still widely used) indexes using
"union", e.g. FieldIndex
and
KeywordIndex
). Moving to a C implementation allows to
take advantage of the fact that document ids used in indexes are
not general Python objects but known to be integers -- the same
fact that makes "multiunion" so fast.
IncrementalSearch2
's "or" implementation is still
slightly slower than "multiunion" but only about 5 per cent.
If a query contains specific "and" components, then
IncrementalSearch2
can significantly reduce querying
time. Our experiments with a query consisting of a large "or"
component and 2 medium specific "and" components (the query had the
form created >= DateTime(2004,1,1) & in_reply_to = ''
& belongs_to = portal_id
) against a dataset
with about 30.000 objects showed gains between 20 and 35 % over
traditional search (with modern indexes). With an empty ZODB cache,
search time varied between 1.6 and 2.0 s (traditional search: 2.35
and 2.6s), and with a completely filled cache between 13 and 15ms
(traditional search: 18 and 24 ms). The gain for the empty ZODB
cache case came from the fact that incremental search touched much
fewer objects: between 490 and 650 (traditional search: 740 and
830). This means also that it is a lower burden for the ZODB cache
than traditional search.
IncrementalSearch2
works directly with the
BTrees
C level data structures. Because these
structure definitions are not prepared to be used from outside the
BTrees
package, I had to copy the relevant parts for
IncrementalSearch2
. This introduces a strong
dependancy: should the BTrees
package change its data
structures in a significant way, IncrementalSearch2
is
likely to crash. A new copy of the relevant definitions will (at
least) be necessary to let it work again. The current version was
tested against ZODB 3.2/3.4 (the ones distributed together with
Zope 2.7/2.8, respectively).
The current version was built and tested under Linux only (I do
not have other development environments). Please let me know when
you use IncrementalSearch2
on other platforms.
PYTHONPATH
, e.g.
INSTANCE_HOME/lib/python
.ZOPE_HOME
to the path
to your Zope installation.ExtensionClass.h
,
cPersistence.h
and friends).IncrementalSearch2
folder with python setup.py build_ext -i
.ISearch.so
(or
ISearch.pyd
on Windows) containing the C
extension.You can find significant technical documentation about
IncrementalSearch2
as source documentation of the
package (i.e. in its __init__.py
). If you use
IncrementalSearch2
via ManagableIndex
and
AdvancedQuery
(recommended), then this documentation
is not essential. It may be relevant however for index or search
engine developpers integrating IncrementalSearch2
.
IncrementalSearch2
comes with an extensive set of
tests that can be run with Zope's testrunner in the normal way. I
recommend to run the test suite after you generated
IncrementalSearch2
-- at least for non Linux
platforms.
Version 1.3 fixes a bug on 64 bit architectures.
Version 1.2 fixes a memory leak.
Version 1.1 supports also types derived from BTrees (and not only true BTrees) as apparently some index authors prefer derived types despite some performance loss.
Version 1.0 supports incremental filtering. See the AdvancedQuery documentation for details.
Version 0.3 fixes a bug that can cause (with low probability) the loss of individual hits of a not query used as subquery of an or query. It also adds some minor optimizations.
Version 0.2 provides ZODB3.4/Zope 2.8.x compatibility.
Download Version 1.3, tested on Linux with ZODB 3.2 (Zope 2.7.x) and ZODB 3.4 (Zope 2.8.x). Attention: will not work for 64 bit BTrees (introduced in ZODB 3.7).
IncrementalSearch
is an efficient incremental
search engine supporting 'and', 'or' and 'not' queries. Unlike
Zope's standard search engine, it determines hits incrementally,
one at a time. It interleaves index lookup and thereby prevents the
creation of large intermediate result sets and touches (almost) as
few index blocks as possible. This can significantly reduce load
time.
Note that IncrementalSearch
can be slower than
standard search if the search containes large "or" subqueries
(against modern indexes using "multiunion" to implement "or"
queries). This is because the Python implementation of
IncrementalSearch
cannot take advantage of the fact
that document ids are in fact integers; "multiunion", implemented
in C, can which makes it faster by a factor of about 100. The C
implementation IncrementalSearch2
removes this drawback.
The table below demonstrates the effect of an incremental search
with a PathIndex lookup. Inc-PI
represents a Managable PathIndex with incremental
search, Zope-PI
a standard Zope PathIndex.
index path hits time loads lsize ltime Inc-PI I 262233 7.075 3140 1563942 5.516 Zope-PI I 262233 7.048 3141 1562231 5.483 Inc-PI I/I 32909 10.748 3542 1872776 6.646 Zope-PI I/I 32909 8.160 3538 1770542 6.336 Inc-PI I/I/B 5397 9.733 3472 1847008 7.081 Zope-PI I/I/B 5397 8.453 3604 1812966 6.599 Inc-PI I/I/B/A 319 2.989 1002 645967 2.068 Zope-PI I/I/B/A 319 9.802 3693 1858580 7.890 Inc-PI I/I/B/A/A 83 1.188 373 332506 0.832 Zope-PI I/I/B/A/A 83 8.909 3696 1860707 7.033 Inc-PI I/I/B/A/A/3 1 0.632 206 176983 0.444 Zope-PI I/I/B/A/A/3 1 9.799 4000 2031824 7.737
The effect is even more drastic when the path query is anded with a quite specifc additional subquery.
index path hits time loads lsize ltime Inc-PI I 529 1.095 372 313970 0.743 Zope-PI I 529 7.184 3156 1573317 5.553 Inc-PI I/I 0 0.621 222 157895 0.443 Zope-PI I/I 0 8.220 3553 1781628 6.396 Inc-PI I/I/B 0 0.301 90 94055 0.205 Zope-PI I/I/B 0 8.504 3619 1824052 6.635 Inc-PI I/I/B/A 0 0.322 94 99164 0.212 Zope-PI I/I/B/A 0 8.868 3708 1869666 6.962 Inc-PI I/I/B/A/A 0 0.198 37 71296 0.102 Zope-PI I/I/B/A/A 0 9.000 3711 1871793 7.078 Inc-PI I/I/B/A/A/3 0 0.213 42 98903 0.127 Zope-PI I/I/B/A/A/3 0 9.869 4015 2042910 7.806
Install IncrementalSearch
somewhere on your
PYTHONPATH
, e.g. in
INSTANCE_HOME/lib/python
.
Version 1.0 supports incremental filtering. See the AdvancedQuery documentation for details.
Version 0.5 fixes a bug that can cause (with low probability) the loss of individual hits of a not query used as subquery of an or query.
New in version 0.4: optimizations for large 'or' searches (with
hundreds and thousands of subsearches). 'or' searches run faster
when the heapq
C implementation from Python 2.4 (and
up) is installed.
New in version 0.3: fixed bug in 'Not' handling.
New in version 0.2: fixed bug in empty tree handling.
ZODB's FileStorage supports interesting information about modifications of the objects it maintains. The information includes when objects were changed, how they were changed and by whom they were changed. It is possible to revert to an old state.
However, FileStorage provides only limited control for the management of this historical information. Packing deletes all non-current object versions before the specified pack time and effectively eliminates historical information before the pack time.
HCFileStorage provides better control in that it allows the definition of history categories. A history category has a name, consists of a set of base classes and has an associated default retension time. When an object version is about to be deleted by packing, it is checked whether the object belongs to one of the defined history categories. If it does, it is only deleted when it is older than the retension time. An object belongs to the first history category such that it is an instance of one of the categories classes.
The only differences between HCFileStorage and FileStorage lie in the constructor and packing. Otherwise, HCFileStorage is identical to FileStorage.
download (6.5k TGZ archive)
Sometimes, difficult problems require an interactive analysis of
ZODB content in an environment very similar to that inside Zope.
Especially, when you need to analyse the effect of traversal hooks,
you need a request object. This module provides the functions
getRequest()
and
getAuthRequest(user,password)
. They return a request
object similar to that created by ZPublisher
. You can
use it to interactively traverse to an object along an URL in the
same way ZPublisher
would do. Note, that you can use a
request object for a single traversal only. Create a new one when
you need to traverse to a different object.
Steve Spicklemire has implemented ZCVSFolder, a product that provides restricted CVS support for Zope objects. CVS operations are performed via an intermediate working directory inside the file system. ZCVSFolder uses Zope's import/export facility to synchronize the Zope objects with the corresponding files in the file system and normal CVS commands to synchronize working directory and repository.
While ZCVSFolder is already very helpful, there are currently two severe restrictions:
The module below provides a significant improvement towards the first restriction. First, an id is only generated, when the corresponding object is indeed referenced again. Second, references to scalar types are replaced by their value to restrict sharing to non-trivial cases. While this may lead to drastical memory consumption, it is very rare that large elementary objects are shared in typical Zope environments. These two measures ensure that there are few, if any, ids in the XML representation of Zope objects.
The module defines exportXML
as a replacement for
OFS.XMLExportImport.exportXML
. Be careful, the code is not yet
thouroughly tested.
I plan to work also towards a solution for the second restriction.
Zope is a multi-threaded application with built in transaction control. As always with multi-threaded applications, synchronization becomes an essential part of the application as otherwise non-deterministic chaos threatens. Synchronization can be very difficult and error prone and can make application development a nightmare. Therefore, the Zope framework takes over responsibility for the synchronization and combines it with transaction control.
In Zope, each thread has its own independent view of the Web
site; it uses private copies of its objects. When a request
modifies an object, the executing thread modifies its own local
copy of the object only. This way, the application code does not
need to worry about synchronization because the modifications do
not affect other threads. When the request terminates, the
corresponding transaction is either aborted or commited. If
aborted, the modified copies are simply invalidated (fresh copies
will be loaded, when they are accessed). If commited, the changes
are stored into the corresponding storages. At this stage, Zope
checks for conflicts with other threads, tries object specific
conflict resolution, if necessary, and raises a
ConflictException
, if it fails. It would then abort
the transaction (the commit is a standard two phase commit) and
usually try to reexecute the request.
As a consequence, it is not easy to implement non-persistent
resources shared between threads in Zope applications. Such
resources usually require explicit synchronization.
SharedResource
is a module that implements such shared
resources. In fact, it provides a framework for the creation and
use of such resources.
You would use it when you need non-persistent resources that all
threads should share and see. Registries of any kind are the most
prominent example. If your registries can be implemented through
Python dictionaries and other elementary Python datatypes and your
registry algorithm is somewhat resilient against asynchrony, you
might not need to use SharedResource
. Otherwise, you
will need such a facility.
More information and download.
Patches are (usually) small instructions to modify source files in order to fix problems or provide enhancements. All patches below are unified or context diffs and can be applied with the GNU patch utility.
Note that current ZSyncer versions contain this patch already. Thus, this patch is obsolete.
Usually ZSyncer uses XML-RPC to exchange commands and data with a remote ZSyncer. Unfortunately, XML-RPC is very sensitive with respect to non-ASCII characters. This patch lets ZSyncer use Python's serialization, called Pickling. This is much more robust than XML-RPC communication.
Warning: Pickle
is not
safe against malicious use; it is easy to create pickles that can
do almost everything when they are unpickled (including execution
of arbitrary (operating system level) commands as the user running
Zope). Therefore, the patch below requires the View
management screens
permission on the target ZSyncer instance
before unpickling is performed. By very careful
about whom you allow to use this service.
Note: it might be necessary to remove the DOS/Windows style line endings in the patch file before you can apply it.
In some case, you want to take special actions once the user has
been determined at the end of traversal. You may, e.g., want to
deny Manager
role to any non HTTPS request.
This patch adds support for a post authentication hook.
It adds code to the validated_hook
, called when the
user is successfully authenticated. The code tries to acquire an
object post_authentication_hook
from
request.PARENTS[0]
. If this is successful, it calls
this object with the arguments request
and
user
. Usually, this call will raise
Unauthorized
when it has objections against the
request.
As it is very easy (making a mistake in your post authentication
hook code) to shut yourself out of the management interface, the
hook can be disabled with the environment variable
SUPPRESS_PAH
.
ReadConflictError
s (for (old) ZODB 3.1;
not safe!) (obsolete)The ZODB usually guarantees that a transaction does not read
inconsistent data by issuing a ReadConflictError
when
it tries to read an object from the ZODB that has been modified
since transaction start. Zope usually restarts the transaction in
this case.
When transactions are long or writes quite frequent,
ReadConflictError
can become a plague. Storages with
UndoSupport can do better by providing the state when the
transaction startet. This is the Snapshot
transaction
isolation level: a transaction sees the state as it was when the
transaction started, independent from other tasks' activities. When
several transactions try to concurrently change an object, all
transactions but the one that commits first, receive a
ConflictError
, as with Zope's normal transaction
policy.
This patch against Zope 2.6.1 provides the Snapshot
isolation level as default. Up to now, it has been tested with
FileStorage
and ClientStorage
. Usually,
it needs a storage that supports undo, at least in a restricted
form. Sessions in a TemporaryStorage
will work, too,
as they do not use the modified code. When a storage without undo
support deliveres an object state that is too young, a
ReadConflictError
is raised. Jeremy thinks he will be
able to provide support for Berkeley storage, as well (although is
does not support undo).
This patch was formerly known as ReadCommitted
patch. Version 0.2 fixes several bugs (bug in transaction handling,
bad call to 'ReadConflictError') and makes the patch simpler (it
now modifies only 3 files rather than 6).
Note, that there are reports that the patch does not work unmodified with ZODB 3.2 because "setLocalTransaction" changes transaction handling significantly.
Attention: One of the ZODB (3.1.2)
invalidation tests fails on high speed servers, when this patch is
applied. This reveals that the patch is not able to garantee
consistent views of the data and can lead to inconsistencies.
This can only be fixed for ZODB 3.2, not for ZODB 3.1.
Note that this patch can no longer be applied to modern DCWorkflow versions because incompatible but less flexible changes have been made for DCWorkflow worklists.
The standard DCWorkflow does not support TALES expressions in
the definition of worklists. This patch interprets the search terms
in worklist definitions as TALES expressions if they contain a
:. It also provides a search
function, in
order not to duplicate the code in the worklist definition and the
actual search for the work items overview.
Note that modern Apache versions support the
ProxyVia
directive which lets Apache pass remote
address information in an X_FORWARDED_FOR
header. Zope
can use this information (at least since Zope 2.7.x) provided the
Apache host is listed as trusted-proxy
. Therefore,
this patch is not obsolete.
VirtualHostMonster
, or short VHM
, is
usually used together with Apache's rewrite rules and proxy mode to
implement virtual hosting. Unfortunately, when used in this way,
Zope's REMOTE_ADDR
is the (usually uninteresting) IP
address of the Apache server and not the one of the browser/its
proxy. There is an Apache patch that passes this IP address via an
VIA
header, but this is against the HTTP 1.1 spec.
This VHM
patch enables VHM
to take the
remote address from the URL. If the URL segments following the
VirtualHostBase/protocol/host
are VirtualHostRemoteAddr/remote_addr
, then
VHM sets Zope's REMOTE_ADDR
to remote_addr.
This feature is used together with Apache rewrite rules of the
form: RewriteRule pattern
http://localhost:8080/VirtualHostBase/proto/host/VirtualHostRemoteAddr/%{REMOTE_ADDR}/
something [P]
.
Note that modern ZSyncer
versions already contain
this patch.
ZSyncer
is a utility (from Andy) which facilitates
the synchronization of several production sites with a central
development site. It works by automating Zope's export/import
mechanism through XML-RPC.
This patch to ZSyncer 0.4.4 allows ZSyncer to synchronize CMF sites more easily. For CMF sites, you usually want to synchronize the portal configuration but not the portal content from development to production. Therefore, you can not synchronize the complete portal. On the other hand, the portal tools are not registered with Zope's meta type registry. Therefore, ZSyncer's standard configuration to control syncable meta types fails for CMF portal tools which contain most of the configuration. The tiny patch adds an additional textarea to the configuration form which allows to add the meta types of unregistered object classes in free form (as lines, to be precise).
Note: The patch is integrated in ZSyncer 0.5.
Note that modern Zope versions already contain this patch.
Zope 2.5 comes with builtin session support. An undoable storage is quite inefficient to store large amounts of rapidly changing data such as typical session data. Therefore, a RAM based storage, the so called TemporaryStorage, was integrated as container mainly for session data. Unfortunately, this storage does not implement versions. There is a high chance that using versions leads to a TemporaryStorage does not support versions exception, even if you are unaware to use sessions. At least, I made this experience using CMF.
The patch below lets TemporaryStorage ignore
versions, when the environment variable
ZOPE_TEMPSTORE_IGNORE_VERSIONS
is set (to a non-empty
value). This seams adequate for session data, as sessions seem
orthogonal to versions. Be aware, however, that this may hit you.
As some objects in the version are usually different from the
corresponding objects outside the version, session information may
be compatible with just one and not both of them.
Formalator 1.0 installs its help pages on startup even if they are already installed. This causes unnecessary writes to the ZODB and makes it impossible to use the ZODB in read only mode. The patch fixes this problems. The help pages are only installed, if something changed. The patch requires Python 2.1 or above.
Note that modern ZPT versions use Python's HTML parser (which hopefully does not have this bug).
When ZPT's (version 1.4) HTML parser encounters an incomplete entity or character reference, it restarts parsing from the beginning. Usually, it will raise an HTMLParseError when it encounters the incomplete reference for the second time. If, however, a macro definition preceeds the incomplete reference, then an "TAL.TALDefs.METALError: duplicate macro definition" exception is raised which is very difficult to understand.
This patch fixes the HTML parser to not restart from the beginning but continue at the current position.
Modern ZPT versions provide this functionality without the patch.
The great ZPT (Zope Page Template) product is currently in its beta phase. It already works quite well but sometimes it hides usefull error information for template problems due to errors in ZPT itself. The patch (against ZPT 1-3-1/2) adds traceback information to the problem report at the top of faulty templates.
You might want to apply this patch, if you are willing to fix ZPT yourself in the case that error information is hidden by a secondary problem in ZPT itself.
Note that modern ZPT versions delay cooking without the patch.
Before a ZPT can be used, it must be cooked. Cooking essentially means parsing the template and transforming it into an internal object.
The current ZPT implementation performs cooking immediately when the ZPT is loaded from the ZODB. This may become a problem, as cooking may take a long time. For example, if you have a folder with lots of ZPT's and the folder get displays in the left frame of the ZMI for the first time, then all contained ZPT's are loaded. It can take minutes before the tree display is finally build.
The following patch delays cooking until the ZPT is actually used for something. This speeds up initial operations considerably.
Zope's FTP support tries hard not to expose acquired objects. The current implementation checks that the acquisition chain of an accessed object corresponds to its physical path. This is much stronger than an acquired check. It prevents FTP support for any object below one that uses acquisition in a non-trivial way. A prominent victim is Mirroring Folder and its content.
The patch removes the acquisition check altogether. I expect, it might have been introduced to avoid either of the following dangers:
Anyway, I am interested to learn about any problem that may result from the installation of this FTP patch.
ZPublisher recognizes ":method" (now aliased to ":action")
parameter name suffixes to call the specified method. This is a
very nice feature for forms with several submit buttons as it
allows to give each button its own action and avoids tedious
dispatching in the single form action. Unfortunately, it breaks
down for HTML 3.2 image buttons because, for them, the user agent
adds .x
and .y
suffixes to the form
control name and ZPublisher no longer recognizes the suffix.
The patch lets ZPublisher recognize :method
suffixes (and its alias :action
) even for image
buttons. For image buttons, ZPublisher defines a records variable
imgclickpos
with the two fields x
and
y
.
The patch also defines an additional suffix :last
that
lets ZPublisher only retain the last value of a sequence of
name-value pairs for the given name. This is handy for batching to
avoid the start parameter to accumulate in the request.
You may want to use this patch if you use image buttons and do
not want to use a JavaScript solution as an alternative to the
:method
feature.
This section contains contributions that can not easily be classified into one of the above classes.
Python 2.1 comes with a great documentation facility
pydoc
. pydoc
uses Python's excellent
inspection mechanisms to derive all available documentation from
Python sources. It presents them (among other) via an HTTP server,
e.g. via the intranet. Therefore, it is very easy to find out about
the available packages, modules, classes and their methods and get
accurate information about their signature and documentation
strings. If more information is needed, the source code itself can
be listed.
This support is now provided by dm.zdoc
, available
via PyPI.