1.1. OO preliminaries

In general, an object is an entity encapsulating data and behavior. The behavior is usually defined by a set of methods, functions^[5] that operate on the objects data. This data is usually implemented as a set of members. Both methods and members, together, are called attributes^[6].

In many object oriented systems, objects belong to a class (or type). The class defines common attributes of its objects. Often, a class is used as a blueprint for object creation. In this case, the object is seen as an instance of the class or class instance. A class definition consists of definitions for its attributes.

Many object oriented systems do not require to list all these definitions explicitly but provide for an inclusion mechanism. It allows to include or inherit all definitions from other classes but to override definitions that do not fit. This class definition facility is called inheritance, the inherited class is called a base class of the defined class and the defined class is called a class derived from the base class. A derived class can be viewed as an extension of any of its base classes. Its objects are at the same time objects of its base classes. In this way, inheritance implements an is a relationship.

Some systems, such as e.g. Java[Java], restrict inheritance to a single class, i.e. a class can have at most one base class. We speak of single inheritance. Other systems, such as Python[Python], support multiple inheritance: a class can have arbitrary many base classes. Multiple inheritance is a powerful feature. It allows to compose classes out of class components. Each component, itself a class, implements a, usually small, special feature set. It is reused whenever a class requires this feature set. Such small components are often called mixin classes as they are designed to be mixed in rather than be used stand alone. Zope, build upon Python, makes heavy use of multiple inheritance and mixin classes. Persistence^[7] and acquisition are for example implemented through mixin classes.

You can use my DocFinder product to analyze the composition of Zope's classes, determine what methods are available, how they need to be called, by what permissions they are protected and what source code documentation is available for them.

The interface concept is related to the class concept. While a class is an implementation device, it is a set of definitions, an interface is a specification device, it is a set of declarations (usually accompanied by additional descriptive text). Currently, the interface concept is heavily influenced by Java's interfaces. A Java interface is essentially a set of method signatures^[8]. Java supports an inclusion mechanism for interface definitions, similar to inheritance for classes, and calls it interface extension. A Java interface can extend several interfaces, similar to multiple inheritance.

While Python, unlike Java, does not have an interface language construct, Zope makes heavy use of interfaces. There are several kinds of interfaces: for the content manager, the DTML programmer or the application programmer. Within each kind, an object usually implements several interfaces, e.g. a security interface, a property manager interface and an object manager interface. The interface descriptions can be accessed via Zope's integrated help system.

1.2. Interface ObjectManagerItem

The interface ObjectManagerItem is implemented by objects that can be managed by an ObjectManager. ObjectManager, in turn, is an interface implemented by the container like objects. Consequently, the objects we are looking at in this section implement ObjectManagerItem. It defines the common aspects of the Zope site building objects.

The interface details can be found in Zope's integrated online help system under Zope Help->API Reference->ObjectManagerItem. We will not go into details here, as the online documentation should be more current than this book can be. We will, however, note the principles.

All Zope site building objects have an id, a title and a meta_type.

The id is used to identify the object inside the containing object manager. It must therefore be unique within this context. The set of characters allowed in ids is severely restricted: they must be valid URL characters, i.e. ASCII letters or digits, _, ~, ,, . or space^[9]. Moreover, they must not begin with _. All names beginning with _ are considered private in Zope, inaccessible through the web. Ids must not end in two _. This, probably, is another privacy convention, but I am not sure. Finally, some reserved names are forbidden, such as REQUEST.

For most object classes, id is an attribute. For others, however, it is a method. At many places, this distinction is not relevant; in some contexts, however, it is. This can seriously confuse programmers. Therefore, the method getId should be used to access the id.

While id is of major importance, its companion title has much less weight. It is a string attribute which may be empty. If non empty, it is used in the management interface and the default standard_html_header. The latter uses it to define the HTML page title. There are several methods combining the retrieval of title and id in different ways.

meta_type is a string attribute describing the object's type. You can use this type, when you use Zope's Find support. Find, a tab in the management interface, lets you find objects meeting given criteria. One of the most important criteria is the object's meta type.

I already mentioned that ZPublisher maps URLs onto web site objects. It is therefore natural, that Zope provides a method that returns an object's URL, the URL that is mapped to this object. This method is absolute_url.

Sometimes, you might want to take over part of the ZPublisher's normal work: locate an object based on an URL path fragment. The method restrictedTraverse does this. It takes a path, either a string with path components separated by / or directly a sequence, and locates the object that is reached from the current object along path. You will probably use this feature to access objects that are not directly accessible through a name. In Zope, all ancestors of an object and their immediate children are directly accessible by the object. For access to other objects, restrictedTraverse is often the most natural way, at least in complex cases. The restricted in the method name indicates, that Zope performs security checks along the path to ensure that you (or your Zope users) are doing only things you are entitled to do.

Unless your site uses virtual hosts (or the virtual hosting tools for a different purpose) restrictedTraverse is almost the inverse of absolute_url. Otherwise, your site can have different URLs for the same object, for example a different URL for each virtual host. In this case, absolute_url returns the correct virtual URL with respect to the current request and its corresponding virtual host. restrictedTraverse, on the other hand, uses the physical path as defined by the web site hierarchy. The true inverse of absolute_url is the method resolve_url of the request object. restrictedTraverse, on the other hand, is the inverse of getPhysicalPath, another ObjectManagerItem method. If no virtual hosting tools are used, the URLs are directly related to the physical paths in the Web site hierarchy.

The method manage_workspace returns the HTML page used to manage the object. This page is part of Zope's management interface through which objects can be created, edited and deleted, access controlled, found, changes analyzed and discarded. An object's manage_workspace generates a page with the management options which are appropriate for the object and which the current visitor has the necessary permissions for.

1.3. Interface ObjectManager

ObjectManager is the interface implemented by Zope's container like objects.

A two level selection is used to create new objects inside an object manager. This two level approach tries to minimize the danger of name clashes between different independently developed products. At the first level, the product is selected. The second level then selects one of the object constructors from this product. With this approach, problems only arise, if different products have the same name; there is no problem, if constructors in different products happen to have the same name. ObjectManager provides the member manage_addProduct to implement the product selection. manage_addProduct is a mapping object indexed by product names. manage_addProduct[product name] returns a factory dispatcher the members of which are the product's constructors. Zope's site building objects belong to the product OFSP^[10].

om.manage_addProduct['OFSP'].manage_addFolder('F')

ObjectManager contains a set of methods to access information about its contained objects: objectIds, objectValues, objectItems. objectIds returns the ids of the contained objects, objectValues the objects themselves and objectItems a sequence of (id,object) pairs. All these methods allow an optional argument, either a single meta type or a sequence of meta types, to restrict the objects retrieved to such with one of the given meta types. An object manager exposes its contained items as attributes. A single item can simply be accessed through its id. Thus, in the context of the example above, om.F is the newly created folder. The contained items can also be accessed through subscription: om['F'] for the example. The subscription will raise a KeyError when the object manager does not contain an object with the given id. The attribute access, on the other hand, would in this case look also for properties and acquired attributes. You will use subscription, when you want to make sure, you only get a contained object.

An ObjectManager has management methods to delete, export and import items.

1.4. Interface PropertyManager

A property is a usually small piece of (meta) information associated with an object. The "meta" indicates that this information is in addition to the primary object "content"; it provides additional aspects about the object. "title", "author", "version", "publishing state" would be property examples.

A PropertyManager manages a set of properties. Properties have a type and a value. Modification or deletion of a property can be restricted. The type is from the list float, int, long, string, lines, text, date, tokens, selection, multiple selection. Properties can be predefined by an object or defined dynamically either through the web or programmatically.

Properties are either predefined or are created with the method manage_addProperty. Property ids must not start with _ and must not contain space, in addition to the restrictions spelled out for ObjectManager ids.

PropertyManager has methods propertyIds, propertyValues, propertyItems with similar effects as the corresponding ObjectManager methods. As for object managers, properties can be directly accessed through their ids. There are some additional methods to check for property existence, access its type and fetch its value or a default.

Properties are changed with the methods manage_editProperties or manage_changeProperties. manage_editProperties changes all properties. If no new value is given, the property is reset. This may be necessary for direct through the web editing, as HTTP requests do not have parameters for unchecked check boxes and empty multiple selections. manage_changeProperties, on the other hand, changes only properties explicitly referenced in its parameters.

1.5. Folder

The Folder class is Zope's most prominent ObjectManager. You use a Zope folder in exactly the same way, you use folders in a file system: to group objects in a clear hierarchy.

A folder is also a PropertyManager. This means, you can assign properties to a folder. As folders are essentially passive objects without interesting application specific behavior, these properties are usually destined for use by the folders content or help the content manager to remember essential things about the folder.

manage_addFolder is the folder constructor, used in programs to create a new folder.

1.6. File

You can place normal files into your Zope site. You might do this, if you want to make them available for download or just publish them on your site. A file can also be used as a text component for other objects that include its content at an appropriate place.

A file is created by the constructor manage_addFile. It can be created programmatically, through the management interface or via FTP. The file content is usually uploaded. The management interface does not provide a way to edit file content through the web.

Files are property managers. One special property is the so called content_type. A Content-Type is a MIME (Multi purpose Internet Mail Extension) concept, adopted by HTTP (HyperText Transfer Protocol), to indicate the content type of a message such as an HTTP response. MIME content types consist of a type and an optional subtype, separated by /. Optional, type specific arguments, separated by ;, can provide further information. Types include text, image, application, audio, video. The most essential subtypes of text are plain, html and perhaps soon xml. Its most prominent argument is charset, it specifies the character set used by the text. Subtypes for image include png, jpeg and gif. octed-stream, pdf, postscript, msword belong to the many subtypes of application. Usually, you need not to worry about a file's content type as Zope guesses it when a file is uploaded based on the filename extension. However, sometimes, the guess is wrong. In this case, you would edit the content_type property manually.

Zope's file objects are maintained in the ZODB, its object database. Usually, they are not accessible by the programs you normally use to edit or work on these files. So, what, if you want to edit them? There are several approaches. You can have a copy of the file in your file system (or download the file via your browser), modify the copy and then upload it again. Or you can use the fact, that Zope exposes the web site through FTP, the File Transfer Protocol. Depending on your platform, there may be products (which have nothing to do with Zope) that allow you to map FTP hierarchies into your file system and use the FTP accessible objects in the same way as local file system objects. In this case, you can use your normal applications for editing^[11]. You can also use the Zope products LocalFS or ExternalFile to access normal operating system files from Zope. These are not file objects as described in this section, but they behave mostly as if they were.

1.7. Image

Image objects are used to manage images in your web site. It inherits from File and almost everything, we heard about files hold for images, too. There are some small differences.

When a file is included in a document template, this is called rendering, then its content is integrated into the generated page. This is appropriate for files containing small amounts of text. It would be in most cases inappropriate for image data, as HTML does not handle images this way. Instead, HTML defines the img tag for this purpose. For this reason, the rendering of an image object generates an img tag such that an HTML browser will show the image when it displays the generated page. There is also a tag method that provides for more control over the img tag generation.

1.8. DTML Method, DTML Document

DTML objects are Zope's main page templates^[12]. DTML stands for Document Template Markup Language. The language consists of a set of page composition commands that use the tag and entity syntax of SGML, familiar from its use in HTML. The commands can be embedded into plain text that is included unchanged into the generated page. We will learn the details later in the DTML section.

The main content of a DTML object is a document template. When the object is rendered the DTML commands embedded in the template are executed. Each command generates some text which replaces the original command in the template. After all commands have been executed, the resulting text is returned as rendering result. According to context, it can be used for example as response page to a Web request or as a fragment in a bigger generation or processing process.

The document template commands are executed in a name lookup context, a so called namespace. It determines how the object names in the commands are resolved to objects. The two different DTML object types, DTML Method and DTML Document use different ways to construct the namespace for their document template execution. A DTML method essentially uses the calling context unchanged while a DTML document places itself on top of the context. As a result, the document template of a DTML method does not see the method but directly looks at the calling context. The template therefore behaves like a method of the calling context. This fact gives the class its name. The template of a DTML document, on the other hand, works primarily with this document object. Only, if this object (and sometimes its acquisition ancestors) cannot resolve a name, the calling context becomes relevant. There is another minor difference between DTML document and method: the former is a property manager; the latter is not, as the properties could not be accessed anyway.

DTML Document inherits from DTML Method. Both classes have the same methods to access and change the document template and to execute it. The constructors are manage_addDTMLDocument and manage_addDTMLMethod.

1.9. PageTemplate

PageTemplate, short ZPT, is the new composition engine currently developed for Zope. It addresses two issues of DTML templates: easy integration with high end design tools, such as DreamWeaver, GoLife or HomeSite, and reducing the amount of magic involved in processing the templates.

PageTemplate uses the XML namespace extension mechanism to add composition commands to complete HTML or XML documents. It uses two XML namespaces http://xml.zope.org/namespaces/tal with default prefix tal and http://xml.zope.org/namespaces/metal with default prefix metal to tag its composition commands.

TAL stands for Template Attribute Language. The name indicates that most composition commands are attached as attributes to source tags. There are commands for definitions, conditional rendering, looping as well as element, element content and attribute replacement. Most commands essentially maintain the structure of the source. This way, the source presentation in an HTML/XML viewing or editing tool can be very similar to the presentation of the final rendered page. Thereby, designers are supported in their work. On the other hand, the templates are complete HTML/XML documents. Tools are not confused by missing structural parts that are only generated at rendering time from constructs the tools do not understand.

To provide for modularity and separation of common parts, PageTemplate uses a macro expansion mechanism to compose templates from components. The macro facility uses the namespace METAL, Macro Expansion Template Attribute Language. It defines attributes for macro definition and use as well as for slot definition and the provision of slot content when the macro is used. Slots, therefore, are the macro parameters. A macro is simply a named element in a template. The same applies to a macro slot. Macro and slot definitions as well as macro and slot use are embedded in the templates.

More information about Page Templates in the readable specifications for TAL, TALES and METAL and the excellent tutorial. Background information can be found in the ZPT Wiki.

Page Templates are constructed with the constructor manage_addPageTemplate.

1.10. Python Script

A Python script object encapsulates a sequence of Python commands. The script can be called to execute the Python commands. If they return a value, this is the result used in the calling context. Depending on context, it can be used as response to a Web request, as fragment for a larger generation or processing process or it can be discarded.

The script has various ways to access different contexts. It can define parameters and bindings. Parameters allow the caller to communicate with the called Python script by providing values for the parameters. The parameters are accessed inside the script through their names and are bound to the provided values during the execution. The bindings allow to bind other kinds of context information to names and make them thereby accessible inside the script. The available contexts include:

Python scripts can use a Python subset that is considered safe in an environment where the scripts can be edited through the inherently unsafe Web medium. While almost all Python commands are available, the set of accessible built-in functions and modules from the Python library are severely restricted. There is an unrestricted variant, too: XXXPythonScript. Because an update of these scripts through the Web poses a serious security risk, they can be changed only if Zope has been started with a special environment variable.

Python scripts are created with the constructor manage_addPythonScript.

1.11. External Method

An External Method executes Python code, like a Python Script does. However, this code is not stored in the ZODB but in a Python source file in the file system. As the file system is considered much safer than the Web editable ZODB, external methods are not restricted by Zope's security system. They can access the complete Python library, all Zope packages and modules and any attribute in objects, even private ones. This makes them ideally suited to implement utilities, e.g. integration with remote systems via HTTP, XML-RPC or other protocols, maintenance work for the ZODB (upgrading objects, deleting garbage), type conversions.

External Method is much older than Python Script. Unlike the clear and flexible bindings for Python Script, an External Method gets context information in an obscure and often confusing way. While ZPublisher calls them without surprises, calling them from other contexts involves some magic. When the first parameter is called self and the method is called with one argument less than the number of mandatory arguments, then the method's acquisition parent is passed as first argument. It corresponds to the context of Python Script and provides access to the Web site context as well as to REQUEST, the object describing the current request. Often, this magic does what one expects, but if it fails, and this happens regularly if default argument values are used, many hours are necessary to understand what went wrong. As a rule of thumb: Do not trust the magic, pass self explicitely. The method this (of ObjectManagerItem) is usually appropriate for this task.

1.12. ZCatalog

ZCatalog provides Zope's indexing and search support. A catalog encapsulates a set of named indexes, a set of cataloged objects and for each cataloged object a record of so called meta data. A catalog is used to perform index based searches for cataloged objects.

When ZCatalog indexes an object, it determines index entries for each of its indexes. In a first step, it interprets the index name as an attribute or parameterless method of the object^[13]. It fetches the value or calls the method to obtain "the object's value" for the index. Depending on the index type, the value is transformed into a set of index terms under which the object is then indexed. An elementary search consists always of the retrieval of all objects that are indexed under an index term. Currently, the various index types support different types of complex queries. I hope this will change in the future.

ZCatalog supports three types of indexes: text, field and keyword. A text index splits the value into words and uses each word as index term, a field index uses the value itself as index term. For a keyword index, the value must be a sequence; it uses each sequence element as index term.

A ZCatalog query is given by a mapping that maps index names (and sometimes names, derived from index names) to query specifications. The result is the intersection of all objects obtained by querying the mentioned indexes for the corresponding query specifications. If the mapping is empty (or at least does not name any of the indexes), then the result consists of all cataloged objects.

Keyword indexes are field indexes with a slightly different indexing procedure. Their search capabilities are identical to that of field indexes. A field index supports or and range searches. An or search retrieves all objects indexed under any one of a given set of values. The query specification is either a single value (in which case the or query degenerates to an elementary query) or a sequence of values to search for. A range search retrieves the objects indexed under a term from a given range. For a range query, the query mapping must map the name index_usage to a range specification. A range specification begins with range: and is followed by min, max or min:max. The query specification is again either a single value or a sequence of values. If the range specification contains min, then the minimal value bounds the range from below, if it contains max, then the maximal value bounds the range from above.

The query specification for a text index is either a single query string or a sequence of query strings. If it is a sequence of query strings, the result is the intersection of the results for the single query strings^[14]. The query string is interpreted as a query expression that consists of query terms optionally combined with query operators and, or, andnot or ... (near) and optionally grouped with parenthesis. A query term is usually a word or a phrase. A phrase is a sequence of words enclosed in ". Phrases are transformed into near searches of the containing words. The catalog delegates all word handling to a Vocabulary, also called Lexicon. If the catalog's vocabulary supports wildcards, then the query terms above can contain wildcards. In this case, they are expanded into a sequence of words (without wildcards) combined with or. If an optional operator has been omitted, the or operator is automatically inserted. The query is then executed from left to right, respecting grouping and interpreting the operator in the natural way. Difficult? Let's look at some examples:

The catalog maintains for each cataloged object a collection of meta data, data about the object. It consists of data_record_id_ and the attributes or parameterless methods defined via the Meta Data Table tab. These attributes/methods are evaluated during catalog time, similar to the determination of an object's value for indexing, and stored inside the catalog independent from the object. Zope's search interface wizard uses these meta data to construct the report columns. I view this feature very critical as the so called meta data are actually not additional meta data but can be directly obtained from the object. Storing it with the catalog increases memory consumption and cataloging time; it introduces redundancy, which usually is not a good idea. I define a field only, if either of the following conditions is met: it is very small, such as id, it helps me to detect inconsistencies between the catalog and the object. This is true for bobobase_modification_time, the time, the object was last changed in ZODB., it is a computation intensive method, such as e.g. an intelligent summarization.

The catalog associates internally a unique number with each object, data_record_id_. It is part of the object's meta data table and available for each retrieved object. During cataloging, the catalog also stores the path to the object in order to find it again later during retrieval. The catalog provides two methods, that return either the path or the object itself, given a data record id: getpath and getobject. From Zope 2.3 on, the result of a catalog search behaves like a sequence of brain objects. Each such object describes one hit. Its attributes are the meta data values associated with the object. Furthermore, it provides methods getPath and getObject to access the corresponding path and object, respectively. With these new methods, you will probably no longer use the unique numbers.

Cataloging is only loosely coupled to the fate of an object. If the object is changed after it was cataloged, the cataloged information may no longer be accurate, if the object is deleted, the cataloged information about the object is still there. To fix such discrepancies, the catalog must be resynchronized. This can be done either globally, through the method manage_catalogReindex or for a single object with catalog_object or uncatalog_object (for deletion). There is a mixin class, CatalogAware, that helps (a bit) with the synchronization. CatalogAware objects handle creation and deletion automatically and provide a method index_object to be called explicitly after an object is modified. index_object is easier to call then the catalog's catalog_object. Of course, the object must find the catalog to be updated. It uses a catalog name, which initially is Catalog but can be changed.

How get objects cataloged? When they are CatalogAware, they will catalog themselves automatically when they come into existence. They can be cataloged explicitly with the ZCatalog method catalog_object. Alternatively, the catalog provides the method manage_catalogFoundItems for mass cataloging. Starting from the container containing the catalog, it searches all objects that meet given search criteria and catalogs them. If necessary, single objects can later be removed manually. These functions are available from Zope's management interface, too.

ZCatalog is an essential but unfortunately rather shaky Zope component. If something behaves strangely, then you have hit a catalog bug with some probability.

ZCatalog inherits from Folder. This implies, it is an ObjectManager. Usually, you will place DTML methods into the catalog that present search results. This implies that they have direct access to the catalog's methods. You may also place the objects to be cataloged there. In this case, they can directly use the catalogs methods for cataloging control.

ZCatalog's constructor is manage_addZCatalog.

ZCatalog's text indexes make use of Vocabularys. A vocabulary is a word expert. It knows how to split text into a sequence of words and then maps known words onto integers. A text index does not index the object under a word but really under an integer provided by a vocabulary for the word. The current Zope knows two types of vocabularies. A standard one and a globbing one. A globbing vocabulary recognizes the wildcards * and ? that match any word fragment or any single character, respectively. The globbing vocabulary expands such search terms into the sequence of words matching the term it knows about. The standard vocabulary does not recognize wildcards.

Zope 2.4 extensions

Zope 2.4 provides a set of new ZCatalog features and extensions. The index interface has been cleaned up and documented as PluggableIndexInterface. It is now much easier to implement custom index types and add them to a ZCatalog. Just create a class implementing the PluggableIndexInterface and register it with Zope.

ZCatalog provides now a builtin fourth index type, the path index. As the name suggests, a path index supports queries for objects with a given subpath in their path. Usually, the path must start with the given subpath, but the starting point can be controlled with a level search option. It is an integer that specifies at what path component matching against the query subpath should start. Its default is 0, starting at the root. A negative level searches objects with the given subpath somewhere in their path. The object's value for a path index is the object's physical path. It is split either with the object specific splitPath method or at / into a sequence of path components. The object is indexed under each pair path component, component index in sequence.

Zope 2.4 also provides more flexibility with respect to search specifications. Formerly, a search was specified by a flat dictionary mapping index names to subqueries for this index. Search options for index were specified by an entry index_usage. This form is still supported but deprecated. The new search specification is a mapping from index names to subquery specifications for the index. Such a subquery specification is either a dictionary or an object. The keys or attributes, respectively, specify the subquery. The main part is query, the search expression as described above; the remaining keys/attributes are search options, such as range (for range queries), operator (or or and to select the combining operator in case the search expression is a list, or the default operator for text indexes) or level (for path indexes). Other indexes can support further options.

Queries are often specified via forms. While it is difficult to create a nested dictionary structure from a form, the mapping to subquery specification objects can easily be created by means of the :record form variable name suffix.

1.13. Database integration

Zope supports database integration through two types of object classes: database adapters and Z SQL Methods.

A database adapter is an abstraction for a database connection. Usually, there is an adapter class for each supported database system, which includes Oracle, Sybase, Informix, MySQL, PostGres, SAP-DB. Furthermore, there are ODBC adapters that support the integration of any ODBC capable database (under Windows). The adapter provides a uniform interface towards the other Zope components and hides the specifics of the respective database system. Especially, the adapter is responsible to provide the necessary methods to play together with Zope's transaction system, to open and close the connection and to execute SQL statements against the database. Usually, the adapters also support data model browsing and the direct execution of queries from the management interface. A concrete adapter instance represents a single connection to a specific database. When you need to connect to a database, you would install an adapter product for the respective database type. After that you can instantiate adapter instances anywhere in your site. All objects below the folder with such an instance can easily use that instance.

A Z SQL Method is used to execute SQL commands against a database. Both queries as well as other, e.g. data manipulation, commands are supported. The configuration of a Z SQL Method consists of a reference to a database adapter instance, a set of arguments and a (DTML) template. While templates usually are used to generate HTML pages, the template of an Z SQL Method is used to generate a sequence of SQL statements when the SQL Method is called. These statements are then executed over the database connection identified by the adapter reference. If the SQL sequence contains a query^[15], then the query result is converted into a Results object and returned as result of the call. A Results object behaves like a sequence of Record objects. Especially, it can be easily iterated over, subscription allows access to individual members, the len function can be used to determine the number of records and an empty result sequence is a Python false value. Each Record describes one row of the query result. It is an object with an attribute for each result column. The attribute contains the value for the respective column in the represented row.

The template can use all DTML commands and some additional commands specially designed to facilitate the construction of SQL commands. The names of these additional commands all start with sql. See the DTML reference in Zope's online help, for details. If several SQL commands are generated, they are separated by <dtml-var sql_delimiter>. The template can access the Z SQL Method's arguments and names acquired from the Z SQL Method's context.

A Z SQL Method's arguments are specified as a whitespace separated list of argument specifications. In the simplest case, an argument specification consists of an argument name. However, it may also specify a default value, given after an =. If the default value is empty or contains whitespace, if must be enclosed in double quotes. In principle, it should be possible to specify an argument type, but this is currently broken. When the method is called, arguments can be specified in either of the following mutually exclusive ways:

by providing corresponding keyword arguments,

from the request object.

Z SQL Methods do not look into the DTML namespace to find arguments. This often confuses users, as it does not allow to automatically pass information from the result of one SQL query into a nested SQL query. A Bad request exception is raised when the method is called and an argument without default value does not get a value.

For efficiency reasons, the result of Z SQL Methods can be cached. As always, caching can significantly increase performance. But there are drawbacks, too: as a result of caching, stale information can be delivered. Therefore, caching is disabled by default. When you enable it, you specify how many results may be cached (thus limiting the amount of resources) and how long a result might be cached (thus limiting the life span of stale information). The cache is maintained on a per thread basis. Therefore, it is difficult to flush the cache explicitly and the standard Z SQL Methods do not provide a method for this. If you want to use long term caching, your database is only modified by Zope and you do not want to accept stale information, you can use my Cache Controlled Z SQL Methods. It maintains its cache thread-independent and provides a method to explicitly flush the cache. You would use this method, when your Zope application modified the database and potentially had invalidated the cache content.

As described above, the result of a query is usually a sequence of Record objects that present the corresponding row as a simple object whose attributes correspond the the result columns. However, you can wrap these records into a class of your own, called a brain. This class can give the records interesting behavior. Although essentially a stupid sequence of values in the database, this feature may make a row into a full blown object^[16] .

If a Z SQL Method has a single argument, then the Z SQL Method can be called directly through the Web by appending the value for the argument as URL segment after the URL to the SQL method. This feature, direct traversal, is usually combined with a brain that provides behavior, e.g. displays the result.

1.14. ZClass

We know already all Zope objects necessary to build simple Web applications. For example, it would not be difficult to implement our guest book with a folder as a container for out guest book entries, a catalog to search for them, a DTML document for each guest book entry and DTML methods to implement the search mask and present the search results. Things become more difficult, when such small applications should be instantiated more than once or tightly integrated with other applications on the same Web site. This is the case, for example with our friends and artists. To handle such requirements, Zope provides ZClasses.

What is a ZClass? Easily spoken, it is a blueprint for a Web application or part thereof. We will e.g. define a friend ZClass. It will implement all friend related operations and provide the basic structure for the friend related attributes: annotation management, profile management, registration information, home page design, to just list a few. Technically, a ZClass is an object class which can be instantiated to create objects, instances of the class, which are also called ZInstances. We will instantiate the friend ZClass, whenever we need a new friend object. Each friend, once instantiated, can use all facilities defined by the class. If we later decide to change or enhance the facilities, we need only to modify the ZClass and not the various ZInstances. All instances will immediately observe and use the changed infrastructure.

A ZClass has associated methods and property sheets. The methods are defined through arbitrary objects. Usually they will be DTML methods or Python scripts. However, nothing prevents you, to use images, files or even folders^[17]. All methods are exposed as ZInstance attributes and therefore can be directly accessed through their id. Each property sheet is a named PropertyManager^[18] and manages a collection of (usually related) properties. The properties from all sheets are exposed as attributes of the ZInstance, i.e. they can be directly accessed through their id. All property sheets are exposed as attributes of the attribute propertysheets. This is necessary for property modification.

A ZClass can use inheritance, even multiple inheritance, for its definition. As mentioned earlier, this allows class composition out of pre-packaged components, each implementing a special facility. Available as base classes are other ZClasses and product classes that have been registered as ZClass bases^[19].

As you see a ZClass is very similar to a class. The major difference is that ZClasses can be build through the Web. Its methods are arbitrary other Zope site building objects.

Currently (Zope 2.3), a ZClass can only be created in a product (or another ZClass). Products are managed via Zope's management interface Control Panel->Product Management. In a product, a ZClass is created via the management interface's add list. Once defined, top level ZClasses (i.e. ZClasses not defined inside another ZClass) can be instantiated in all object managers, provided the instantiator has the necessary permissions. This implies, that not the ZClass but the ZInstance is a site building object. The ZClass is only the blueprint for application specific site building objects.

To learn more about ZClasses, the corresponding chapter in the Official Zope Book [Zope] is recommended reading.

2.1. HTTP

Today, almost all Web publishing uses the HTTP (HyperText Transfer Protocol) [RFC 2616] or its "secured" version HTTPS [RFC 2818] which it HTTP over TLS (Transport Layer Security) [RFC 2246] or its predecessor SSL (Secure Socket Layer).

HTTP is a very simple protocol: a client sends a request to a server; the server processes the request and replies with a response. Requests and responses are send as messages over a TCP connection. The message format is essentially MIME (Multi purpose Internet Mail Extension) [RFC 2045-2049], a format used, too, to transfer multi media mail messages across the Internet. A MIME message consists of a set of headers and a body, also known as message entity. The body is optional for HTTP messages. The standards speaks of it as the HTTP entity. For HTTP, a request or response line, respectively, is prepended to the message.

A request line consists of the request method, the resource locator and the protocol version. A resource can be anything: a HTML page, an image, a file, a database, a service, an application. It is identified by the resource locator, a path to easily locate the resource in a hierarchical structure such as e.g. a file system or Zope's folder structure. HTTP uses the URL syntax for the resource locator. It is up to the receiving HTTP server to determine what resource the resource locator does really identify.

HTTP knows a set of request methods. The most essential are GET, POST, PUT, HEAD. GET requests all information about the resource to be transfered in the response. HTML's link traversal and image references are mapped to GET requests. HTML form submission may use a GET request. Additional request headers can make the request into a conditional request or a request to transfer part of the information. Both request modifications are intended to reduce communication traffic. A GET request should not have side effects and should be idempotent. Idempotence means, that a repetition of the same request results in the same response. HTTP clients often use this fact and assume that GET requests can be cached. They save the response to such a request and take the response from their cache when a later GET request targets the same resource. For a Zope Web site, many requests do have side effects. This is especially true, if a session management device is employed. In such cases, it may be necessary to fight with the caching behavior.

A POST request sends information to a resource that this should integrate subordinate to itself. POST requests usually have side effects. You would use a POST request for example to create a new database record, update the properties of a Zope object or post a news item to a discussion board. HTML form submission often uses POST requests.

A PUT request sends information that the server should use to create a resource located by the request resource locator. If there is already such a resource, it may be overwritten. PUT requests are not used from HTML. HTML editing tools use PUT requests to publish an object on a Web site.

A HEAD request is similar to a GET request. It is expected to have no side effects and be idempotent. It should return the same headers as a GET targeted to the same resource but should not transfer the message body. HEAD is not used from HTML. It is, however, used by link validation and indexing tools, to efficiently check the existence, the type, size and other meta information for a resource. It is difficult for Zope to meet the requirements for this request type. Most of its objects are templates that need to be rendered in order to obtain full header information. Rendering, however, can have unwanted side effects. Zope, therefore, returns only approximate, sometimes even wrong information in response to HEAD requests.

A resource may be finer grained then the location of an object in a hierarchical structure. Resource locators for GET and HEAD requests may have a trailing query string that provides additional parameters. The query string is started with a ? which is followed by a sequence of & separated parameter definitions of the form name=value. For POST and PUT requests, parameters may be present in the request body. Their packaging there is indicated by the request's Content-Type header (by e.g. application/x-www-form-urlencoded or multipart/form-data). In Zope, ZPublisher takes care of the parameters, independent whether they are provided as part of the resource locator or in the request body, and makes them accessible in a standard way.

A response begins with a response line. This consists of the protocol version, the numerical status code and the textual status phrase. The status code, a three digit decimal number, tells the client, what happened with the request. The code is divided into classes based on the first digit.

The codes 1xx represent informational responses.

The codes 2xx tell the client that the request has been successfully executed. Usually, the remaining response contains an entity as the primary request result. The usual return code is 200. Other codes indicate, that special information is available in response headers or that the browser should behave in a special way.

The 3xx class calls for redirections. The request is not completed but requires special actions from the user or his user agent. For security reasons, HTTP requires that redirections are only performed automatically for GET and HEAD requests. All other request types require user confirmation. The redirect method of Zope's RESPONSE object uses a 302 status code with the location header set to the new URL. For some objects, especially files and images, Zope responds with a 304 response to GET requests made conditional with an If-Modified-Since header, if the object has not been modified since the given date. In fact, this response is not a redirection. It completes the response without sending the entity data. Conditional requests of this type are usually send for objects in a client's cache when the cache validity should be checked. A 304 response indicates in this case, that the cache entry is still valid and the request can be served from the cache. If the object has been changed since the given date, Zope responds with a 200 response that contains the new information. By default, Zope does not employ this mechanism for template objects as their modification date is not decisive to determine whether or not the generated page remains the same. As it is too difficult to get this right in the general case, Zope always processes such requests as unconditional. However, applications with special efficiency concerns may explicitly generate a 304 response if they can guarantee validity. As of version 2.3, Zope provides an integrated cache manager that can help you to control caches both inside and outside of Zope.

The 4xx status codes indicate a client error. Usually, the response contains an entity that explains the problem and what can be done about it. The most essential codes are

5xx status codes indicate a server error. Zope uses code 500 (internal server error), when some application code tries to set an invalid status code or when it raises an exception that Zope is unable to map to another status code (such as redirect or unauthorized). When Zope is connected to through a proxy, a client may observe other status codes from this class. It usually indicates that either Zope or another proxy on the way to Zope died or a connection broke down.

HTTP is a stateless protocol. This means that a request must contain all information necessary to process it. The server is not expected to have saved state information from previous requests that may be necessary to process this one. This HTTP property makes it quite hard to build more complex Web applications. Of course, users expect from most applications that they are aware of their preferences and remember essential facts from previous interactions. A whole mess of kludges have been defined and implemented to work around this limitation: authentication headers, hidden form variables, cookies, session products. We will learn about these concepts later in this book. I expect future HTTP versions to remove this limitation.

2.2. URL

The URL (Universal Resource Locator) is one of the most essential Web publishing concepts. As the name says, it is used to locate a resource. As we explained in the last section, a resource is used in a very wide sense: it can be almost anything, a person, an object, a service, an application etc. Almost the only requirement is that it can be identified with an identifier, an URI (Universal Resource Identifier). There are different kinds of identifiers. Some kinds contain a description how to locate the resource. These form the subclass of the resource locators, the others are the resource names, URN's. The URI syntaxes for the various kinds have many commonalities. Therefore, the common aspects can be described in a single URI syntax standard [RFC 2396]. Each URI kind is identified by a scheme. Although the scheme determines the precise syntax, URIs, especially URLs, usually consist of up to 4 components: the scheme, an authority, a path and a query. The scheme is always present. It determines which of the other components may or must be present. This means, the generic syntax looks like:

scheme: [authority] [path] [?query]

For resource locators, the scheme usually identifies a protocol which can be used to access the resource. The remaining URL parts provide the parameters necessary for this access. The most prominent protocols in the Web publishing domain are http and its secured version https, as well as mailto and ftp. The mailto protocol accesses the resource, usually a mailbox or mail group, by sending an email to it. The URLs use only the authority part which usually has the format user@host.

The other listed protocols belong to the family of hierarchical URI schemes. Their commonality is the use of the path component, a sequence of (path) segments separated by /. Paths can be used to navigate in a (typically) hierarchical structure: to locate path/segment, segment is used as a local selector in the context of the resource located by path. You know this type of navigation from your file system and indeed the resources are often folders and files in a standard file systems and the URL path component directly mapped to the file hierarchy.

A segment has the form of an id optionally followed by a sequence of parameters, each one preceeded with a ;. These parameters are a recent extension to the URI syntax. Formerly, the path component as a whole could have a single parameter section as suffix. Zope's URI parsing algorithm^[20] still follows this older specification and terminates the path as soon as it sees the first ;. It does not interprete parameters or makes them accessible to the application.

For hierarchical URI schemes, the authority component typically has the form

//[userinfo@]host[:port]

The query component is a string of additional information, interpreted by the resource. For HTTP, it usually is a sequence of name=value components, separated by &. Zope interprets them as argument definitions and makes them available directly via their name.

What is usually used in documents are not URIs but rather a more general concept, an URI reference. An URI reference is essentially an URI, but two aspects make it more general than an URI. An URI reference may have an attached fragment identifier, introduced with #. It identifies a fragment, a part of the resource identified by the URI. And the URI reference may be relative, i.e. it may only specify part of the complete URI with the missing parts given by a base URI. The fragment part is only interpreted by user agents, usually to position the view onto the displayed resource. It is never sent in a request. Likewise, relative URI references are resolved with respect to their base URI to form an absolute URI and only these absolute URIs are sent in a request. A relative URI reference does not follow the above mentioned URI syntax, it is rather a suffix thereof, especially it does not have the scheme component. The rules to resolve a relative URI reference with respect to its base into an effective absolute URI are as follows:

As we speak about Web publishing, most resource references embedded in pages will be URL references: references to images, other pages, mail addresses. If the resources are local, you should usually try to reference them by relative references with respect to the current page. Relative references have the advantage that you can often rename or move a substructure without the need to change all your references. Using relative URLs with respect to the current page works because the default base URI for resolution of relative URIs is the URI of the current document. HTML provides the base tag, a header component, to explicitly specify the base URI. Under some circumstances, the URL used by the HTTP request is not the canonical URL of an object, as would be necessary for correct resolution of relative URLs. Zope knows about many of these circumstances and generates automatically a corresponding base tag. For cases where this is not possible, Zope provides a method that allows the application to set the base.

As we have seen, there are many characters that have special meaning for URI parsing. If they need to be used literally, i.e. as part of one of the URI components rather than as component separator, then they must be encoded. Furthermore, some characters have platform dependent representations. This induces problems for cross platform applications such as Web publishing. There are other problems with some control characters. The URI standard therefore severely restricts the set of characters that can occur unencoded as part of the various URI components. The only characters that can be used unrestrictedly are the ASCII letters (upper and lower case), the ASCII digits and the characters from the set -_.!~*'(). Depending on the URI component, other characters may be allowed, too. For example, the characters :@&=+$, are additionally allowed in path segments. However, you should think twice, whether you really want to use such facts. Any character not allowed in a context must be encoded. The encoding consists of % followed by the two hex digits representing the character's code in the ISO-8859-1 encoding, also known as Latin-1. This is a superset of the ASCII character set. You do not need to worry about the coding details: Zope provides a function url_quote that encodes strings correctly to be used as URI components. You must be aware, however, that encoding is necessary at some places and use url_quote at these places. Zope decodes URLs automatically. Thus, there is usually no need to worry about this aspect.

2.3. HTML FORMs

Currently, most Web published pages use HTML, the HyperText Markup Language. There are tools that allow you to create and modify HTML pages WYSIWYG or at least guided by menus. These are so call HTML editors. Examples are DreamWeaver from Macromedia, Microsoft's FrontPage or Netscape's Composer. However, these tools are designed to build static, stand alone pages. They are of lesser practical use to work on page templates with their additional tags not understood by the tool and their usually non HTML-conformant macro structure^[21]. This implies that for page template design you need a basic understanding of HTML and a tool that lets you use unknown tags and build non-standard pages. Such a tool would usually not support full WYSIWYG editing but may provide menu guidance. Personally, I use XEmacs' HTML mode. But there are many other simple HTML editors for this task, such as HTMLKit.

Although a basic HTML understanding is necessary to build dynamic Web sites with Zope, it is beyond this book's scope to provide a thorough HTML introduction. I, personally, look into the HTML4.0 specification when I need information about HTML. In my view, it is a very good specification which provides introductions, well structured overviews and detailed information combined with good navigation support such as element and attribute indexes. In this book, we will only look at HTML forms, as they are especially important for dynamic Web sites.

An HTML form is the major device that allows users to provide input for Web applications. It is implemented by the HTML form tag. The form tag contains special form controls beside normal text and HTML markup. Controls have a name, an initial value and a current value. The user interacts with the form by changing the current value of its controls, either directly or through script invocations. He may then submit the form. Form submission results in a request being send to an agent, e.g. an email server or an HTTP server. Depending on the context of submission, some controls are being considered successful. For each successful control, the request contains an association control_name=current_value. The order is the same as the controls appear in the document.

Controls are implemented as HTML tags. Their name is given by a name attribute. Their initial value is usually given by a value attribute, for some controls by their content (textarea; option if no value attribute is present). The current value is initially set to the initial value and can later be changed either by the user or a script. Values are strings.

There are controls for (single line) text input, image, submit, check, radio and reset buttons and file input, all implemented by the input tag. Menus are implemented by select, which is a container for options and is available both as a single and a multiple selection. Multi line text input is implemented by textarea. HTML 4 provides additional button and object controls. As a special case, there are hidden controls, also implemented by input. They are used not for user interaction but to transfer information between the page generation process and the request processing after form submission. Such a transfer may be necessary to work around HTTP's lack of state which requires that each request is self describing. Cookies provide an alternative to the use of hidden controls.

Whether a control is successful during form submission is usually determined by its type and its current value. Text input and hidden controls are always successful. Check and radio button controls are only successful, if checked. For selections, each selected option defines a successful control associated with the selection's name. Thus, there may be one, several or none successful controls for a single (multiple) menu control in the submitted form data. A submit or image button is only successful, if it was used to submit the form^[22].

It should be noted, that unsuccessful checkbox controls can make problems during form processing. Similar problems result from multiple selections when no option has been selected. In all these cases, the submitted form data does not contain a definition for the associated control name. The application must take care, to interpret this lack of a value correctly. Zope provides various facilities to handle these cases.

The form tag has one required attribute, action. Its value is an URI reference and specifies the resource that should process the form data when it is submitted. Usually, it is either a mailto or http/https URI. In the first case, the form data is send by email to the given URI, in the second case, an HTTP request is sent. form has several optional attributes, the most essential being method and enctype. method's value is either GET (the default) or POST. When the GET method is used, then the form data is provided as query string in the request locator of an HTTP GET request. As we have noted, the allowed characters inside an URI are severely restricted. Characters not allowed must be encoded, which results in a three byte code for each single byte character. Therefore, this method is inefficient for non-ASCII strings or binary data. You should use POST when your form transmits large non-ASCII strings or even files. If the action specifies a HTTP URI, then an HTTP POST request is used to transfer the form data. Here, the form data is contained not in the resource locator but in the request body. The enctype (encoding type) determines in this case the content type of the request body (and thereby the encoding of the form data). The default enctype value is application/x-www-form-urlencoded, which is the same encoding used for URL encoding and therefore, is inefficient in the same cases as the GET method. Do not use it when your form contains files. Use in these cases multipart/form-data. This uses a multipart MIME message to encode the form data. It can contain binary parts and therefore transfer binary data efficiently. If the form data is sent as email to a human, then text/plain may be appropriate as value for enctype. With this encoding, each successful control results usually in a line of the form name=value without an encoding of characters in name or value. This is adequate for humans. If the email recipient is a program one of the other encoding may be more appropriate as they present no parsing ambiguity and there are standard tools for parsing. If a form is submitted to Zope, any of the request methods and encoding types (exception text/plain) are handled transparently and the form data made accessible in a convenient way.

 <FORM action="http://somesite.com/prog/adduser" method="post">
      <P>
      <LABEL for="firstname">First name: </LABEL>
                <INPUT type="text" id="firstname"><BR>
      <LABEL for="lastname">Last name: </LABEL>
                <INPUT type="text" id="lastname"><BR>
      <LABEL for="email">email: </LABEL>
                <INPUT type="text" id="email"><BR>
      <INPUT type="radio" name="sex" value="Male"> Male<BR>
      <INPUT type="radio" name="sex" value="Female"> Female<BR>
      Interests:
        <SELECT name="interests" multiple>
          <OPTION value="1">Sports</OPTION>
          <OPTION value="2">Politics</OPTION>
          <OPTION value="3">Arts</OPTION>
          <OPTION value="4">Economics</OPTION>
          <OPTION value="5">Family</OPTION>
        </SELECT><BR>
      Origin continent:
        <SELECT name="origin">
          <OPTION>North America</OPTION>
          <OPTION>South America</OPTION>
          <OPTION>Asia</OPTION>
          <OPTION>Australia</OPTION>
          <OPTION>Europe</OPTION>
        </SELECT><BR>
      Interested in further information:
         <INPUT name="info" type="checkbox" checked>
      </P>
      <h4>Remarks:</h4>
      <P><TEXTAREA name="remarks" cols=60 rows=10></TEXTAREA></P>
      <P>
      <INPUT type="submit" value="Send"> <INPUT type="reset">
      <INPUT type="hidden" name="sessionId" value="2417369">
      </P>
 </FORM>

The Forms chapter of the HTML specification contains more detailed information about forms and form processing. It is very recommended reading.

2.4. Authentication

Unlike a static Web site where visitors usually can only retrieve data, a dynamic Web site built with Zope allows in principle all types of site extensions and modifications performed through the Web. It is clear that an administrator wants to control who is entitled to perform such operations. Authentication, the determination of the identity of a requesting agent, is vital for a dynamic Web site.

HTTP provides for an elementary authentication scheme. It is rightfully termed basic authentication [RFC 2617]. HTTP requests can contain an Authorization header. The authorization header identifies the authentication scheme, an authentication token and optionally parameters necessary to interpret the authentication token. For the basic authentication, the scheme is Basic, the token is the base64 encoding of username:password and parameters are not used. When an HTTP server receives a request requiring authorization, it will examine the authorization header. If it is missing or it does not provide for the required authorization, the server will return an Unauthorized response to the client. The response contains a WWW-Authenticate header with one or more challenges. Each challenge consists of an authentication scheme and a sequence of parameter definitions. For basic authentication, a single parameter is defined: realm. Its value is a quoted string. The server URL and the realm value define the protection space of the required authentication. An interactive user agent (a browser) will usually pop up a login dialog for the server and realm when it receives such a challenge, unless login information for the given protection space is already available. The login dialog will ask the user for its username and password with respect to the protection space and will remember this information at least for the session duration. It will then construct an Authorization header from the login information and include it in all requests sent to this server until it receives a new Unauthorized response from this server, maybe for a different realm.

Basic authentication has two weak points. The first is security: username and password are essentially sent in clear text (They are sent base64 encoded. However, it is trivial to reconstruct the original information from the encoding) with every request. Anyone that intercepts such a request can extract the username and password and use it to obtain a false identity. The second is comfort: the lifetime of the login information is controlled by the browser. It usually maintains it during the current session (i.e. the lifetime of the browser process)^[23]. In this case, the user has to reauthenticate each time he restarted its browser.

Recently, a more secure authentication scheme has been defined for HTTP: digest authentication. However, it is not widely implemented. Especially, Zope does not yet support it (but many browsers do not, too).

The authentication scheme used by a Zope Web site is not hard-wired into Zope. Instead, a component, the so called UserFolder decides about all authentication aspects. The standard UserFolder which comes as part of Zope supports only basic HTTP authentication. There are, however, products that use cookie authentication.

With cookie authentication, the authentication token is not sent in an Authentication header but in a cookie. The token usually is not the clear text username and password but some meaningless hash value that is only identified with the user inside the server. Therefore, this scheme appears to be more secure. However, security is increased only marginally. Any request will carry the cookie. An interceptor of such a request can fetch the cookie and use it himself to steal the associated identity with respect to the corresponding server. It is a bit more safe, as it is more difficult to get at the clear text login information. This is essential if the same password is used for other purposes or passwords for different purposes are constructed according to some scheme. However, the cookie authentication provides much more control for the application. It can specify the cookies lifetime. This can either be used to limit the validity period for the authentication token and thereby for a potentially stolen identity. It can also be used to save the cookie for a long time and eliminate the need for logins (almost) altogether. This can be seen as a big advantage by casual users that might otherwise tend to forget their passwords.

Some people (I am one of them) do not like cookies because of privacy concerns. Cookies are often used by Web sites to identify their visitors across visits, collect long term information about their visits and visit patterns and use this information in various ways: to improve their Web site (good), to analyze their visitors interests and use it for personalized marketing (I do not like that), maybe even sell this information (I hate that). Therefore, I look regularly in my cookie file (where the browser maintains long living cookies). When I detect cookies with a lifetime of more than a month or so, I get very suspicious about the site's intentions. I delete such cookies and may disable cookies altogether when visiting such a site.

2.5. Cookies

HTTP is a stateless protocol. This means that each request must be self contained. There is nothing like a context build from previous requests that can be used to interpret the current request. On the other hand, many applications need to be state full. Think of a shopping card. When you look at your card, it must of course contain the items you have sent to it in previous requests. Or think of a form with a complex form field. To fill it, you may need to look at supporting information. When you come back, the form fields you have filled previously must of course retain their values even though the new visit is a new request to the server. How to implement such applications despite the stateless HTTP protocol?

There are several workarounds for this HTTP deficiency. Usually they combine two strategies: first, store information on the server associated with an id, and second, encode the id somehow in the URI or the HTML content. To encode something in the URI, either a path segment or a query parameter might be appropriate. Hidden form controls are appropriate to encode state information inside HTML forms. Usually, these work arounds are tedious and several encoding techniques must be used in combination, for example hidden variables for pages with forms and ids encoded in URI references for link traversal. That's where cookies come in.

The cookie mechanism is very similar to the HTTP authentication scheme we have seen in the last section. Authentication is a typical example where you want state full behavior. You should not need to authenticate for each request separately. After you have logged in once, all following requests should use the login information you provided during this first login. This is possible despite the stateless HTTP protocol, because the user agent provides this information with each request. With a cookie, it is very similar.

A cookie is a named value, defined by the HTTP server and sent to the user agent. The user agent stores the cookie and automatically includes all cookies defined by a given server when it sends a request to this server. Looking at its cookies, the server can access information effectively determined by earlier requests. That's a bit simplified but it gives the general idea. Cookies are great by providing state information for HTTP processing without the need to switch such information between query strings and hidden variables.

Cookies have been invented by Netscape. The cookie specification can be found on Netscape's web site. As cookies solve a fundamental problem with HTTP, they were soon be implemented by other browsers. Nowadays, almost all browsers support cookies.

Earlier I said a cookie were a named value and all cookies defined by a server were sent with any request to this server. As already mentioned, this was a simplification. Actually, a cookie is described by the following attributes:

User agents usually impose limits on the number and complexity of cookies. There is a total limit (300) and a limit per server and domain for the number of cookies (20). The name and value part of a cookie must not exceed 4kB.

In Zope, the RESPONSE object provides methods setCookie, appendCookie and expireCookie to set or unset a cookie. setCookie has the parameters name and value and the optional parameters expires, domain, path and secure, which can be provided as keyword parameters. It sets the cookie specified by the parameters. appendCookie has the parameters name and value. It sets a cookie with this name and value. If the current response object has already a cookie with this name, the new value is appended to the old one separated by a colon. expireCookie has the parameters of setCookie with the exception of value and expires. It deletes the cookie by setting a cookie with an expiration time in the past. The cookies send with a request are made accessible in the cookies component of Zope's REQUEST object. They can directly be accessed through their name in document templates.

Cookies can pose a significant thread to privacy. Be aware that some potential users will disable cookies in their browser.

2.6. ZPublisher

ZPublisher is the component of the Zope framework that is responsible for the following tasks:

We will now look into these tasks in more detail.

2.7. Special Zope objects

ZPublisher defines two objects REQUEST and RESPONSE. They facilitate the interaction between the ZPublisher framework and the (usually) application specific published object.

It is essential to note, that these objects are not persistent. As soon as the current request is completed, they disappear. Many Zope beginners are not aware of this.

<dtml-var REQUEST>

3.1. DTML namespace

The DTML namespace is the namespace implicitly used in document templates for name lookup. It can be explicitly accessed under the name _.

In fact, the DTML namespace is both an object with attributes and a mapping. The mapping is what is implicitly used for name lookup. The attributes provide access to built in objects and functions. You find a list in Zope's online help system.

It will help you to understand the effect of many document template commands, when you know how the mapping part of the DTML namespace is implemented. It is not a flat map, but rather a stack of namespaces, all of them themselves mappings. When you ask the DTML namespace to resolve a name, it asks the top level namespace. If it can resolve the name, that is the result. Otherwise, the next namespace is asked until either the name is resolved or there are no more namespaces on the stack. In the latter case, a KeyError exception is raised.

Figure 3.1. DTML namespace stack

The DTML namespace is a stack of mappings. Mappings are pushed onto the namespace at its top and later popped again. Name lookup starts at the top mapping and proceeds towards the bottom mapping until either the name is found or the bottom is reached.

Many DTML commands which have both a start and end tag push one (or more) namespaces onto the DTML namespace during the execution of their start tag and pop it (them) again during execution of their end tag. We will come back to this during the description of the various commands.

The DTML namespace is not a transparent device. This means, it does not only look up the name and return the associated object (unchanged). Rather, it usually processes the object looked up. This is because the main purpose of the DTML namespace is to support name resolution inside document templates. Document templates should usually generate strings. This implies, everything must finally be converted into a string. Now, most objects have a static, and very unimpressive string representation. It is rarely context sensitive. Therefore, the DTML namespace subscription calls the object, if it is callable. It even knows about some object classes and calls them with adequate argument, for example document templates and Python scripts. If you need the object unprocessed, then you can use the DTML namespace's getitem method. Thus, _[x] calls the object the name of which is given by x, while _.getitem(x) returns the object unprocessed. You need to use the second form, whenever you want to access the object's attributes.

I have said that the DTML namespace is something special in Zope. I lied a bit. Stacks of namespaces are commonly used in all block structured languages. DTML is just another (specialized) block structured language. Its namespace is not so special. Special is, however, that the namespace lookup is not transparent.

3.2. Acquisition

We now come to a powerful, but not easy to understand Zope concept: acquisition.

Usually, an object's namespace is context independent. Thus, if o is an object and name a name, then o.name always yields the same object. Acquisition makes the object's namespace context sensitive. If the object itself cannot resolve the name, then the context in which the object was accessed is asked to resolve the name. This is similar to the DTML namespace. However, while the DTML namespace is a linear sequence, acquisition defines a tree like structure. Let's look at the details.

There are two concepts that play together to implement acquisition: the class ExtensionClass and acquisition wrappers. ExtensionClass was developed by Jim Fulton who also developed Zope and is Zope Corporation's technical director. ExtensionClass is an extension to Python, implemented in C. It defines a Python type. Unlike other Python types, it behaves also like a class. Especially it can create instances and be subclassed. It provides various enhancements for the classes that inherit from it. For example, it allows that methods get attributes and that members are in fact methods (ComputedAttribute). What is relevant for acquisition is a special implementation of attribute lookup. It is no longer transparent. When an attribute a is looked up in an ExtensionClass instance i, then the resulting object o is not simply returned. It is rather checked, whether o has a method __of__. If it has, the attribute lookup result is not o but o.__of__(i). You can read this as "o in the context of i" or "the o of i".

The next important thing are acquisition wrappers. An acquisition wrapper is a class instance with two members (we will see later that there are in fact further members): aq_self and aq_parent. An acquisition wrapper directs attribute lookup first to aq_self, and if this lookup fails, to aq_parent. There are two classes of acquisition wrappers: implicit and explicit ones. Implicit wrappers automatically continue lookup in aq_parent for normal attribute access, while explicit wrappers only do this, if the lookup uses the special method aq_acquire. In the Zope framework, there are two mixin classes Acquisition.Implicit and Acquisition.Explicit, both derived from ExtensionClass, that define __of__ methods. The __of__ constructs implicit or explicit acquisition wrappers, respectively, and put the object into aq_self and the context into aq_parent. Most Zope object classes are derived from Acquisition.Implicit. This implies, they support implicit acquisition. The objects you handle in Zope are usually implicit acquisition wrappers consisting of an object (often another acquisition wrapper) and a context (often another acquisition wrapper, too). As we have explained, the attribute lookup will first look in the object itself and will then look in the context, if necessary. If the result has an __of__ method, then it will again be wrapped with the original wrapper as context.

Figure 3.2. Acquisition wrapper

An acquisition wrapper has two components, aq_self and aq_parent. It directs name lookup first to aq_self and if this fails to aq_parent.

Let's see what that means! If you have o=a.b^[30], then o will not only have the attributes of b but it appears to have also all attributes of a. It has acquired a's attributes. In the same way, if o=a.b.c, then o can access the attributes of c, b and a. In a Zope Web site, this implies, that whenever you have accessed an object o, then the wrapper returned for o will not only have access to o's attributes but also to all attributes of its ancestors. This is because there must be an access path from the root to o which necessarily contains all of o's ancestors^[31]. The wrapper has access to all attributes of objects on this access path and especially that of the ancestors.

Note, however, that the order in which attributes are looked up is not the reverse access order. Let's look at an example. Let's assume, object a has attributes b and c. Let o=a.b. Then o acquires c of a. Call o.c oc. Then oc will have access to all attributes of c, b and a. Let's assume, both a and b have attributes named d. What attribute will oc.d be? It will be (a wrapper for) the one of a and not the one of b. To understand why, we must look, how oc is constructed. o is in fact b.__of__(a). When c is looked up by this wrapper, then first b is asked whether it can resolve this name (we assume not), then a is asked and it responds with c. This lookup in a wraps the result in its context: it yields c.__of__(a). This partial result is then wrapped together with o to yield the final result: (c.__of__(a)).__of__(o). When d is looked up in this wrapper, first its aq_self is asked. This is c.__of__(a). The lookup succeeds as a has the looked for attribute; o and therefore b are not consulted. If neither c nor a had an attribute d, then it would have been looked for in o and the d attribute of b would have been found.

Let's look again, what that means in terms of the Zope Web site structure. Suppose, you have two objects a and b and a accesses b. How does b finds an attribute in this context? It will look, starting from itself towards the root, in search for the attribute. If it fails along this path, it will continue the search in an order that is not easy to describe and finally search it via a. Thus, b has access to any attribute of a, but only if none of its ancestors (itself included) and maybe other intermittent objects has itself an attribute of the given name.

This acquisition feature often confuses Zope users. Acquisition lies between static binding and dynamic binding. With static binding, an object would only see its attributes or that of its ancestors but never that of other objects on the access path. With dynamic binding, the attributes would be looked up in the reverse order of the access path, independent of the ancestors. Acquisition, however, first tries to resolve the name via static binding. If this fails, it looks for the access path but not necessarily in reverse access order as dynamic binding would do.

Jim Fulton, the acquisition's father, called this property "Containment before Context" in his Acquisition Algebra Talk. Acquisition provides access to the attributes of the containing objects, i.e. the object's ancestors, and to the attributes of the access context. Acquisition looks first in the containment and only then in the context. Please, keep this in mind, when you should face an apparently wrong attribute lookup.

4.1. General syntax

When such a template is rendered, it (usually) returns text, most often HTML text although document templates can generate any type of text: plain, HTML, XML, SGML, whatever. A document template consists of literal text intermingled with DTML commands. As the primary target are SGML documents, especially HTML documents, DTML uses SGML syntax to describe its commands: elements, attributes and entity references. This way, document templates look very similar to HTML pages.

Elements, attributes and entities all have names. Elements are delimited by a start and end tag. Usually, they look like <name [attribute_assignment...]>content</name>. There are elements that, by design, must have empty content. For such elements, the end tag can be omitted. For DTML, the end tag of such elements must be omitted. An attribute_assignment usually has the form attribute_name=value. As attribute assignments are separated by white space, value must be quoted, if it contains white space. DTML only allows the double quote character ". DTML allows to omit the attribute_name= part for the first argument, provided that the attribute name would be either name or expr. Quoting of the value determines, whether DTML assumes name (no quoting) or expr (quoting). Many attributes of DTML commands allow to omit the =value part of the attribute assignment. In this case, an attribute specific default value is provided. The names of all elements used as DTML commands consist of the prefix dtml- followed by the command name. Attributes are used to provide parameters for the DTML commands. Many commands render their element content, maybe repeatedly, in a DTML namespace that was specifically enlarged for this rendering.

The content of many commands is structured into various parts by special empty elements. The classical example is the if command. It has the general format

<dtml-if name> if_text <dtml-elif "expr"> elif_text <dtml-else> else_text </dtml-if>

The structuring elements are not used stand alone but only inside the content of the commands they belong to.

4.2. Object argument

Most DTML commands operate on an object. Their result is primarily determined by this object (and the command's content). The object can either be designated by a name or by an expression. In the first case, the object is given by the value of the name, in the second case by that of the expr attribute.

To resolve a name attribute to an object, the name is looked up in the current DTML namespace. This means the object corresponding to name is _['name']. Note that this lookup is not transparent. As we already described, the object determined by direct lookup is called, if it is callable, and the result is the object returned from the call.

An expr attribute is resolved to an object by evaluating its value as a Python expression. If the expression contains (Python) names, then they are looked up in the current DTML namespace. However, the transparent lookup, provided by the namespace's getitem, is used for this lookup. This means, if name is looked up in an expression, the resulting object is _.getitem('name'). Therefore, the attribute assignments name=name and expr=name sometimes yield the same object but not in general. The resulting object is only the same, if the directly looked up object, i.e. the one returned by the expr attribute, is not callable. With very few exceptions, name=name is equivalent to expr="_['name']".

Note that the value of an expr attribute must be a valid Python expression. Especially, names in such expressions must be valid Python names. A Python name must start with a letter (or underscore) and can consist of letters, digits and underscores. DTML names, on the other hand, have almost no restrictions: they must not start with an underscore and some few names are reserved. This implies that many DTML names are no valid Python names. This is especially the case for DTML names containing a hyphen. A hyphen is used in Python as subtraction operator and especially as name delimiter. What is in DTML a single name is interpreted inside a Python expression as several distinct names combined with the subtraction operator. This often results in NameError exceptions as the name components are not defined individually. This is a bit of a problem for newcomers, as many Zope commands like to use names containing hyphens to provide information for the rendering of their content.

Note that attribute values must be enclosed in double quotes if they contain white space. This is often the case for Python expressions. It is good practice to always enclose the value of the expr in double quotes.

Because of their importance, DTML provides a shorthand notation for the name and expr attributes. The attributename= can be omitted, provided it is the first attribute. Moreover, in such a case, the quoting of the attribute value determines whether the value is interpreted as a name or an expression: if unquoted, it is a name, if quoted, an expression. The omission of the expr attribute is now deprecated as it caused too much confusion, especially among newcomers.

4.3. Commands

Most DTML commands have attributes and content. There are a few commands without content. In the following description, commands are described using an abstract synopsis rather than the concrete SGML syntax. Only the most essential commands and attributes are described. The object argument is not described with the exception that it is noted, when the command does not support it.

Some arguments require an explicit value, while most have a default value. Usually, there is a big difference between an argument not mentioned at all and the argument given without a value. In the first case, the operation indicated by the argument is not performed; in the second case, the operation is performed as given by a default argument value.

&dtml[.argument]...-variable;

<dtml-var variable [argument]...>

<dtml-if message>
  <div class="message"><dtml-var message></div>
</dtml-if>

<dtml-if message>
  <dtml-if "message[0]">
    <div class="error"><dtml-var "message[1]"></div>
  <dtml-else>
    <div class="success"><dtml-var "message[1]"></div>
  </dtml-if>
</dtml-if>

<dtml-unless names>
  <dtml-call "REQUEST.set('names',())">
</dtml-unless>

<dtml-in objectValues>
  <dtml-var title_and_id><br>
<dtml-else>
  The folder is empty.
</dtml-in>

<dtml-in objectValues start=start size=20 sort=id>
  <dtml-if previous-sequence>
     <p>
     <a href="<dtml-var URL
             ><dtml-var sequence-query
             >&start=<dtml-var previous-sequence-start-number>"
     >Previous <dtml-var previous-sequence-size> objects</a>
     </p>
  </dtml-if>
  <dtml-var title_and_id><br>
  <dtml-if next-sequence>
     <p>
     <a href="<dtml-var URL
             ><dtml-var sequence-query
             >&start=<dtml-var next-sequence-start-number>"
     >Next <dtml-var next-sequence-size> objects</a>
     </p>
  </dtml-if>
<dtml-else>
  The folder is empty.
</dtml-in>

<dtml-in shopping>
  <dtml-if sequence-start>
     <table cellspacing=10 width=80% align=center>
       <tr>
         <th align="left">Article</th>
         <th align="right">Price</th>
       </tr>
  </dtml-if>
       <tr <dtml-if sequence-even>style="bgcolor=#c0c040"</dtml-if>>
         <td><dtml-var article></td>
         <td align=right><dtml-var price fmt="$%.2f"></td>
       </tr>
  <dtml-if sequence-end>
       <tr><td colspan=2><hr></td></tr>
       <tr>
         <td>Total</td>
         <td align=right><dtml-var total-price fmt="$%.2f"></td>
       </tr>
     </table>
  </dtml-if>
<dtml-else>
  Your shopping basket is empty!
</dtml-in>

<dtml-call "REQUEST.set('variable',15)">

<dtml-call "manage_changeProperties(REQUEST)">

<dtml-let query_result=queryDatabase
          hits="_.len(query_result)"
>
  <p>
  <dtml-if hits>
    Your query resulted in <dtml-var hits> hits.
    <dtml-if "hits > 10">
       The first ten are shown below.
    <dtml-else>
       They are shown below.
    </dtml-if>
    <dtml-in query_result size=10>
       ....
    </dtml-in>
  <dtml-else>
    Your query did not yield any hits.
  </dtml-if>
  </p>
</dtml-let>

<dtml-in seq>
  <dtml-let index=sequence-index>
     ...
     <dtml-var expr="pseq[index]">
     ...
  </dtml-let>
</dtml-in>

<dtml-let name="'Charles'" age="57">
  <dtml-var showPerson>
</dtml-let>

<dtml-var "showPerson(_.None,_, name='Charles', age=57)">

<dtml-with images>
  ...
  <dtml-var image1>
  ...
  <dtml-var image2>
  ...
</dtml-with>

<dtml-with "REQUEST.form" mapping>
  ...
  <dtml-var form_variable1>
  ...
  <dtml-var form_variable2>
  ...
</dtml-with>

<dtml-unless "REQUEST.form.has_key('selection')">
  <dtml-call "REQUEST.form.update({'selection' : (), })">
</dtml-unless>

<dtml-tree branches=objectValues>
  <dtml-if icon>
     <a href="&dtml-tree-item-url;" target=display>
     <img src="&dtml-icon;" border=0 alt="&dtml-meta_type;">
     </a>
  </dtml-if>
  <a href="&dtml-tree-item-url;" target=display>
  &dtml-title_or_id;
  </a>
</dtml-tree>

<dtml-tree book branches_expr="objectValues(['Folder', 'DTML Document'])
     sort=position leaves=showLeaf>
  <dtml-var title_or_id>
</dtml-tree>

<dtml-var standard_html_header>
<dtml-var "_.render(this())">
<dtml-var standard_html_footer>

<dtml-try>
  <dtml-with "_.namespace(REQUEST=REQUEST)" only>
     <dtml-with REQUEST mapping>
        <dtml-let object__="PARENTS[0]">
          <dtml-with "object__">
            <dtml-var object__>
          </dtml-with>
        </dtml-let>
    </dtml-with>
  </dtml-with>
<dtml-except>
  <dtml-call "RESPONSE.setStatus(error_type)">
  <html>
    <head>
       <title>Error</title>
    </head>
    <body>
      <h1>Error occurred</h1>

      <p>
      Error type: <dtml-var error_type><br>
      Error value: <dtml-var error_value>
      </p>

      <h3>Traceback</h3>
      <dtml-var error_tb>
    </body>
  </html>
</dtml-try>

<dtml-unless check>
  <dtml-raise type="ValidationError">
     You request is not valid
  </dtml-raise>
</dtml-unless>

<dtml-if "AUTHENTICATED_USER.getUserName() != 'Anonymous User'">
  <dtml-raise type="Unauthorized">To logout, please submit the login dialog without entering a password.</dtml-raise>
<dtml-else>
  <dtml-var standard_html_header>
  <p>You are now logged out.</p>
  <dtml-var standard_html_footer>
</dtml-if>

<dtml-return "[
  {'label' : 'Name',     'control' : 'text', },
  {'label' : 'Email',    'control' : 'text', },
  {'label' : 'Interest', 'control' : 'selection', 'multiple': 1, 
   'options' : ((1,'Sports'), (2,'Art'), (3,'Music'), (4,'Film'), (5,'Journeys')),}
  ]">

<dtml-unless Name>
  <dtml-return "'You must specify a name'">
</dtml-unless>
<dtml-unless Password>
  <dtml-return "'You must specify a password'">
</dtml-unless>
<dtml-if "Password != Password_retyped">
  <dtml-return "'The Password and retyped Password must be identical'">
</dtml-if>
<!-- successful -->
<dtml-return "0">

4.4. Calling DTML objects

Document templates are executed, or rendered, by calling them. They accept two optional positional arguments, called client and REQUEST, and arbitrary many keyword arguments. They use their arguments to build an initial namespace and then render the template in this namespace. Logically (though not physically), the namespace is constructed as follows:

When a DTML object is called implicitly during namespace lookup, through namespace subscription or via the namespace method render, the namespace recognizes it as a DTML object by its member isDocTemp. It will call objects with a true isDocTemp with 2 arguments, None and itself. This means, with client=None and REQUEST=_. This passing of the namespace as REQUEST parameter ensures that the called DTML object has access to the callers context. For a DTML method, nothing else is available as context. A DTML document can use its own context, in addition. When you call a DTML object explicitly, either inside a Python expression in DTML or inside a Python script or external method, you need to ensure that it gets the necessary arguments. While trivial DTML objects do not need any arguments or only keyword arguments, it is very likely that more complex ones require at least a namespace or a client. Watch out for missing arguments when you get unexpected KeyErrors or NameErrors.

5.1. Through the Web management

A Zope site is usually managed through the Web by means of a Web browser. Each object has a Web callable method manage. It displays a frameset or page used to manage this object: view the object, its contents, properties and change any of these aspects. The information is usually organized into a set of views, such as Contents, View, Properties, Security, History, Undo, Find. Each view provides access to a set of related items associated with the object. The Properties view, for example, displays the object's properties, allows to define new properties and to edit or delete current ones. The Contents view displays the objects contained in an ObjectManager, allows to add new objects and manage or delete existing ones. Sometimes different views show related information but process them differently. For example, the view and edit views of DTML objects both show the associated DTML template. However, view renders the template while edit shows the source in an editable HTML textarea.

The available views are shown as tabs on the management pages. Standard tab selection is used to switch between views. To use a view, a user must have a view specific permission. The management interface shows just the views the current user is entitled to use.

While most objects have just the Web management interface as described: a set of views selectable through tabs at the top of each view, you will usually see a different interface. The interface of object managers consists of two frames. The right frame contains the views display, at described above. The left frame visualizes all the ObjectManager sub objects directly or indirectly contained in the object as an interactively explorable hierarchy. This hierarchy display is implemented through a DTML tree command. It provides fast access to these object managers. Clicking on one of them shows the default view in the right frame.

The management interface is documented in the embedded Zope help system. Each view has a help button in the upper right corner. It opens the Zope online help window with the description of the current view: its form controls and the possible actions.

5.2. Programmatic Management

All views and management interface actions are implemented through methods of the managed object. By convention, most of them have names starting with manage_. The available methods are of course type specific. These methods can be called from DTML templates and scripts as well as via the Web.

There is currently no good and complete documentation about the available management functions. Some of them, actually few, are documented in the API section of the embedded Zope online help. Fortunately, Zope is open source and its source is documented quite well. The available views of an object are defined by the class member manage_options. It is a tuple of dictionaries. The dictionaries usually have keys label, action and help. They define the label to be used in the management interface tab, the method name implementing the view and the help page documenting the view, respectively. Most standard Zope objects are defined in the (Python) package OFS, their sources can be found in the (file system) folder lib/python/OFS. Some objects are defined in products. Their sources can be found in lib/python/Products/productname.

Another useful resource is __ac_permissions__. It defines the permissions necessary to call methods or access the object's members. It is a tuple of tuples. Each tuple describes one permission the name of which is given as first component. The second tuple field is a tuple of method names. These are the methods controlled by the permission. A user must have this permission to be able to call the method. An empty method name indicates access to the object's members. The optional third tuple field is a tuple of roles. These roles have the permission by default; it can be overridden through the management interface. Thus, the second tuple component shows you the methods that can be accessed through the Web. These include all management functions. You must keep in mind, that permission specifications can be inherited from base classes. Thus, to get a list of all management functions, the base classes must be explored, too.

The views are usually implemented through file system based DTML objects collected in a dtml sub folder. Most standard Zope objects are defined in the (Python) package OFS, thus most DTML files are located in lib/python/OFS/dtml. Some objects are defined in products. Their view definitions can be found in lib/python/Products/productname/dtml. It is quite instructive to explore how Zope's management interface is implemented, especially when you want to do similar thing programmatically.

Once, you have determined the appropriate method and its parameters, you can call it with the DTML call command from DTML or directly from a script.

After I wrote this, I was convinced their must be a better way to find out about the available management functions and how to call them. My answer was a small product DocFinder. DocFinder allows you to ask Zope about the documentation for any of its objects. The documentation contains the available attributes, the roles allowed to access them, the arguments provided the attribute is a method and a documentation string, if provided by the implementer. It gives you a very clear overview about the object's infrastructure and how it was build from Zope's various building blocks. Take a look at it.

    def manage_changeProperties(self, REQUEST=None, **kw):
        """Change existing object properties.

        Change object properties by passing either a mapping object
        of name:value pairs {'foo':6} or passing name=value parameters
        """

<dtml-call "manage_changeProperties(REQUEST)">

Most management methods return an HTML page that is used as response when they are used from Zope's management interface. If you feel the need to call these methods directly, there is a high chance that you want to provide your own response page. Otherwise, your user will suddenly see Zope's management page. Thus, you are likely to discard the method's return value as is done by the DTML call command in contrast to the var command. A few management methods behave even worse. They perform a redirect to a management page. Such a redirect tells the browser to discard the response page and load the other page. When you observe such a behavior, you can counter it with a RESPONSE.setStatus(200) after the management method call.

The description above should tell you how to proceed when you feel the need to do management functions programmatically inside your Zope system. With the help of the ZClient, you can do this even from outside Zope in an external Python script or from the command line. ZClient, more precisely ZPublisher.Client, is a utility that helps you to call functions and methods via HTTP from a Python script. It can be used to automate arbitrary HTTP sites not just Zope sites.

The central facility defined by ZClient is the Function class. It provides a function interface for access to a given URL. Its constructor has the following synopsis:

A function object f can later be call with the following synopsis:

Client defines two wrappers around Function, the class Object and the function call. Object instances are local proxies for objects identified by an URL. The attribute access syntax obj.attributeName, can be used to access the attributes of the remote object. The result is not another Object instance^[38] but a Function instance which can be called as shown above. The Object constructor supports the parameters url, method, username, password, timeout and **headers with a semantics similar to the corresponding parameters for the Function constructor. The function call supports the parameters url, username, password and arbitrary keyword arguments. It uses the explicitely named parameters to construct a Function object and then calls this with the remaining keyword arguments.

from ZPublisher.Client import call

call('http://localhost:8080/Control_Panel/manage_restart', username='admin', password='x22qv?s.')

from ZPublisher.Client import call

call('http://localhost:8080/Control_Panel/Database/manage_pack', username='admin', password='x22qv?s.', days=30)

An application of this technique is the program load_site. It supports mass imports from the file system into a Zope site. You find this utility in the utilities sub folder of your Zope distribution. You can look at its code for an advanced example of the technique.

5.3. FTP/WebDAV management

Zope has an integrated FTP server and supports WebDAV. This implies that you can explore the side with an FTP or WebDAV client, make modifications and, with some restrictions, create new objects. As there are tools that can virtually map FTP sides or WebDAV sites into your local file system, this provides for a very smooth integration into your usual working environment. There are two drawbacks, however: error messages are much less informative than they are through an HTML browser, at least for FTP management. This is because FTP only knows about very few error reasons and Zope's standard FTP support does not provide informative additional texts. Richard Jones posted a patch to Zope-dev that provides for better error information. I am not sure about WebDAV in this respect as I did not try this out. The second problem is the way how many tools operate on files in the file system: they implement changing a file by creating a new temporary file, write the modified content into it and then rename the file to the original which is deleted by that operation. This is nasty for Zope objects for two reasons: all information associated with the original object beside its content is lost: versions, history, meta-data^[39] and the new object is often created with a wrong type. Tools implementing modification by creating a new object can therefore not be used to modify Zope objects mapped into the file system. Fortunately, many editing tools for Web content directly support FTP and/or WebDAV. In this mode, they often do not create temporary objects.

Both FTP and WebDAV can only create standard Zope objects. This is because they are designed to manage files but the objects in Zope are not simple files but have a finer grained structure. Zope implements a mapping from filename extensions to object types to create objects of the correct type. Of course, this mechanism can only create object types that it knows about; it will not create ZInstances or instances of application specific extensions. Again, this is hookable. If an object has or acquires an attribute PUT_factory, then this method is called with the parameters name, content_type and file_content: the name of the object to be created, the content type as determined either from the request's Content-Type header or guessed from the name extension, and the content of the uploaded file. If the method is able to determine the object type, it should return a new object of this type. Otherwise, it should return None. In this latter case, Zope's default factory is used which is based on name extensions and which creates a DTML Document if it is unable to determine the type.

Many modern Web editors support WebDAV, the Web Distributed Authoring and Versioning protocol. It was designed to enable safe concurrent authoring over the Inter/Intranet. As Zope supports this protocol, too, the editors can directly access, view and modify content inside Zope -- in the same way as if they were local files. Examples of such editors are Adobe's GoLive and Macromedia's DreamWeaver. Under Unix, the general purpose WebDAV client cadaver can be used. Different editors can plug in to become WebDAV enabled.

6.1. Permissions

Zope is an object oriented system. The primary operation in such a system is attribute access. Zope allows the access to any attribute with a non-elementary data type to be protected individually by a permission. A permission is an abstract entity without inherent semantics, it is represented by a string. Of course, this string should express the purpose of the permission, e.g. View management screens expresses (some kind of) management permission. Usually, a set of related attributes, often from different classes, are protected by the same permission: for example, View management screens protects the attributes manage, manage_menu, manage_top_frame, manage_page_header and manage_page_footer from class App.Management.Navigation and manage_cutObjects, manage_copyObjects, manage_pasteObjects from class OFS.CopySupport.CopyContainer, among others. Zope will prevent attribute access, if the current user does not have the required permission.

6.2. Roles

Zope does not associate permissions with individual users. This would make for too high a management overhead. Instead, Zope uses another abstraction, the role, to reduce the amount of permission association. A role abstracts several related tasks: for example, the Manager role is the culmination of all management tasks: creating user records, assigning roles and mapping roles to permissions, changing and controlling the site structure. On the other hand, the Editor role stands for all kinds of content work: changing, reviewing, releasing content. As with permissions, roles are abstract entities without intrinsic semantics. They are represented by strings. You can define your own roles, if this becomes necessary for your site.

6.3. Permission and role assignment

While an attribute can be protected only by a single permission, a user can have assigned any number of roles. Furthermore, you can give a role any number of permissions. Thus, you can, for example, assign to the Mr. Chief Editor the roles Editor and Supervisor. You can give to the role Editor the permissions addArticle and changeArticle while Supervisor gets the permission releaseArticle. Our example user will then indirectly have the permissions addArticle, changeArticle and releaseArticle.

When a user tries to access an attribute, it is checked whether the user has a role which in turn has the permission protecting the attribute. It he does not, an Unauthorized exception is raised. As we have seen, this triggers a login dialog.

Users and their roles are maintained by special Zope objects, so called User Folders. A User Folder is a collection of users and associated information. This information always includes the credentials necessary to authenticate a user, such as user name and password, and the user's roles. Third party user folders can associate additional information with users, such as email address, full name, address and so on.

The role assignment in the User Folder is global. This means, that the user has the assigned roles anywhere in the site that is governed by this User Folder^[40]. Sometimes, you may want to assign a role only in a restricted context. For example, you may want to assign the Supervisor role to Mr. Chief Sports Editor only in the Sports folder. You could use a local role for this purpose. A local role is just a role that is not assigned globally, in the User Folder but only locally in the context of a given object (including its sub objects). Local roles are defined in the Security tab of the object's management screen.

The assignment of permissions to roles is also managed in the Security tab. This tab is essentially a big table with the rows corresponding to the permissions and the columns to the roles. The checkboxes in the table cells tell you and allow you to control whether the corresponding permission is granted to the corresponding role. The first column controls, whether permission assignments are inherited from the containing object. If checked, the checkboxes to the right provide additional grants, adding to that inherited from above. If unchecked, the permission is granted just to the roles specified by the checkboxes to the right; grants higher up in the hierarchy are irrelevant.

6.4. Proxy roles

Sometimes, it is necessary that a user can perform actions in a restricted context that he is not entitled to in general. For example, you will usually not allow anonymous users to call SQL methods as they could try to search for private information or even change your database. However, for registration purposes, it might be necessary that a yet anonymous user is able to write a user database record providing his personal information.

You can achieve this with a proxy role. A proxy role provides an object with alternative permissions during its "execution". Usually, an object is "executed" with the permissions of the user. However, if the object has proxy roles, the user's roles are irrelevant and "execution" uses the object's proxy roles.

Thus, to implement the user registration, you can restrict the permission Use Database Methods to the role Manager and give the registration action the proxy role Manager. This action would check the validity of the registration information and only use a database method to update the user database table if everything is okay.

As of Zope 2.2, proxy roles are no longer passed to called objects. Thus, if an object with proxy roles calls another object, then the called object is not executed with proxy roles, unless it itself has proxy roles defined. This provides for fine grained control over roles. In addition, proxy roles can both extend and restrict permissions, although this may not be of great practical importance.

Of course, proxy roles are managed through a corresponding tab in the management interface of objects for which they are applicable.

6.5. Owner restrictions

The security model described so far was essentially that of Zope before version 2.2. Then, it was discovered that it is very prone to Trojan Horse attacks.

In a Trojan Horse attack, the intruder drops a malicious object at an appropriate place. The object is malicious as it may try to do things, the intruder itself is not entitled to do, for example delete the complete Web site or (more likely) place (potentially) interesting but unwanted material into the entrance page. Of course, when the intruder executes the object itself, the malicious action is prevented by Zope's security system. If, however, the intruder achieves to somehow attract a user with sufficient authority to execute this object, it will perform its disastrous task.

Because Zope provides explicit control over error handling and sufficient introspection facilities, it was quite prone to such attacks. In a system without such features, the intruder's malicious object would probably have been discovered by the exceptions raised when a non-authorized user executes the object. In Zope, however, the intruder can simply catch these exceptions or even ask Zope whether he is entitled to execute the malicious operation and do it only, if Zope answers "yes".

From Zope 2.2 on, this security whole has been blocked. Most Zope objects, and all Zope objects a normal security sensible administrator would allow to be created by an untrusted user, have an Owner. When Zope executes an object, then it uses as effective permissions the intersection of the permissions of the object's owner and that of the current user. This almost removes the danger of Trojan Horse attacks, as even if the Trojan Horse object is executed by a user with very extensive permissions the execution can not do things, the intruder could not do directly. The danger has not completely vanished, though. If by some trick, the intruder is able to convince a trusted user to take over ownership of the object then it may perform its malicious work. However, this is much more difficult than trick him to view the object.

6.6. AUTHENTICATED_USER

The REQUEST object has an attribute AUTHENTICATED_USER that describes the current user. You can ask this object about the current user's name, his roles and potentially about additional attributes associated with the user, such as email address, real name, address and so on. The standard Zope user objects currently do not support additional attributes but third party user objects often do.