1.1.1. Modularization

The information technology often starts with simple, i.e. easily implementable, solutions. Their success leads to an overwhelming mass of entities which soon requires improvements to the basic solutions to organize and manage this new wealth of possibilities. This adds complexity to the simple solutions and sometimes requires completely new solutions. Something like this is currently happening to the web. Its success produced hundreds of millions of web pages interlinked with one another. It is very easy to create a web page and it is easy to maintain a set of, say, some dozen web pages. But it becomes a nightmare, when one wants to maintain hundreds or thousands of web pages. There are some standard recipes to tackle this form of complexity:

modularize and factor out common aspects

separate different concerns, such as e.g. content, logic and presentation

reuse existing components whenever possible

parameterize components and make them configurable to improve reusability

The key idea here is modularization: build your web pages from a set of modules or components. Compose your web page from existing elements with a few new elements rather than build them each time from scratch. Let's be a bit more specific and look at some concrete examples:

We have seen that modularization can help you to increase consistency and uniformity. We also saw, that often modifications need to be done at only a single place rather than many places. This increases efficiency and avoids errors. To make effective use of this modularization feature, you will need automatic composition support that ensures that the final pages are up to date with respect to the modules they are composed of. There are special tools that provide this support for static web pages, such as e.g. Schematext. When you made relevant changes to your modules, you can tell these tools to regenerate all your static pages. However, your static site does not change automatically but only after the pages have been recreated (you may view this as a feature). There is no need for special tools for dynamical web pages. Such pages are composed from their components automatically during the request. Changes to a component are automatically visible at the next request.

To summarize, dynamic web pages automatically support modularization, the composition of web pages out of components during publication.

1.1.2. Searching

A well structured static web site supports the exploration of its content through browsing. The home page presents of set of content categories, e.g. News, Products, Support etc. implemented as links to the entry pages for these categories. These entry pages are similarly structured with respect to the set of their sub-categories. A visitor descends through the hierarchical structure until he finds what he is looking for. This exploration form is adequate for unfocused visitors or for relatively small content domains. It becomes unfeasible for large content bases and ineffective for visitors with precise information requirements. Such cases call for an alternative way of exploration: searching. With searching, the visitor specifies what content he is interested in, usually in the form of a set of relevant keywords, and is presented with a search result overview. The overview provides a list of pages matching the query. Usually, the overview contains meta information for each page, such as title, author, size, summary. It is almost always impractical to use static pages to present such search result overviews. They call for dynamic page generation.

1.1.3. Interfacing with external systems

Suppose, you run an E-shop that indicates for each article whether it is stocked. Or you present the latest stock market values for your visitors or the latest news items. In each of these (and many other) cases, you present information that is maintained and changed by external systems, independent of your web environment. Your only chance to present up to date information is to ask the leading system for this information and build the web page dynamically.

1.1.4. Interaction

Suppose, you run a news site where your visitors can post relevant news articles. Your visitors will surely expect that they can see their articles soon after posting. This is best accomplished through a dynamic generation of the current news list.

For a high quality news service some delay may be acceptable. In fact, it is quite normal, as the contributed articles are often reviewed before publication. In such a case, the news service could be build as a static web site, too. However, a dynamic page based news service can easily be used as a discussion and collaboration forum. Longer delays between post and publication hamper such applications considerable. Dynamic page generation becomes a requirement.

1.1.5. Personalization

Visitors of a large web site may estimate high when they can personalize their view of the site: what content should be easily reachable, what content is relevant and what should not be shown at all, which layout and color scheme should be used, etc. My Yahoo! is a good example for the personalization possibilities. Personalized web sites require dynamic page generation as the visitor needs immediate feedback to his decisions to feel comfortable.

1.1.6. Push services

A site may wish to offer interesting push services. This usually requires some kind of personalization as pushing information may be felt as privacy violation when the information is unwanted. Therefore, the visitor must explicitly express his desire to receive the information. Often, I expect a wish to tailor the pushed information, e.g. with respect to the content and pushing frequency. Although the push services themselves do not require dynamic web page generation, the customization and tailoring will. And, of course, the pushed information will probably need to be dynamically generated. You may be lucky and can use the same tools and techniques for the dynamic push service generation that you use for the dynamic web page generation.

1.1.7. Summary

Let's summarize: you probably will think of a dynamic web site, when

1.2.1. Powerful composition engine

We have seen in the preceeding section that a dynamic web site is essentially characterize by dynamically generated web pages. These web pages are composed out of components during the web request. It is natural that good support for this composition, a powerful composition engine, is essential for dynamic web page creation.

What do we need? Of course, it is necessary that the engine provides support for the inclusion of components into the composed web page. In order to be able to adapt to the context, a conditional generation of parts of the page is required. It allows to test conditions and decide based on the outcome whether or not to put some material into the generated text. Many encountered structures are iterative: sequences or lists, e.g. the results of a search or the options of a selection. The composition engine should provide iteration over such structures. Of course, the composition engine needs access to all information in the web request, the URL, the query parameters, the request body, and the cookies. All this information must be usable both in conditions and for inclusion in the generated site. Likewise, the engine must have access to configuration data to improve reusability and adaptability. A web request is answered by a web response. In fact, the engine is generating this response. For a normal response, the response body contains the generated web page. But there are exceptional responses, e.g. for error reports, for optimization, configuration or adaption. The composition engine, therefore, does not only need access to the request information, but must also be able to control the generated response. An abstraction facility is essential to prevent tedious repetition of primitive steps. The abstraction allows to partially assemble components into larger pre-assembled components that themselves can be used to either build the final page or even larger components. Such pre-assembled components are identified by name (as are all components). To increase reusability and adaptability, components whether elementary or pre-assembled should be parameterizable and customizable. They resemble procedures or functions in programming languages; they are templates themselves.

There are two broad classes of composition engines. The first class uses programs or functions described in a programming language as the components for dynamic page generation. The generation process can use all facilities, libraries and modules available for the language. The actual content is either embedded as constants in the programs or read form external sources, such as files. The second class uses page templates. A page template is essentially the page to be generated. It contains its structure and its actual content. However, in addition, there are embedded commands to the composition engine. During page construction, called rendering, the composition engine simply passes the literal parts of the template on to the generated page. However, it processes its commands to generate additional content. When you look at a page description, then, for a composition engine of the first kind, you will see essentially a program with intermingled page content but, for an engine of the second kind, you see a page with intermingled programming constructs.

1.2.2. High level web publishing framework

In the introductory comments I told you that the web publishing protocol, i.e. HTTP (HyperText Transfer Protocol), is a very simple, a basic protocol, and that this simplicity was key to its success. It is very easy to implement and it is not difficult to use, but its use is sometimes tedious. Today, you probably will not want to use neither HTTP nor CGI directly for the web page generation. CGI (Common Gateway Interface) was the first interface for dynamic web page generation. It separates the web server with its concerns for HTTP from the proper dynamic web page generation. CGI, too, is quite basic, easy to implement but sometimes tedious to use. Its main target have been command line interpreter scripts. Due to restrictions for command lines at the time of CGI definition, there is a split between information passed on the command line and that passed in the request body. Essentially the same information is sometimes passed as command line parameter and sometimes as part of the body. Command line parameters are encoded, request body parameters may be but need not be encoded. A good dynamic web publishing tool should hide such differences from you. It should be a high level framework that allows you to concentrate on page generation and hides the details of CGI and HTTP. All request information should be presented uniformly, whether they have been passed via the command line or in the request body, whether or not they have been encoded. The framework should activate the composition engine for the correct page and present the request information in a uniform, unencoded and easily accessible form. It should usually handle standard cases, such as Page not found errors, authentication and page protection automatically and transparently. It should, however, be possible to customize or take over these processes, if necessary.

1.2.3. Support for content, presentation and logic

We saw in the previous section that a separation of concerns is one of the key success factors to manage large numbers of entities. This separation facilitates modularization and reuse and thereby reduces the costs for creation and especially maintenance. The primary concerns for a dynamic web site are the content (what is presented), its presentation (how is it presented) and the logic to accept new content, combine existing content or trigger processes and control workflow. You will want support for each of these concerns: an easy way to author or acquire and integrate new content, facilities for a designer to create a uniform, consistent and pleasing presentation and support for programmers to implement the business logic, control workflows and interface with external components.

1.2.4. Integration with external systems

There is a growing tendency to use Inter/Intranet sites as portals to most information relevant to an enterprise or organization. Most of this information is not directly in the web publishing environment. Usually, it is in backend systems, such as external databases, ERP systems, data warehouses, customer relationship management systems, production systems or even the Internet. Dynamic web sites are especially suited for such portals, as they allow the presentation of up to date information. This means, however, they need to interface with these systems. Each may call for special protocols or interfacing strategies. Your dynamic web site toolkit should include support for the most essential external systems and be flexible enough to add interfacing components for other systems on a demand base. The same integration and flexibility demand is required for other types of applications, e.g. E-commerce.

1.2.5. Transaction support

A high end dynamic web site is a complex application where many users may concurrently access the same data and where many potential causes for problems exist. Some of these events may cause a request to be aborted. Without special precautions such aborts may result in data inconsistencies. Usually, transaction systems are used to prevent such inconsistencies. They ensure, that a sequence of operations protected by a transaction is either seen not at all or completed. In case of a problem, the transaction is aborted: all operation effects so far are automatically undone. Transactions also provide support to detect and avoid problems caused by concurrent access to the same data. Your dynamic web site toolkit should support transactions associated with web requests when your site allows modifications of potentially sensible data.

1.2.6. Security

A dynamic web site, especially one that supports strong kinds of interaction and flexibility, is much more vulnerable to attacks than a static site. When a system only supports fetching of web pages, then the chance is quite low for a hole allowing unauthorized actions. When the system, however, provides for content management through the web, then there might be flaws in the systems itself or (more likely) in the security configuration that allow modifications for unauthorized users. Therefore, build in security with a concise, clear and easy to manage security policy is essential for a dynamic web site toolkit.

1.2.7. Scalability

If you have build a good, attractive web site, the visitor number may grow rapidly. Then, it should be possible, to scale your system without a complete redesign or even the need to switch to a new tool kit. There should be few or no platform dependencies such that you can switch to newer and faster computers or more powerful operating systems. If this is not enough, it should be possible to spread the application over several separated host systems. This, too, will increase availability and fault tolerance.

1.2.8. Automation support

The dynamic web site should provide support for automation. Tasks to be automated include backup creation, database reorganization, removal of old and no longer relevant content, archival.

There is a different aspect to automation. The time when the primary use of the Internet is by visitors accessing your site by browsers may soon be over. New interaction modes where Internet sites automatically exchange information and initiate business transactions are being developed such as e.g. automated procurement systems. In such systems most information exchanges are automated via the Internet: product information (catalogs) are transfered from provider to customer. For example, if the stock for an article falls below a threshold, the customer's procurement system checks with the providers for current price and availability of the article. It sends an order, automatically processed at the providers site. The order activates a delivery workflow case at the provider. During the process, an automatic order confirmation is sent to the customer via Internet and so on. Automation via the Internet will become much more widespread than it is today. XML based formats will be used as platform independent interchange formats for this new kind of web interaction. To be open for this development, you may want to look for fine grained automation via the web and a strong XML support in your dynamic web site toolkit.

2.2.1. ZServer

The ZServer is a multi-threaded web server. It can serve several web requests concurrently and supports the Internet protocols HTTP (HyperText Transfer Protocol, the web's standard protocol), FTP (File Transfer Protocol) and PCGI (Persistent CGI). In fact, it is itself a framework^[1] into which the ZPublisher plugs in. ZServer provides threads, buffers and a standard web logging facility. It receives the requests from the Internet, preprocesses them according to the CGI standard and then calls out for its application, the ZPublisher. Later, ZServer streams the response delivered by the ZPublisher to the requester on the Internet.

2.2.2. ZPublisher

The ZPublisher locates the resource addressed by the request's URL (Uniform Resource Locator). This resource may belong to the Zope framework or to the application. From ZPublisher's point of view, it is a hook that gets called. The hook function may have parameters. ZPublisher calls the function with the available parameter values. It takes them either from the request or the resource's context. It even tries to perform type conversions as only strings can be passed via HTTP but the function may need a different type. In order to provide easy access to all request information (arguments, header fields, cookies), the ZPublisher builds a request object. It further builds a response object that will later control ZPublisher's response creation. Both objects are made available for the called function (if it is interested in these objects). When the function returns, ZPublisher builds a response from the function result and hands it over to ZServer for delivery. This response building is guided by the current state of the response object which might have been modified by the function call. Usually, ZPublisher handles any errors during the publication automatically and generates an appropriate error response for them.

2.2.3. ZODB

Unlike most of its competitors, Zope does not store the web site's components in the file system. Rather, it maintains them in the ZODB (Z Object DataBase), an object oriented database. What's the advantage? In the file system, there are just two types of objects: folders and plain files. The file system defines what properties and methods these objects have. You need much more object types for your web site components: documents, images, files, folders, presentation objects, business logic objects, database queries etc. If they are all in the file system, somehow these types must be mapped to the available types. Usually, naming conventions (you surely know the use of the filename extension to indicate a content type) or location conventions are used for this mapping. The methods used to process the objects can not be maintained with the objects but must be coded inside the web application framework. This makes maintenance and extensions more difficult. It is even more difficult to add additional properties for the object types. In Zope, each of the required object types is implemented by a class. It is easy to give the class all necessary properties and define methods as needed. When an application needs specialized objects, new classes can be defined for them. Inheritance, a powerful object oriented concept, can be used to define new classes from existing base classes. The new class inherits all capabilities from its base classes. Its objects can be used wherever an object of one of its base classes could be used. Maintenance and extensibility become much easier.

The ZODB has further advantages: it is a transactional database. Modifications to the ZODB can be associated with a transaction. In case of a problem, all modifications associated with a partial transaction can be undone without compromising the integrity of the ZODB. ZODB detects when two transactions are in conflict with one another. In such a (rare) case, ZODB aborts and undoes one of the conflicting transactions and ensures its integrity. In fact, the ZODB supports two types of transactions: short living transactions used to protect the operations in a single web request and long living transactions, called versions, to support concurrent development.

The ZODB itself is a sub-framework which uses a storage module to store its content. There are several storage module types (more can easily be implemented):

2.2.4. Content

Content is the most essential part of a Web site. Of course, Zope provides object classes to contain this content: files, images and templates. It also provides folders to organize the Web site into a hierarchical structure.

2.2.5. Templates

Document Template is currently Zope's main composition engine. As the name suggests, it is template based, i.e. the page specification looks essentially like a static page. Usually, it has however embedded commands to be processed during page generation. These commands normally generate additional text that is included in the page as a substitution for the command. But there are also commands that suppress page content from being included. As HTML pages are the main target, Document Template uses an HTML like syntax: new elements and specially named entities are used to embed composition commands. The set of commands is called DTML (Document Template Markup Language). All commands are identified by a dtml- prefix.

<dtml-var header>
<h2>Simple DTML example</h2>

<p>This simple example demonstrates the embedding of
composition commands in HTML content.
The dtml-var command is used to dynamically include
the "content" of the objects header and footer at the top
and bottom of the rendered page, respectively.
A page's top and bottom parts usually contain essential HTML
elements, as e.g. the page title and style definitions,
the primary navigational elements as well as the main corporate
identity elements, e.g. logo's and images.
Localizing these elements in the header and
footer objects provides for a consistent
structure and presentation of all pages that include them.</p>

<p>The dtml-var command is similar to a macro inclusion mechanism.
However, what is included, is not the body of a macro definition
but the "content" of an object. The object determines
what its "content" is. For simple objects, such
as properties, it is just their value converted to a string.
Callable objects are called to determine their value. They can use
acquisition  to return a context dependent result.
If the object is a DTML object, then a name space argument is
automatically passed that gives the called object full information
about the callers context.</p>

<dtml-var footer>

Document Template contains commands for

When you look at the example above (and you know HTML), then you will recognize that the DTML example does not look like a complete HTML page. This is because large parts of the page are generated by the header and footer inclusions. Extracting these parts has the advantage (besides saving you typing effort) to provide for a consistent structure of your Web site. For example, the navigational elements would normally be implemented in the header and footer while the proper template is responsible for the content of this page. This is very nice, but it breaks with WYSIWYG HTML editors. They are unable to render the DTML commands. Therefor, the page they display looks completely different from that rendered by Zope. This is not good for designers. Furthermore, the editors see missing HTML components and will add them after modifications. This is not good for Zope, as then the elements inside the header and footer become duplicates. To facilitate work with advanced HTML editors, a new template engine is currently being developed for Zope: Zope Page Templates, or ZPT, for short. ZPT uses the XML namespace extension mechanism to add its composition commands as attributes to existing HTML (or XML) tags. This way, it is quite easily possible to design templates that look very similar in an HTML browser to their rendered version. In a ZPT predecessor, this visual similarity destroyed one major advantage of templates: that common parts can be extracted, maintained in a single place and used as components whereever necessary, as done in the header and footer components above. ZPT reinstates these advantages through parameterized macros. Any structurally valid part of a template can be defined as a macro by associating a macro name with it. Part of the macro's body can be defined as a parameter by giving it a slot name. Templates can use macros by referencing their macro name and provide new values for the slots, if necessary. This way, the common parts of many templates can be extracted as a macro, maintained at a single place and used as components by all the templates. If later something needs a change, just the macro definition is modified. Templates using the macro usually do not need to be touched for the modification to become effective. As HTML editors do not understand the macro mechanism, Zope performs macro expansion when the source of a page template is called for. ZPT will soon become Zope's primary composition engine and push away DTML. For new projects, you should go with ZPT.

2.2.6. Scripts

Zope provides various kinds of objects implemented in scripting languages such as Python or Perl. Their main purpose is the implementation of business logic: computing values, checking validity, enforcing rules. The different members in this category distinguish themselves by the implementation language and whether or not they can be edited through the Web. Through the Web editable objects, Scripts, are subject to Zope's access control mechanism. File system based objects, e.g. External Methods are allowed to do everything, as an intruder is expected not to be able to modify files in the filesystem.

Scripts can also be used as a type one composition engine. As they are accessible by an URL (like most Zope objects), they can be used immediately from the Web. Of course, they can be called from Document Template and can in turn use document templates to compute their content.

2.2.7. Database interface

Zope's interface to relational databases is provided essentially by database adapters and [Z] SQL methods.

Database adapters are the database type/vendor specific part of the interface. They handle and hide the differences in the call interface implementations of the various database client libraries, such that the Zope core can treat various database systems in a uniform way. There are database adapters for most major relational database systems and a generic database adaptor to access databases via ODBC.

A Zope web site is interfaced with a relational database by instantiating an appropriate database adapter object somewhere in the web site hierarchy. Such objects represent connections to a database. Their constructor asks for the necessary connection id. Database connections support execution of arbitrary SQL statements (understood by the database, of course) against the database and interactive exploration of the database's data model.

A SQL method is a Zope object that represents a template for an SQL command (in fact, it can be a sequence of SQL commands). It has an associated set of arguments and a command template. When called, the SQL method constructs an SQL command from the template, the passed in parameters and acquired context information and sends it via an associated database connection object to the database for execution. If the SQL command is a query, it returns the query result as a sequence object. This sequence object provides for easy access to each row and within each row to the field values.

SQL methods use Document Template for their template specification. In this context, special document template commands are available for e.g. type safe inclusion, appropriate quoting, conditional SQL operator generation.

SQL methods implement the Searchable Interface. This interface contains methods to determine the set of arguments (input parameters) and result columns (output parameters). The Z Search Interface wizard uses this interface to automatically generate a search interface. Such a search interface consists of an input form to get values for the arguments from the user and a report list to present the query results. The generated search interface can later be modified manually to fit the applications needs.

2.2.8. ZClass

Applications often consist of several related objects, e.g. an input form, an action, an SQL Method, a confirmation and an error page. If you need the application just once in the Web site, such a set of related objects is easily implemented inside a folder. If you need it several times, the folder based solution rapidly becomes unfeasible, as it would introduce lots of redundancies. If you want to change your application, you must visit each such folder and apply your changes. That's where ZClasses come in. From the user's point of view, a ZClass is a container for properties and methods. Properties define the state for the class' instances, the methods their behavior. The methds are arbitrary (site building) Zope objects. Thus, it is quite easy to lift a folder based application into a ZClass: just make the related objects methods of the class. Then, whenever, you need an instance of your application, you instantiate your ZClass. If your application should change, you change your ZClass, all its instances will follow automatically.

ZClasses can be easily constructed through Zope management interface, i.e. through the Web. They provide an essential tool for packaging and customization.

2.2.9. Python based products

When new functionality becomes necessary that can not (easily) be obtained through combination of existing Zope objects, then the development of a Python based product provides an alternative. Such a product is implemented in Python, can use the extensive Python library and all of Zope's infrastructure to implement new site building classes. One integrated into Zope by placing the product into Zope's Products folder, instances of the defined class can be instantiated anywhere inside the Web site where their services become necessary. Products can do almost everything. Zope.org contains more than 400 contributed products for all kinds of tasks: e.g. user management, content objects, presentation, integration, feedback etc..

2.2.10. Application

A Zope application consists of a set of interrelated objects organized in a hierarchical structure. Each object is an instance of an object class. Some classes are part of the Zope core, e.g. Folder, DTML Documents, Z SQL method, database adapters. Other classes are defined in components that are installed separately, so called products. Then, there are classes defined by the application itself. These classes are objects at the same time, instances of the class ZClass.

A typical Zope application is build via Zope's Web management interface by instantiating objects at appropriate places in the hierarchical structure. The management screen for a folder contains a select box with all object classes, that are available for the current user in this folder. When the user selects a class, its constructor is called. It asks for all necessary parameters to instantiate the object and adds it to the folder.

2.2.11. Extensions

Zope is a very flexible and easily extensible framework. Extensions are possible on different levels: by defining ZClasses, through external methods and through Python based products.

A ZClass is a class that can be built through Zope's web management system. A ZClass has an associated set of properties, organized in property sheets. They contain the class instances' state. The ZClass can define default values for all properties. The class also has a set of associated methods. These are arbitrary other Zope objects, e.g. Document Templates, SQL methods, external methods. ZClass instances can use all methods defined by their class. They have its class' properties but can give them new values. If the class is changed, all its instances can immediately use the changes. Inheritance, indeed even multiple inheritance, can be used when a ZClass is being defined. Parent classes can be Zope core classes, classes defined by products provided they have been registered as possible base classes and other ZClasses. As you may have recognized, a ZClass is more a packaging than an extension mechanism.

An external method is a Python function defined in a file outside Zope in the file system. They are used to provide functionality that has not yet been build into Zope or that is Zope functionality but is dangerous to be made available in general for security reasons (e.g. reading and writing files). External methods can not be changed through Zope's web management interface.

A Python based product usually consists a one or more classes. Frequently, the product package includes document templates, icons and further elements for the various web based user interfaces. After a product has been extracted in the Products package, the product's classes become available to create new objects in the web site.