Subject Author Date
Roxen Webserver 5.0.381 available Henrik_Grubbström <grubba[at]roxen[dot]com> 02-06-2009
Roxen WebServer 5.0.381 is now available from

Note: This is a major release, and as such there are more structural
       changes than usual which have larger effects on compatibility.
       Wherever feasible, compatibility is kept through the compatibility
       level setting (found under the Settings tab for each site). There
       are however several incompatibilities that are not overcome that
       way. All compatibility and upgrade considerations are detailed in
       the sections labelled COMPAT NOTE below.

WebServer-specific changes

   Core improvements:

     o Moved to Pike version 7.8, which among many other things enables
       full use of 64-bit hardware on most OS:es. Roxen 5.0 does not run on
       any earlier Pike version.
       COMPAT NOTE: There are a number of incompatibilities between Pike
       7.4 and 7.8, but that is of no concern if you do not have your own
       custom modules. Roxen module developers should take a look at the
       Pike release notes for 7.6 and 7.8. Most incompatibilities are
       mitigated by running in 7.4 compatibility mode, which is enabled by
       simply putting "#pike 7.4" at the top of each pike (and pmod) file.
     o Moved to MySQL version 5.0. Roxen no longer works with an earlier
       version. By default it does not accept 5.1 or any later version
       either, since that combination is unsupported. That check can
       however be disabled with a define ALLOW_UNSUPPORTED_MYSQL.
       COMPAT NOTE: There are many MySQL compatibility considerations, but
       unless you have your own databases or tables in the Roxen MySQL you
       need not bother very much with them. Otherwise, please consult the
       upgrade notes in the MySQL manual for further details, for instance
       to get information on how to update your query syntax. Note that
       there might be actions you need to do on your MySQL data before
       On upgrade, Roxen automatically updates the privilege tables in
       MySQL to cover all new privileges, similar to what the mysql_upgrade
       script does. Read-only access in the DB interface now translates to
       Select_priv and Execute_priv, and read/write access gives
       Select_priv, Insert_priv, Update_priv, Delete_priv, Create_priv,
       Drop_priv, References_priv, Index_priv, Alter_priv,
       Create_tmp_table_priv, Lock_tables_priv, Create_view_priv,
       Show_view_priv, Create_routine_priv, Alter_routine_priv, and
     o It is now easier to make Roxen use another MySQL installation,
       either one separately installed or one that comes with the OS
       distribution. Paths to the MySQL installation directory and the most
       important executables are specified in the file mysql-location.txt
       in the Roxen server directory.
     o Databases: Introduced charset support for database connections. The
       server can now keep track of the charset a database uses for queries
       and returned text, and automatically do conversions to the Unicode
       representation used internally in RXML etc. This currently works for
       MySQL and Postgres (using the new pgsql driver). (Later 4.5 releases
       also had this feature to a limited degree.)
     o Databases: Fixed security issues where arbitrary databases in the
       Roxen server could be reached through the SQL tags. The SQL tags
       module now has a setting Allowed databases that specifies which
       databases may be accessed through the SQL tags. This setting is also
       used by some other modules that access databases, e.g. <emit
       source="timerange" query="...">.
       COMPAT NOTE: Database access is disabled by default, meaning that
       essentially all SQL tags that don't use the default database will
       not work until the Allowed databases setting has been configured to
       local needs. The module logs connection attempts to blocked
       databases in the event log, so it is possible to see that way which
       ones that need to be let through.
     o Added new start up script for Roxen CMS on OS X. Handles start, stop
       and restart.
     o IPv6 support. Roxen can now bind IPv6 ports, initiate connections to
       other IPv6 servers, etcetera. (4.5 also had this to a limited
       degree, if run on Pike 7.6 instead of the included 7.4.)
     o SNMP (Simple Network Management Protocol) support. This allows
       monitoring of the Roxen server through SNMP v1 or v2c (but not v3).
       It is enabled simply by registrering another port with protocol snmp
       for a site configuration. Each site has its own SNMP prefix, so
       several sites can share the same SMTP port.
     o Removed the old argcache system so that the new one is used by
       default (i.e. ENABLE_NEW_ARGCACHE is no longer necessary). This
       primarily means that links to auto-generated images are handled in a
       more robust way, especially in replicated setups.
       COMPAT NOTE: There is no compatibility fallback to read argcache
       entries from the old database. An upgrade instead depends on that
       the image cache remains intact for long enough so that old images
       can be served without requiring their argcache entries.
       COMPAT NOTE: The old arguments table in the local database is not
       dropped automatically. Administrators are adviced to do that to free
       up space.
     o The server now automatically runs a consistency check of the MySQL
       tables at startup.
     o Experimental support for gzip compression of http responses. Enabled
       by the define HTTP_COMPRESSION.
     o The protocol cache can now continue to deliver stale cache entries
       while new results are generated in the background. This can mitigate
       very long response times and server choking when a cache further
       back is being invalidated (typically the persistent disk cache in
       the CMS). (This was also available as an experimental feature in
       4.5rel4 when ENABLE_SPCI was defined.)
     o Handling of incoming and outgoing charsets has been improved and
       primarily works better in UTF-8 mode. UTF-8 is also the default
       charset for form and query variables, with fallback to ISO-8859-1 if
       UTF-8 decode fails. That should lessen the need for
       <roxen-automatic-charset-variable> in forms (although it still is
       useful to make forms work better for users with really old
     o The size calculations for entries in the various RAM caches (seen
       under Tasks/Status/Cache status in the Administration Interface) are
       now much more accurate.
     o Added support for software packages. This is a system to be able to
       install complex module distributions without putting everything in
       the Roxen module path.
     o COMPAT NOTE: Roxen is no longer distributed with its own copies of
       fonts that may override bitmap versions shipped to customers, since
       the FreeType renderings are different enough to affect spacing etc.
     o COMPAT NOTE: If a port is opened for more than one server
       configuration, the fallback in case there is no site with the
       "Default site" flag set in the server has changed slightly: Now the
       configuration with the least specific port URL is used, while in
       earlier versions the most specific port URL was chosen. This is a
       fringe case that should only affect badly configured servers.

   Administration interface improvements:

     o New patch management system, usable both to apply patches from Roxen
       Internet Software and to manage your own local changes. It is
       accessible from Tasks/Maintenance/Patch management, and there is
       also a command line tool bin/rxnpatch in the server directory.
     o The database browser under the DBs tab has been overhauled and is
       now much faster and has more features. Among other things there are
       tools to optimize or repair tables, and the permission matrix is on
       a separate subtab.
     o The Tasks/Debug Information/Resolve Path dialogue can now send form
       variables and cookies to simulate sessions and to aid in debugging
       input forms. The logging has also been extended to provide more
       details from various facilities.
     o Added a debug tool that dumps all threads whenever any handler
       thread or background job has been running for a set number of
       seconds. It's configurable from Globals/Logging and can be disabled
       with the define NO_SLOW_REQ_BT.
     o Support IDN hostnames for port bindings.
     o Added experimental support for scheduled database backups. It is
       only enabled if ENABLE_DB_BACKUPS is defined, and it then adds a new
       tag DBs/Backup schedules.

   RXML core improvements:

     o The RXML type system has been extended with array and mapping types,
       and it is now fully deployed in various RXML tags through type
       attributes and type context sensitivity. See the new RXML Type
       System chapter in the Web Developer manual for all the details.
     o Cleaned up handling of the RXML nil value (RXML.nil):
          1. Do not allow an RXML variable to be set to RXML.nil. That
             deletes the variable instead.
          2. <emit source="sql" ...> maps SQL NULL to a null value instead
             of RXML.nil.
          3. <if variable=""> is false both for undefined variables
             (i.e. RXML.nil) and the SQL null values.
          4. A new test <if variable-exists=""> is added to test
             whether a variable is defined or not, i.e. it is true for null
             values but not for undefined variables.
          5. <if sizeof=...> evaluates to zero both for undefined and null
             variables, for compatibility.
          6. <copy-scope> no longer copies undefined variables.
          7. <emit source="values"> and <insert> sources variables and
             scopes no longer list undefined variables.
       COMPAT NOTE: Items 2, 6 and 7 are only activated if the compat level
       is 5.0 or higher (items 3 and 5 don't affect compatibility since
       null values didn't exist earlier).
     o Added an <emit> attribute filter-exclude which does the opposite of
     o Added encodings utf16, utf16be, utf16le and hex which can be used in
       the encoding spec for variable entities, e.g. &var.x:hex;.
     o Accessing the cookie scope no longer implicitly disables the
       protocol cache. The protocol cache instead starts to vary on the
       cookie value.
       COMPAT NOTE: The old behavior meant that any RXML accessing cookies
       implicitly disabled the protocol cache. That means the new behavior
       can introduce overcaching side-effects in old code, even if the
       protocol cache entries are cookie specific. Therefore the
       cache-disabling behavior is kept if the compat level is 4.5 or
     o The truth value (page.last-true) is now always set to false when an
       RXML error is caught.
       COMPAT NOTE: This is a change that might have compat implications.
       It's only enabled on compat level 5.0.
     o The formatting of the online tag documentation has been improved and
       shows the structure more clearly.

   RXML tag improvements:

     o Added a <value> tag to do type casting and to build compound values
       like arrays and mappings.
     o Added a <substring> tag that can pick out parts of strings in a
       number of ways.
     o Added a <range> tag which is similar to <substring> but operates on
       arrays instead.
     o <insert source="variables"> is made context sensitive so that it
       returns the scope mapping as-is in an array or mapping context.
     o The type handling in the <set> and <append> tags has been cleaned
       COMPAT NOTE: These changes have compatibility effects in some cases.
       Their old behavior is retained on the 4.5 compat level.
       In particular, the <append> tag behaves differently with arrays now.
       If you get errors in code that appends string elements to arrays,
       then you probably want to add type="text/*" to the <append> tag.
     o Extended the <set expr=...> expressions to allow various set
       operations on arrays and mappings. The docs for it is also adequate
     o SQL tags: The charset handling has been changed to fit the design
       principle that strings are always unencoded Unicode internally in
       RXML. It therefore not only controls the connection charset, but
       perhaps more importantly it also encodes queries and decodes results
       with that charset. That makes it useful also with databases without
       charset support in the client library.
       COMPAT NOTE: This change of the charset handling is incompatible.
       The old behavior is retained on the 4.5 compat level.
     o Additional RXML tags: Added two tags <dirname> and <basename> to
       pick out the respective parts of a path.
     o Additional RXML tags: Added an <xml-rpc-call> tag to make simple
       synchronous xml-rpc calls. It is enabled by the same option that
       enables synchronous <insert href>.
     o Additional RXML tags: Made it possible to pass data content in
       <insert href> POST requests.
     o Image converter and GXML modules: A filename attribute has been
       added to the <cimg> and GXML tags to append a filename to the
       autogenerated URL. The GXML module has also been blessed with the
       option to add image extensions to the URLs, like <cimg> already
       could do.
     o Graphics tags: Several of the attributes that these tags accept
       refer to files that are used for purposes such as textures or
       backgrounds. We now properly register timestamps to detect changes
       for some attributes that weren't handled earlier. If any of these
       resources are protected the RXML parser will now answer with a HTTP
       Auth Required response to the browser to force authentication
       instead of generating and caching an incorrect image.
     o Business graphics: Introduced a color-scheme attribute to the
       <diagram> tag to autogenerate colors for data.
     o E-mail module: Improved error handling in the <email> tag and added
       a new attribute error-variable. Also added an attribute
       envelope-from to set the envelope sender address.
     o Added type attribute to <redirect> to make it possible to do e.g.
       permanent redirects.
     o Added attribute http-time to <date>.
     o Added new mode safe-utf8 to <recode from>. When provided, silently
       ignore any illegal UTF-8 sequences.
     o Added a showvar attribute to the <debug> tag to be able to print out
       the value of a variable without conversion in an unambiguous format
       (useful to figure out charset conversion issues, for instance). The
       <debug> tag also works in any type context.
     o COMPAT NOTE: A bug has been fixed in the <contents> tag, used within
       <define>, when both the value-of and result-set attributes were used
       at the same time. The fix might have compatibility implications in
       code that tried to work around the bug, but the old behavior was too
       quirky to keep compatibility with on the 4.5 compat level.
     o COMPAT NOTE: A type problem that could cause extra entity quoting in
       <strlen> has been fixed and hence wrong length reports. Old code
       might possibly depend on the broken behavior, so it is kept on 4.5
       compat level.

   Module improvements:

     o The Yahoo! User Interface Library: This is a new module can be used
       to make the YUI available to web applications on the site. It
       supports several YUI versions simultaneously and it is easy to add
       another by downloading the tar distribution and letting the module
       unpack it.
     o Path info support: Added path limit option.
     o Redirect module: Added permanent keyword to send permanent (301)
       redirects instead of temporary (302).
     o RXML parser module: Added an option to censor potentially sensitive
       user authentication data from requests before RXML evaluation
     o CGI interface now exports REQUEST_URI, REDIRECT_URL and
     o CGI scripting support: Added a setting to be able to run CGI scripts
       in a chrooted environment.
     o CGI scripting support: Added a setting to disable the Roxen extended
       environment variables.
     o Javascript support: Don't output empty <script> tags. Marginally
       smarter quoting.

   Minor improvements:

     o Administration interface: Updated the default logging format to the
       Combined Log Format which extends Common Log Format with two fields.
       The proposed format with extended usage info is changed accordingly.
       It also uses $ip-number instead of $host by default, to avoid the
       DNS overhead.
     o Administration interface: The Add modules page now shows the site
       name, to avoid adding modules to the wrong site by mistake.
     o Administration interface: A bit nicer sort order for threads in
       Tasks/Debug Information/Thread backtrace, e.g. the backend thread is
       always listed first.
     o Administration interface: Improved display of Unix sockets in
       Tasks/Debug Information/Open files.
     o Administration interface: The binary distribution identifier is now
       shown at the bottom of the pages.
     o Core: Optimizations in the protocol cache, and in the vary callback
       system in particular.
     o Core: The image cache now allows protocol caching of authenticated
       images, which is made possible by the new arg cache implementation.
     o Core: Speed up accepting of connections.
     o Core: Improved the p-code codec to handle references to arbitrary
       Pike modules.
     o Core: The sample start script in the tools directory now has a
       "status" command to query whether the Roxen instance is running or
       not. It can also be configured to handle several instances by
       keeping their configuration directories in a special "configuration
       collection" directory.
     o Core: Added an experimental mode where the RAM cache retention
       policy is based on the time to create the entries. This is enabled
       by the define TIME_BASED_CACHE.
     o Core: Log a warning if a background job takes more than one minute.
     o Core: Roxen modules are now always loaded in the same order.
     o Core: There is now a bat file bin\mysql_client_nt.bat to make it
       simpler to start a MySQL command line client against Roxen's MySQL
       process on Windows.
     o RXML tags: The <nocache> tag now properly disables protocol and
       client caching, just like the <cache> tag does by default. It has
       also gotten the attributes enable-client-cache and
       enable-protocol-cache that works like in the <cache> tag, to control
       this behavior.
     o RXML tags: <remove-cookie> no longer requires the cookie to exist.


Henrik Grubbström					<grubba[at]>
Roxen Internet Software AB