Sami Dalouche

Debian/Ubuntu sympa packages rely on apache1, so one may have problems running sympa on Ubuntu Edgy, for instance.

This simple HOWTO explains how to configure Sympa and WWS to run with Apache2 and FastCGI. FastCGI is used instead of CGI because of the performance impact it has. (Basically, CGI forks a new instance of the CGI script wheareas a FastCGI-enabled script runs as a daemon to handle requests, much like a decent system like Java Servlet API.)
Installation

Install the sympa package, and copy the installed apache1 configuration files to apache2 folder

# apt-get install sympa libapache2-mod-fastcgi

# ln -s /etc/sympa/httpd.conf-fcgi /etc/apache2/conf.d/sympa-fcgi

# dpkg-reconfigure -plow sympa

A wizzard will come up, make sure to :

• Use a database if you want to use WWS
• Select “Other” when asked what type of web server you are running
• Tell the wizzard that you want FastCGI enabled

Once this is done, check the /etc/sympa/wwsympa.conf file, and make sure

use_fast_cgi is set to 1

and the last step is to restart apache :

/etc/init.d/apache2 restart

Now, my personal thoughts about mailing-list systems. My impression is that there is no perfect Open Source mailing-list system (however, there are several ones that “do the job”). The characteristics of a good mailing-list system would be, to my opinion :

• Be entirely configurable through a Web UI. Sympa does a pretty good job at this, since most of the settings are tweakable from WWS. However, the UI is pretty ugly (sure, one can tweak the templates, but…)
• Would provide a Mailing-List system as well as a Web forum. To make it simple, something like Google Groups, that also allows to post (and subscribe/unsubscribe orders) via email. It is then up to each subscriber to choose between email and Web.
• Re-uses a well-known templating system ( Smarty in the PHP world, or Freemarker in the Java world, …). Why re-inventing yet another templating language if very good ones already exist ? Having to learn a templating language per web-application is pretty much cumbersome.
• Be extensible : It should provide a decent plugin system, that people can use to provide additional modules without touching a single line of the current base code. Requiring the modification of a 9980-line (cat /usr/lib/cgi-bin/sympa/wwsympa.fcgi | wc -l) perl script is, In my humble opinion, very bad practice. This implies coding against interfaces, and using some kind of IoC framework like the Spring Framework.
• Would be independant of the persistence layer, through the use of a sophisticated persistence engine such as Hibernate. Anyone can then configure it to use his preferred database engine
• Authentication would also be extensible. A security framework such as Acegi could be used so that anyone can easily have the mailing list system authenticate the users against the configured authentication backend. (be it database, system/PAM, Single-Sign-On, etc..). It doesn’t make sense to re-implement every authentication backend in every webapp, since some frameworks already do the job.

Prerequisite : You have Xen running correctly for less than 3 virtual interfaces. This HOWTO explains how to get it to work on Ubuntu Edgy.

Xen does not support using more than 3 virtual interfaces on the guest machines (the so-called DomU). It is stated in the Xen FAQ, and attempting to use more results  in what Ernie Fontes experienced in this post.
The usual trick for having more interfaces in a stand-alone system is to use IP Aliasing. Ubuntu Linux, among others, support IP Aliasing without any problem. However, IP Aliasing seems not to work (according to my tests) for a Xen DomU.

For some reason that I cannot explain nor understand, there is still a way to use more than 3 virtual interfaces in a Xen DomU by using the offical Xen way of adding interfaces, and by using IP Aliasing on top of that. It is weird, but it works :

• using Xen virtual interfaces is limited to 3 interfaces.
• using IP aliasing interfaces makes interfaces that are not pingable from the outside
• BUT using exactly 3 Xen virtual interfaces, and adding more interfaces thanks to IP Aliasing works beautifully….

Here is a quick HOWTO explaining this procedure :

Xen Configuration

Add 3 interfaces for the DomU (you might replace xenbr1 by xenbr0 if your bridge name is the standard one) :

vif = [ 'bridge=xenbr1','bridge=xenbr1','bridge=xenbr1' ]

And you can create your domain using the usual xm create command.

xm create config.cfg

xm console

DomU Network configuration

You can now configure your domain interfaces and the aliases. For the sake of giving a complete example, here is how to achieve that under Debian/Ubuntu :

Let’s say that we want to configure the 216.240.153.78, 216.240.138.247, 216.240.146.76 IPs for, respectively, eth0, eth1 and eth2, and 216.240.134.6 as well as 216.240.128.182 for the 2 aliases eth0:0 and eth0:1.

/etc/network/interfaces

auto eth0
iface eth0 inet static
address 216.240.153.78
netmask 255.255.255.0
gateway 216.240.153.1

auto eth1
iface eth1 inet static
address 216.240.138.247
netmask 255.255.255.0
gateway 216.240.138.1

auto eth2
iface eth2 inet static
address 216.240.146.76
netmask 255.255.255.0
gateway 216.240.146.1

auto eth0:0
iface eth0:0 inet static
address 216.240.134.6
netmask 255.255.255.0
gateway 216.240.134.1

auto eth0:1
iface eth0:1 inet static
address 216.240.128.182
netmask 255.255.255.0
gateway 216.240.128.1

You can now tell the system to reconfigure the network (/etc/init.d/networking restart), and if it still doesn’t work (especially for the aliases), you can restart the DomU (xm reboot domU-name).

Now, if someone has an idea of why this tip works, I am really really interested to know. Because right now, it looks like magical stuff that I’m not even sure of how I discovered

If you have recently migrated your AWstats to a different Architecture (Pentium4 to AMD64, for instance), awstats may report you the following error :

Warning: Error while retrieving hashfile: Byte order is not compatible at ../../lib/Storable.pm (autosplit into ../../lib/auto/Storable/_retrieve.al) line 331, at (eval line 1

This is caused by the fact that awstats/perl caches DNS Entries in a machine-dependant way. Others have experienced the same problems, like on this forum.

The solution is actually quick. Let’s say your stats are in /stats, you can do :

find /stats -name ‘*.hash’ -exec rm {} \;

Here is a simple HOWTO explaining how to configure AWstats to analyze Apache2 logs, and provide detailed statistics, under Ubuntu Edgy Eft. This should also work for other Ubuntu versions, as well as any Debian derivative.

Apache

The first step is to activate Logging in Apache, so that Awstats has something to analyze. For instance, you can add something similar in your VirtualHost configuration :

ErrorLog /var/log/apache2/sirika.com-error.log
CustomLog /var/log/apache2/sirika.com-access.log combined

Another important thing is to configure a few things for awstats in apache, like where the icons are, and more importantly, to activate CGI-scripts (since AWstats is written in perl…) . This can be done thanks to the following /etc/apache2/conf/awstats.conf :

# This provides worldwide access to everything below the directory
# Security concerns:
# * Raw log processing data is accessible too for everyone
# * The directory is by default writable by the httpd daemon, so if
# any PHP, CGI or other script can be tricked into copying or
# symlinking stuff here, you have a looking glass into your server,
# and if stuff can be uploaded to here, you have a public warez site!

Options None
AllowOverride None
Order allow,deny
Allow from all
# This provides worldwide access to everything below the directory
# Security concerns: none known

Options None
AllowOverride None
Order allow,deny
Allow from all

# This provides worldwide access to everything in the directory
# Security concerns: none known
Alias /awstats-icon/ /usr/share/awstats/icon/

# This (hopefully) enables _all_ CGI scripts in the default directory
# Security concerns: Are you sure _all_ CGI scripts are safe?
ScriptAlias /cgi-bin/ /usr/lib/cgi-bin/

Awstats

The next step is to install awstats, with all the necessary perl modules.. Using several optional modules implies that you have installed them. liburi-perl is useful if you use the “decodeutfkeys” module, even though it is NOT listed as recommended or suggested in the awstats package.

sudo apt-get install awstats libnet-dns-perl libnet-ip-perl libgeo-ipfree-perl liburi-perl libnet-xwhois-perl

Once this is working, it is now necessary to configure awstats, to tell it which logs it should monitor, and where it should write its working files (stats). This is done by creating a file /etc/awstats/awstats.website.conf (replace website by your apache2 virtual host name, for instance, and do NOT forget the .conf !).

sudo cp /etc/awstats/awstats.conf /etc/awstats/awstats.website.conf

Editing this file should be pretty straightforward, since it is well commented. In particular, pay attention to the following entries

LogFile=”/var/log/apache2/sirika.com-access.log”

SiteDomain=”sirika.com”

HostAliases=”www.sirika.com”

DirData=”/srv/data/stats/sirika.com”

LogFile should point to the log file configured in Apache2.

SiteDomain is the main domain name, as configured in Apache2.

HostAliases should list ALL the aliases listed in Apache2’s VirtualHost configuration. Usually, you will want the same domain prefixed with www, (or without the www prefix if is it already specified in the main domain name). This is really annoying, error-prone and not having a global definition of a “virtual host” on the system is one of the issues I pointed in “10 things that still suck under linux“. Virtual hosts and aliases should be defined once, globally. Every single concept/thing should be repeated/configured once, and only once, on a perfect system. Anyways…

DirData should point to an empty directory, whose content will be managed by Awstats.
Cron

Everything is now configured. On a perfect system, the setup would stop now. Awstats could be automagically notified for every change in the logs (awstats could register to a so-called “http access event”, and its internal behaviour could define its policy for updating the stats (update synchronously, update asynchronously when the system is idle, etc..). This is one of the things I pointed out in “10 things that still suck under linux“. However, we’re not there yet, so, we need to run the script every day to update the stats. (yeah, that’s the reality today..)

So, this can be done thanks to the following /etc/cron.daily/awstats (do not forget to chmod +x /etc/cron.daily/awstats after creating it) :

#!/bin/sh

/usr/share/doc/awstats/examples/awstats_updateall.pl -awstatsprog=/usr/lib/cgi-bin/awstats.pl now > /dev/null

This will update all the statistics for all the hosts defined in /etc/awstats/*, on a daily basis. Yes, it’s not as beautiful as having a full-featured event-system for which every application could attach to events generated by others, but it has the merits of working…

Logrotate

What happens when your apache logs get rotated (and possibly gzipped, etc) by logrotate (apt-cache show logrotate for more information), and awstats still hasn’t analyzed the end of the logs that is about to be rotated ?

To avoid this situation, it is necessary to tell logrotate to launch awstats BEFORE rotating the logs. This can be done by adding the following lines to /etc/logrotate.d/apache2 :

prerotate
/etc/cron.daily/awstats
endscript

Permissions

And of course, permissions must be tweaked :

• Since Awstats runs as the web users for viewing stats (CGI-script), the web user needs read access to /srv/data/stats/*
• Additionnally, you may want to provide the “update now” button on your website stats. So, the web user also needs write access to /srv/data/stats/*
• Finally, awstats needs access to the apache2 logs to create the stats. This is not a problem when it is run from a cron script, since it is run as root. But, in the case of “update now”, it runs as the web server, so the web server needs read access to its logs. (the default permissions are 660 with root:adm), so www-data doesn’t have access to its logs

The problem with traditional permissions is that there is no decent way of specifying default permissions. So, we are going to use ACLs for that. You can find more information about them here (Using POSIX ACLs to complement traditional Linux permissions). So, this gives, for instance :

# read write execute access for web user to the stats directories
find /srv/data/stats -type d -exec setfacl -m “g:www-data:rwx” {} \;

# read write execute access for FUTURE stats files for the web users

find /srv/data/stats -type d -exec setfacl -d -m “g:www-data:rwx” {} \;

# read write access to the stats files for the web user
find /srv/data/stats -type f -exec setfacl -d -m “g:www-data:rw-” {} \;

# read only access to the logs directory for the web user
find /var/log/apache2 -type d -exec setfacl -m “g:www-data:r-x” {} \;

# read only access to the logs for the web user, for future files

find /var/log/apache2 -type d -exec setfacl -d -m “g:www-data:r-x” {} \;

# read only access to the apache2 logs for the web user
find /var/log/apache2 -type f -exec setfacl -m “g:www-data:r–” {} \;

And it should work.. The last thing would be to protect access to your logs, if you don’t want your users to see them. This can be done using a .htaccess file, and there are plenty of tutorials on the web that explain how to achieve that.