IP Failover

IP Failover (proof of concept)

Essentially the idea is that if your primary web server goes down, your backup server automatically takes over (failover). When your primary server is back online, it takes over again (failback). It sounds simple, but it can be very complicated and expensive to eliminate all the single points of failure – you could need multiple redundant routers, switches, servers, power supplies, UPS and storage all on separate networks.

The simple setup I am proposing uses a minimum of two servers; the primary and the backup, both on separate networks.

An alternative to IP failover (where the IP is changed) is IP takeover (where the backup server actually takes on the IP address of the failed primary), but for this to work, the IP addresses need to have the same router, so would need to be at the same web host. That’s fine if the primary server failure issue is local hardware, but if it’s a power failure in the datacentre or a problem with the network, a switch, a router or any connection problems, transit or peering, then both servers would have the same problem, so it makes more sense to make them geographically disparate on entirely separate networks.

It should be noted that IP failover services are provided by sites like dnsmadeeasy.com and zoneedit.com – this would be much simpler to set up, but of course you pay for the privilege.

There are two options for the backup server. Either it just displays a “service currently not available” type page (simple to set up) or it is a complete mirror of the primary server. For my purposes it is a status page, but it is perfectly possible to replicate the primary server if necessary – you could run rsync (preferably through an SSH tunnel with a key pair), something along the lines of “rsync -avz –delete -e “ssh -i /root/rsync/mirror-rsync-key” /home/website user@server.com:/home/website” run every few minutes by cron – with MySQL replication for the databases, for example. Google has multiple articles and how-tos for both scenarios.

A possible issue with a replicated server is if users change files via FTP or change the database using a database script when the backup server is active, you would either have to prevent them doing so or make sure you mirror those changes back to the primary server when it comes back online. Unless of course you are using a different server for the database and shared storage for the web files.

This is how I propose to do it: there’s a script on the backup server that monitors the primary server (a heartbeat-type service, run as a PHP script every few minutes by cron). The script can optionally check with any other spare servers to see if they can contact the primary server (to make sure it is definitely down). If the primary server is definitely down, the backup server updates the DNS entries for the server domain(s) to point to itself (with a low TTL in case the primary comes back online).

It’s simplest if the primary server is also the primary DNS server and the backup server is the secondary DNS server, then if a user cannot connect to the primary server, it also cannot get the incorrect DNS records (assuming the whole server is down, not just the web service, which the heartbeat script should check).

Major caveat: some ISPs may ignore the TTL of DNS records and cache the wrong results for too long, but nothing can be done about that.

Updating the DNS records could be done using the nsupdate command, but there may be issues with permissions both to run the program and for the server to update the DNS records, so it’s simpler if the backup is running Virtualmin, which comes with a remote API which can be called from a PHP script.

This is the flowchart of what happens:

BACKUP checks PRIMARY is up every x mins and loads previous state from database:
     > PRIMARY was up and is still up – do nothing
     > PRIMARY was up and is now down:
          > check with SPARE servers:
               > cannot contact SPARES – internet probably down at BACKUP – do nothing
               > at least 1 SPARE reports PRIMARY up – network issues – do nothing
               > otherwise – update DNS and log to database
     > PRIMARY was down and is still down – do nothing
     > PRIMARY was down and is back up – update DNS and log to database

This is how the PHP script could update the DNS records:

<?php

//if primary down, failover IP addresses, remove www record then add new one:

$result1 = shell_exec("wget -O - --quiet --http-user=root --http-passwd=root_pass --no-check-certificate 'https://www.backupdomain.com:10000/virtual-server/remote.cgi?program=modify-dns&domain=primarydomain.com&remove-record=www.primarydomain.com. A'");

$result2 = shell_exec("wget -O - --quiet --http-user=root --http-passwd=root_pass --no-check-certificate 'https://www.backupdomain.com:10000/virtual-server/remote.cgi?program=modify-dns&domain=primarydomain.com&ttl=60&add-record=www.primarydomain.com. A 1.2.3.4'");

//echo $result2; //should end with: Exit status: 0 if successful

// if primary back online, reverse the process (though the primary as the primary DNS server should eventually update the record on the secondary DNS server anyway)

?>

I’ll post more actual examples when I get around to implementing this 🙂

Roundcube: Vacation message with Virtualmin Plugin

None of the vacation/forwarding plugins work with Virtualmin so this is my workaround. Firstly, users need to be created as Mail and FTP users, since we will be using FTP to place the .forward files. Download and unzip the plugin:

wget http://downloads.sourceforge.net/project/rcubevacation/vacation-1.9.9.zip?use_mirror=ovh&ts=1280145947
unzip vacation-1.9.9.zip
cd vacation
vi config.ini

We are going to use Usermin’s own function (installing the Vacation program doesn’t work) – so change these lines:

binary = "/etc/usermin/forward/autoreply.pl"
alias_identities = false

At this point I usually edit default.txt and change the default text. Now, I have mail usernames in Virtualmin formatted user@domain, so a few changes need to be made to accommodate this:

cd lib
vi dotforward.class.php

We are going to change some lines. Type :set number to see line numbers.
Line 42 change to:
$arrDotForward[] = $this->options['keepcopy'] = "\\".str_replace('@','-',$this->options['username']);
Line 94 change to:
$this->options['keepcopy'] = ($first_element == str_replace('@','-',$this->options['username']));

We need to tell the script the path to the mailbox, so we need to add some code. There’s a gap around line 68 which we can put the following in – note that this is very hacky and will only work with top level domains (not subdomains) where the username is the first part of the domain. If anyone has a better way to do this, do let me know (the virtualmin files containing the home directory paths are in /etc/webmin/virtual-server/domains but are readable by the root user only or we could parse /etc/passwd for the usernames…):

$user_parts = explode('@',$this->options['username']);
$domain_parts = explode('.',$user_parts[1]);
$this->options['flags'] = '/home/'.$domain_parts[0].'/homes/'.$user_parts[0].'/.vacation.msg';

[EDIT – thanks to Iain, the code to get the correct directory from /etc/passwd follows – use this instead of the previous code block:]

$user_accounts = file_get_contents( "/etc/passwd");
$matches = preg_grep( "/" . $this->options['username'] . "/i", explode( "\n",$user_accounts));
$user_info = explode( ":", current($matches));
$this->options['flags'] = $user_info[5] . '/.vacation.msg';

[/EDIT]

Now just add the plugin to the plugins array:

cd ../../../config/
vi main.inc.php

Change the line to:
$rcmail_config['plugins'] = array('vacation');

Update: in Roundcube 0.5 the page layout has changed and the plugin displays incorrectly. To fix this, edit plugins/vacation/skins/default/vacation.css and add in some padding:
#pagecontent {
width: 800px;
padding-top:70px;
}

Roundcube: Change Virtualmin Password Plugin

I can never get the Virtualmin binary file compilation to work properly, so here’s my workaround to enable changing a user’s Virtualmin password in Roundcube Webmail.

Firstly, let’s go to the plugins/password directory (your path will be different):

cd ~username/public_html/webmail/plugins/password
cp config.inc.php.dist config.inc.php
vi config.inc.php

We need to change the password driver to virtualmin (hit i for insert mode), so it reads:

$rcmail_config['password_driver'] = 'virtualmin';

(Hit Esc to exist insert mode and then ZZ to save. You’ll need these commands throughout this tutorial.)
Now we’ll use sudo instead of the binary compilation:

cd drivers
yum -y install sudo
visudo

Comment out the line: Defaults requiretty
At the end, add the following line (change the username to whatever user your webmail directory runs as): username ALL=NOPASSWD: /usr/sbin/virtualmin

If you can’t work out which user it is, create and execute a PHP script in the webmail directory with the following contents: <?php exec('whoami',$output,$return_code); print_r($output); ?>

Now we need to change the exec line in the virtualmin script:

vi virtualmin.php

Change the line that starts with exec to:
exec("sudo /usr/sbin/virtualmin modify-user --domain $domain --user $username --pass $newpass", $output, $returnvalue);

Finally, add password to the plugins array to activate it:

cd ../../../config
vi main.inc.php

Change plugins array to: $rcmail_config['plugins'] = array('password');