Best Practices for a Fault Tolerant Web Layer

Category: LAMP infrastructure

Author: Walle Lindberg
Principal Consultant
Elysium Solutions LLC


March 26, 2009

When creating a production LAMP environment, ensuring maximum uptime requires creating a fault tolerant environment. Fault tolerance should be attained at all levels of the stack, but today I’ll be focusing on the web layer.

One way of achieving this is to use Apaches built in Load Balancer, mod_proxy_balancer. Or another option would be to use round robin DNS. The preferred way would be to use a commercial load balancer.

The objective is to have a multiple node web layer, that is getting traffic distributed evenly across all nodes. In the event that all the nodes fail, you should have a failover mechanism that sends traffic to another cluster. If not there should be a redirect in place that will redirect the user to a maintenance page; and this should happen all happen automatically. The ultimate goal is to never have a DOS (denial of service) situation presented to the customer.

Most commercial load balancers allow you to check if a host is available by various methods. The most simple would be a ping check. A better method would be a port check. (Port 80 in this case) The best way would be to do a content check. You should create a static page that is very small in size, and the page should contain a simple phrase such as “content_check_ok” or “keepmeinthepool”. If the application depends on a database, the content page should do a simple database query and display your OK phrase if the query was successful. This will enable the load balancer to do a quick end to end test every time it checks the host.

For example, if you are using a F5 BigIP Load Balancer, then you would do this by creating a HTTP profile for your servers. Your profile would have the Fallback Host setting enabled, and the fallback host should be your maintenance page. (i.e. http://elysol.com/temp_maint.html) You need to set up a HTTP monitor that has a “Send String” of “GET /status/monitor_check.pl HTTP/1.0 “ and a “Receive String” of “keepmeinthepool” or whatever string that you are returning. Finally you should set your new Virtual Server or (Virtual IP) to use this new HTTP monitor and HTTP profile.

You can use PHP or a simple perl script in you cgi-bin directory. I prefer to use perl, as most linux distros already have perl installed. If you need to install PHP on your web server just for this, it probably isn’t worth it. Below is a quick perl script you can use that will :

  1. create a table "foo"
  2. generate a random number > 100000000
  3. insert the random number to the database
  4. retrieve and compare the number in the database to the number you generated
  5. display "KEEPME" if everything is good
  6. display "Check Failed" if it cant connect or write to the database

*Don't forget to create the database! an mysql statement like... "create database testdb;grant all on testdb.* to testuser identified by 'testpw';flush privileges;" will suffice.

**Disclaimer- there are many ways to do this. Some people may argue that you shouldn't create a table and that you shouldn't create data to test- as it can be destructive. My argument is that you should create something, as you need to know if the database/filesystem has dropped into read only mode. But there may be repercussions- (for example recovering space from an innodb database) Please do your research and consult a DBA!

################

#!/usr/bin/perl -w

print "Content-type: text/plain; charset=iso-8859-1\n\n";

use strict;
use DBI();
use IO::Socket;

#variables
my $dbuser = 'testuser';
my $dbpass = 'testpw';
my $dbname = 'testdb';
my $dbport = '3306';
my $dbhost = 'localhost';
my $status;
my $minimum = 100000000;
my $range = 100000000;
my $random_number = int(rand($range));

#Print the random Number for debug
print $random_number . "\n";

eval {
     local $SIG{ALRM} = sub { die "TIMEOUT"; };
     alarm 2;
my $sock = new IO::Socket::INET (
     PeerAddr => $dbhost,
     PeerPort => $dbport,
     Proto => 'tcp',
     );
die "SOCKETFAIL\n" unless $sock;
die "SOCKETSUCC\n";
alarm 0;
};
alarm 0;

if ( $@ && $@ =~ /TIMEOUT/ ) {
     print "mysqld port check 3306 timeout...\n";
     };
if ( $@ && $@ =~ /SOCKETFAIL/ ) {
     print "mysqld port 3306 not open...\n";
     };
if ( $@ && $@ =~ /SOCKETSUCC/ ) {
     #Connect to the database.
     #sub mysqlConnect() { # e.g. y $dbh = DBI->connect("DBI:mysql:database=$db{'Name'};host=localhost","testuser","testdb",{'RaiseError' => 1});
     my $dsn = "DBI:mysql:database=$dbname;host=$dbhost;port=$dbport";
     my $dbh = DBI->connect($dsn, "$dbuser", "$dbpass",{'RaiseError' => 1});

     # Drop table 'foo'. This may fail, if 'foo' doesn't exist. Thus we put an eval around it.
     eval { $dbh->do("DROP TABLE foo") };
     print "Dropping foo failed: $@\n" if $@;

     # Create a new table 'foo'.
     $dbh->do("CREATE TABLE foo (id INTEGER, randomnumber DECIMAL)");

     # INSERT some data into 'foo'. We are using $dbh->quote() for quoting the name.
     $dbh->do("INSERT INTO foo VALUES (1, " . $dbh->quote($random_number) . ")");

     # Now retrieve data from the table.
     my $sth = $dbh->prepare("SELECT * FROM foo");
     $sth->execute();

while (my $ref = $sth->fetchrow_hashref()) {
     print "$ref->{'randomnumber'}\n";
     $status = $ref->{'randomnumber'};
     print $status . "\n";
   }
$sth->finish();

# Disconnect from the database.
$dbh->disconnect();

if ($status =~ /$random_number/) {
     print "KEEPME\n";
     }
     else {
     print "check failed\n";
     }
   }