scaling drupal step three - using heartbeat to implement a redundant load balancer

if you've setup a clustered drupal deployment (see scaling drupal step two - sticky load balancing with apache mod_proxy), a good next-step, is to cluster your load balancer.

one way to do this is to use heartbeat to provide instant failover to a redundant load balancer should your primary fail. while the method suggested below doesn't increase the loadbalancer scalability, which shouldn't be an issue for a reasonably sized deployment, it does increase your the redundancy. as usual, my examples are for apache2, mysql5 and drupal5 on debian etch. see the scalability overview for related articles.

deployment overview

this table summaries the characteristics of this deployment choice
scalability: fair
redundancy: good
ease of setup: fair

servers

in this example, i use:

web serverdrupal-lb1.mydomain.com192.168.1.24
web serverdrupal-lb2.mydomain.com192.168.1.25
data serverdrupal-data-server1.mydomain.com192.168.1.26
load balancerapache-balance-1.mydomain.com192.168.1.34
load balancerapache-balance-2.mydomain.com192.168.1.35
load balancer clusterapache-balance-cluster.mydomain.com192.168.1.51

network diagram


install and setup heartbeat on load balancers

setup two load balancers, apache-balance-1 and apache-balance-2 as described in my previous blog. install heartbeat and it's dependencies on both:
# apt-get install heartbeat-2
configure /etc/ha.d/ha.cf (see http://www.linux-ha.org/GettingStarted for more info) identically on each of the load balancers as follows:
logfile /var/log/ha-log
bcast eth0
keepalive 2
warntime 10
deadtime 30
initdead 120
udpport 694
auto_failback yes
node apache-balance-1
node apache-balance-2
uuidfrom nodename
respawn hacluster /usr/lib/heartbeat/ipfail
note:
  • the nodenames must be the output of uname -n
  • i had problems with my nodenames flapping, using uuidfrom fixed this.
configure /etc/ha.d/haresources. this must be identical on apache-balance-1 and apache-balance-2. really! . this file should look like:
apache-balance-1 192.168.1.51 apache2
note:
  • apache-balance-1 here refers to the "preferred" host for the service
  • 192.168.1.51 is the vip that your load balancer will appear to be on
  • apache2 here refers specifically to the name of the script in the directory /etc/init.d
configure /etc/ha.d/authkeys on both load balancers. if you're paranoid, see more secure options, in "configuring authkeys" here. this fiile should look like:
auth 2
2 crc
set authkeys permissions on both load balancers:
# chmod 600 /etc/ha.d/authkeys
configure apache to listen on the vip. edit /etc/apache2/ports.conf on both load balancers:
Listen 192.168.1.51:80
note: after this change, apache won't start on the load balancer in question, unless it has the vip. relax. that's as it should be.

theoretically you should configure each load balancer to stop apache2 starting on boot. this allows the ha daemon to take full control of starting and stopping apache. in practice i didn't need to. you might want to.

restart the ha daemons and test

restart the ha daemon on both load balancers and test:
# etc/init.d/heartbeat restart
keep an eye on the apache and heartbeat logfiles on both servers to see what is going on when you shut either loadbalancer down.
# tail -f /var/log/apache2/access.log
# tail -f /var/log/ha-log

final word

this is a fairly simplistic configuration. there is more you can do on detecting abormal situations and failing over. for more information, visit http://www.linux-ha.org

tech blog

if you found this article useful, and you are interested in other articles on linux, drupal, scaling, performance and LAMP applications, consider subscribing to my technical blog.

what would you need to

what would you need to change to load balance https?

actually, i'd like to rewrite http requests to https automatically like this:

RewriteEngine On
RewriteCond %{HTTPS} !on
RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI}

and then have the virtual host balance https-- but what should change in httpd.conf on the load balancers? should the load balancers and the web servers all have the same ssl cert on them?

John, great summary. If

John, great summary.

If you're a little hardware constrained and only have two machines, could you install heardbeat, drupal/apache/php on each, mysql_proxy and a db on each and then have two machines that both scale and are fully (and automatically) redundant for each other?

John Why did you choose for

John

Why did you choose for Heartbeat + Apache Mod proxy ?
Did you look at LVS ? Ultramonkey etc ?

The setup you described is a good one , but there are alternatives .. and I`m interested to know why you opted for this solution.

kris, my main motivation for

kris, my main motivation for mod_proxy_balancer was to use a tool that people were likely to be already familiar with (apache2). i was trying to use as few technologies as possible to scale. i also got a sense that ultramonkey wasn't a very active project. lvs is a great tool, but a bit trickier to configure.

post new comment

the content of this field is kept private and will not be shown publicly.
  • web page addresses and e-mail addresses turn into links automatically.
  • allowed html tags: <h2> <h3> <h4> <a> <em> <strong> <cite> <code> <ul> <ol> <li> <dl> <dt> <dd>
  • lines and paragraphs break automatically.
  • you may post code using <code>...</code> (generic) or <?php ... ?> (highlighted php) tags.

more information about formatting options

captcha
are you human? we hope so.
copy the characters (respecting upper/lower case) from the image.