Last updated at Fri, 03 Nov 2017 20:33:46 GMT
We’re going to explore high availability and load balancing using Keepalived and HAProxy.
Keepalived is a routing software designed to provide simple and robust facilities for load balancing and high-availability to Linux systems and Linux-based infrastructures.
HAProxy is an open source load balancer/reverse proxy generally used for load balancing web services, but also has the functionality to load balance TCP traffic.
Together, Keepalived and HAProxy provide some unique benefits.
Specifically, they provide a low cost solution for high availability compared to proprietary hardware-based load balancers.
Keepalived uses the VRRP protocol to detect when HAProxy is down and fails with minimal downtime.
The following documentation is focused on setting up Keepalived in AWS (or potentially any cloud) where multicast is not supported, and you’re using a debian-based OS.
Keepalived and HAProxy Setup
First, download the latest version of Keepalived.
(http://www.keepalived.org/software/keepalived-1.2.13.tar.gz)
You will need this version since the previous version in your repository is probably too old to support VRRP Unicast.
wget http://www.keepalived.org/software/keepalived-1.2.13.tar.gz tar xf keepalived-1.2.13.tar.gz
cd keepalived-1.2.13/
./configure
make && make install cp keepalived/etc/init.d/keepalived.init /etc/init.d/keepalived
mkdir /etc/keepalived/
Keepalived and HAProxy Architecture
We’re going to assume two load balancer servers are running Keepalived and HAProxy, where one is the master, the other is a failover, and we have multiple web servers.
In this scenario, we have an elastic IP on ServerA while ServerB is set as a passive server not being hit by the public internet. We also have five web servers running our app.
By default ServerA is set as MASTER state and ServerB is set as BACKUP, which means on startup the servers know their state, and also means on a failover/fail back scenario both servers know their role.
In a failure scenario where ServerA becomes unresponsive (i.e HAProxy process hangs, server dies etc.) ServerB will be unable to contact serverA. Therefore, ServerA enters a FAULT mode and ServerB becomes master. When ServerB becomes master it runs a script which will basically disassociate the Elastic IP from ServerA and assign it to ServerB. ServerB is now the only active HAProxy server and handles requests to the app servers.
Configs
A few online references seem to have incorrect configs for setting up Keepalived in a Unicast environment, however the following instructions seem to work correctly.
We need to set up our master and slave configs for Keepalived slightly different as they both have different roles to play in failover. The vrrp_script is used to verify the other party is functioning correctly. We also have in our notify_master section, /etc/keepalived/master.sh, a script which assigns the Elastic IP to the master in a case of failure or recovery.
Server A (master) (/etc/keepalived/keepalived.conf)
vrrp_script chk_haproxy {
script "pidof haproxy"
interval 2
}
vrrp_instance VI_1 {
debug 2
interface eth0 # interface to monitor
state MASTER
virtual_router_id 51 # Assign one ID for this route
priority 101 # 101 on master, 100 on backup
unicast_src_ip 1.2.3.4 # My IP
unicast_peer {
5.6.7.8 # peer IP
}
track_script {
chk_haproxy
}
notify_master /etc/keepalived/master.sh
}
Server B (backup) (/etc/keepalived/keepalived.conf)
vrrp_script chk_haproxy {
script "pidof haproxy"
interval 2
}
vrrp_instance VI_1 {
debug 2
interface eth0 # interface to monitor
state BACKUP
virtual_router_id 51 # Assign one ID for this route
priority 100 # 101 on master, 100 on backup
unicast_src_ip 5.6.7.8 # My IP
unicast_peer {
1.2.3.4 # Peer IP
}
track_script {
chk_haproxy
}
notify_master /etc/keepalived/master.sh
}
script notify_master (/etc/keepalived/master.sh)
#!/bin/bash
EIP=9.8.7.6
INSTANCE_ID=i-abcd1234
/usr/local/bin/aws ec2 disassociate-address --public-ip $EIP
/usr/local/bin/aws ec2 associate-address --public-ip $EIP --instance-id $INSTANCE_ID
Starting Keepalived
Keepalived startup messages are in /var/log/syslog, to see if your process started up correctly you should see an output similar to below:
ServerA – Master
Keepalived[23638]: Starting Keepalived v1.2.13 (11/28,2014)
Keepalived[23639]: Starting Healthcheck child process, pid=23641
Keepalived[23639]: Starting VRRP child process, pid=23642
Keepalived_healthcheckers[23641]: Initializing ipvs 2.6
Keepalived_healthcheckers[23641]: Registering Kernel netlink reflector
Keepalived_healthcheckers[23641]: Registering Kernel netlink command channel
Keepalived_vrrp[23642]: Registering Kernel netlink reflector
Keepalived_vrrp[23642]: Registering Kernel netlink command channel
Keepalived_vrrp[23642]: Registering gratuitous ARP shared channel
Keepalived_vrrp[23642]: Opening file '/etc/keepalived/keepalived.conf'.
Keepalived_vrrp[23642]: Configuration is using : 60724 Bytes
Keepalived_vrrp[23642]: Using LinkWatch kernel netlink reflector...
Keepalived_healthcheckers[23641]: Opening file '/etc/keepalived/keepalived.conf'.
Keepalived_healthcheckers[23641]: Configuration is using : 5129 Bytes
Keepalived_healthcheckers[23641]: Using LinkWatch kernel netlink reflector...
Keepalived_vrrp[23642]: VRRP_Instance(VI_1) Transition to MASTER STATE
Keepalived_vrrp[23642]: VRRP_Instance(VI_1) Entering MASTER STATE
Keepalived_vrrp[23642]: Opening script file /etc/keepalived/master.sh
ServerB – Backup
Keepalived[28033]: Starting Keepalived v1.2.13 (11/28,2014)
Keepalived[28034]: Starting Healthcheck child process, pid=28036
Keepalived[28034]: Starting VRRP child process, pid=28037
Keepalived_healthcheckers[28036]: Initializing ipvs 2.6
Keepalived_vrrp[28037]: Registering Kernel netlink reflector
Keepalived_vrrp[28037]: Registering Kernel netlink command channel
Keepalived_vrrp[28037]: Registering gratuitous ARP shared channel
Keepalived_healthcheckers[28036]: Registering Kernel netlink reflector
Keepalived_healthcheckers[28036]: Registering Kernel netlink command channel
Keepalived_vrrp[28037]: Opening file '/etc/keepalived/keepalived.conf'.
Keepalived_vrrp[28037]: Configuration is using : 61884 Bytes
Keepalived_vrrp[28037]: Using LinkWatch kernel netlink reflector...
Keepalived_vrrp[28037]: VRRP_Instance(VI_1) Entering BACKUP STATE
Keepalived_healthcheckers[28036]: Opening file '/etc/keepalived/keepalived.conf'.
Keepalived_healthcheckers[28036]: Configuration is using : 5127 Bytes
Verify Connectivity
There are many ways to test connectivity, tshark, tcpdump etc, here are some sample commands:
tcpdump "ip proto 112"
tshark -f “vrrp”
The reason we need to verify connectivity with either of the commands above is because Keepalived uses multicast, which is not supported in AWS. The commands verify our config is correctly sending via Unicast.
Testing Failover
Assuming you have your failover.sh script setup correctly, on stopping HAProxy on ServerA, you should see it entering a FAULT state and see ServerB entering a MASTER state. When ServerB enters a master state it will run a script and reassign the Elastic IP to ServerB.
Ready to start getting insights from your applications? Sign up for a Logentries free trial today.
