Forums/Knowledge Base/HAProxy

Install HAProxy and Keepalived (Virtual IP)

Ashraf Sharif
posted this on April 19, 2013, 12:11

To avoid a single point of failure with your HAProxy, one would set up two identical HAProxy instances (one active and one standby) and use Keepalived to run VRRP between them. VRRP provides a virtual IP address to the active HAProxy, and transfers the Virtual IP to the standby HAProxy in case of failure. This is seamless because the two HAProxy instances need no shared state.

In this example, we are using 2 nodes to act as the load balancer with IP failover in front of our database cluster. VIP will be floating around between LB1 (master) and LB2 (backup). When LB1 is down, the VIP will be taking over by LB2 and once the LB1 up again, the VIP will be failback to LB1 since it hold the higher priority number. 

We are using following hosts/IPs:

VIP: 192.168.10.100
LB1: 192.168.10.101
LB2: 192.168.10.102

DB1: 192.168.10.111
DB2: 192.168.10.112
DB3: 192.168.10.113
ClusterControl: 192.168.10.115

You may refer to following diagram for the architecture:

haproxy_keepalived.PNG

 

Install HAproxy

1. Before we start to deploy, make sure LB1 and LB2 are accessible using passwordless SSH. Copy the SSH keys to the load balancer nodes:

$ ssh-copy-id -i ~/.ssh/id_rsa 192.168.10.101
$ ssh-copy-id -i ~/.ssh/id_rsa 192.168.10.102 

2. Install HAproxy into both nodes, select in the UI Manage -> Load Balancer

Screen_Shot_2015-06-03_at_17.38.14.png

Click "Install HAProxy" when you are happy with the settings. The HAProxy Configuration template is stored on the controller in /usr/share/cmon/templates/haproxy.cfg   and  in that directory you also have a the template for the mysqlchk script.

3. You will noticed that these 2 load balancer nodes have been installed and provisioned by ClusterControl. You can verify this by login into ClusterControl > Nodes and you should see similar screenshot as below:

haproxy.png

 

Install Keepalived

Requires that you have two load balancers installed

1. Navigate to Manage -> Load Balancer, and select the tab Keepalived.

Screen_Shot_2015-06-03_at_17.43.23.png

Installation completed! You can now access your database servers through VIP, 192.168.10.100 port 33306.

 

Comments

User photo
JOE YU

How to avoid brain split in the above keepalived configuration?

eg. when the communication is broken between keepalived master and slave host ?

July 1, 2013, 18:44
User photo
Ashraf Sharif
Severalnines

You can refer to following pages for detailed explanation on how to avoid Keepalived split-brain:

http://scale-out-blog.blogspot.com/2011/01/virtual-ip-addresses-and-their.html

http://www.austintek.com/LVS/LVS-HOWTO/HOWTO/LVS-HOWTO.failover.html

July 1, 2013, 19:36
User photo
JOE YU

Thanks for reply.

Have read these two docs already. Still can't get clue to solve the problem without using pacemaker or other component to make a complex architect.

Not sure if using the same state (Like BACKUP) and same priority like (100) in both keepalived.conf  can avoid the brain split of keepalived.

 

Thanks,

July 1, 2013, 19:44
User photo
Aiman Farhat

Hi Ashraf,

Thanks for this nice article, I am having oneissue and can't figure it out. The virtual IP gets assigned to the master and on fail over the VIP get's assigned to the backup, but the issue is I can't ping the IP Address (10.134.41.180)  from the backup or externally. 

 ip addr show:

eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:50:56:a8:0d:cf brd ff:ff:ff:ff:ff:ff
inet 10.134.41.103/25 brd 10.134.41.127 scope global eth0
inet 10.134.41.180/32 scope global eth0
inet6 fe80::250:56ff:fea8:dcf/64 scope link
valid_lft forever preferred_lft forever

 

Thanks for your help

Aiman

July 24, 2013, 21:51
User photo
Ashraf Sharif
Severalnines

Hi Aiman,

 

Try to check firewall setting on backup host and ARP table on client host. You can use command "arp -an" to verify the latest virtual IP mapping. It should be mapped to the backup host's MAC address. Depending on your router or switch, you might face "arp cache problem" if the virtual IP has been failover but not updated in your client's ARP table.

July 25, 2013, 08:07
User photo
Aiman Farhat

Hi Ashraf,

   Thank you for the speedy reply. It turn out that the VIP I am using it is not on the same subnet. I got it working now.

Thanks again.

Aiman

July 25, 2013, 12:45
User photo
Shaun Botsis

Hi Ashraf

 

I get the following error when trying to provision HAProxy to a vanilla debian7 install.

 

ll# ./s9s-admin/cluster/s9s_haproxy --install -i 1 -h 172.16.200.48
cmon12341
load opts 1
Testing ssh to 172.16.200.48: ssh -q -p22 -o UserKnownHostsFile=/dev/null -o Str                    ictHostKeyChecking=no -oNumberOfPasswordPrompts=0 -oConnectTimeout=10 -oIdentity                    File=/root/.ssh/id_rsa -oNumberOfPasswordPrompts=0 root@172.16.200.48  ls -al /u                    sr
[ok]
Using loadbalancing policy 'leastconn'.
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/tmp/mys                    ql.sock' (2)
No hostnames found.

What am I doing wrong ?

Thanks

November 3, 2014, 20:38
User photo
Ashraf Sharif
Severalnines

Hi Shaun,

Could you update in /usr/bin/s9s_haproxy line 76 to:

chmod 644 $MYCNF_CMON

Then try again the HAproxy deployment. Please verify if it is working.

Regards,
Ashraf

November 4, 2014, 03:09
User photo
Shaun Botsis

hi Ashraf

I get exactly the same response after updating

chmod 600 $MYCNF_CMON >> chmod 644 $MYCNF_CMON

November 4, 2014, 04:29
User photo
Ashraf Sharif
Severalnines

Hi Shaun,

Can you edit /usr/bin/s9s_haproxy starting from line 71:

[mysql_cmon]
user=cmon
password=
EOF

to:

[mysql_cmon]
host=127.0.0.1
port=3306
user=cmon
password=
EOF

 

Then save and try again the deployment. It seems like CMON didn't use correct credentials when connecting to CMON DB.

 

Regards,

Ashraf

November 4, 2014, 05:02
User photo
Ashraf Sharif
Severalnines

Hi Shaun,

Can you edit /usr/bin/s9s_haproxy starting from line 71:

[mysql_cmon]
user=cmon
password=
EOF

to:

[mysql_cmon]
host=127.0.0.1
port=3306
user=cmon
password=
EOF

 

Then save and try again the deployment. It seems like CMON didn't use correct credentials when connecting to CMON DB.

 

Regards,

Ashraf

November 4, 2014, 05:02
User photo
Shaun Botsis

Thanks for the help Ashraf :)  I got it working. 

 

There was one more issue relating to the package manager not having the HAproxy package available.

On my Debian 7 install I had to add deb http://ftp.debian.org/debian/ wheezy-backports main  to /etc/apt/sources.list.

November 8, 2014, 16:47
User photo
Ashraf Sharif
Severalnines

Hi Shaun,

Yes, we are aware of that. Certain package managers do not have Haproxy in their repository, so installing with "use source" option inside ClusterControl UI would solve this. 

Regards,

Ashraf

November 10, 2014, 03:49
User photo
Rafael Oliveira

Hi guys,

After following all steps to get Keepalived installed and configured, I realized that it is assigning the VIP to my slave LB server when the master is online yet.

I tried restarting keepalived service from both LB servers but after they are back online, VIP gets assigned to both again.

What would be the reason for that to be happening?

January 29, 2015, 20:57
User photo
Baptiste Assmann

Hi Rafael,

Can you check the message reported by keepalived in your syslog?
There may be some interesting hints inside. Also, running a tcpdump and capture the VRRP traffic on both hosts may help.

Baptiste

January 29, 2015, 23:45
User photo
Rafael Oliveira

Hi Baptiste, thank you in advance for your interest on helping. Here it goes:

LB1:

[root@mktapps-lb1 centos]# tail -f /var/log/messages
Jan 30 13:25:48 mktapps-lb1 Keepalived_healthcheckers[10278]: Opening file '/etc/keepalived/keepalived.conf'.
Jan 30 13:25:48 mktapps-lb1 Keepalived_healthcheckers[10278]: Configuration is using : 7273 Bytes
Jan 30 13:25:48 mktapps-lb1 Keepalived_healthcheckers[10278]: Using LinkWatch kernel netlink reflector...
Jan 30 13:25:49 mktapps-lb1 Keepalived_vrrp[10279]: VRRP_Instance(VI_1) Transition to MASTER STATE
Jan 30 13:25:50 mktapps-lb1 Keepalived_vrrp[10279]: VRRP_Instance(VI_1) Entering MASTER STATE
Jan 30 13:25:50 mktapps-lb1 Keepalived_vrrp[10279]: VRRP_Instance(VI_1) setting protocol VIPs.
Jan 30 13:25:50 mktapps-lb1 Keepalived_vrrp[10279]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.10.25
Jan 30 13:25:50 mktapps-lb1 Keepalived_healthcheckers[10278]: Netlink reflector reports IP 192.168.10.25 added
Jan 30 13:25:52 mktapps-lb1 ntpd[329]: Listen normally on 14 eth0 192.168.10.25 UDP 123
Jan 30 13:25:55 mktapps-lb1 Keepalived_vrrp[10279]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.10.25

 

LB2:

[root@mktapps-lb2 centos]# tail -f /var/log/messages
Jan 30 13:25:46 mktapps-lb2 avahi-daemon[325]: Registering new address record for 192.168.10.25 on eth0.IPv4.
Jan 30 13:25:47 mktapps-lb2 ntpd[382]: Listen normally on 13 eth0 192.168.10.25 UDP 123
Jan 30 13:25:51 mktapps-lb2 Keepalived_vrrp[10226]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.10.25
Jan 30 13:36:29 mktapps-lb2 systemd: Stopping LVS and VRRP High Availability Monitor...
Jan 30 13:36:29 mktapps-lb2 Keepalived[10224]: Stopping Keepalived v1.2.10 (06/10,2014)
Jan 30 13:36:29 mktapps-lb2 Keepalived_vrrp[10226]: VRRP_Instance(VI_1) sending 0 priority
Jan 30 13:36:29 mktapps-lb2 Keepalived_vrrp[10226]: VRRP_Instance(VI_1) removing protocol VIPs.
Jan 30 13:36:29 mktapps-lb2 systemd: Stopped LVS and VRRP High Availability Monitor.
Jan 30 13:36:29 mktapps-lb2 avahi-daemon[325]: Withdrawing address record for 192.168.10.25 on eth0.
Jan 30 13:36:31 mktapps-lb2 ntpd[382]: Deleting interface #13 eth0, 192.168.10.25#123, interface stats: received=0, sent=0, dropped=0, active_time=644 secs

January 30, 2015, 14:44
User photo
Rafael Oliveira

I found an article online that says that by default keepalived uses 224.0.0.18 IP address for VRRP (Virtual Router Redundancy Protocol) for communication between two nodes for health check. So I ran tcpdump as follows on eth0 (please correct if this information is wrong):

Commands: tcpdump -v -i eth0 host 224.0.0.18
                       tcpdump -vvv -n -i eth0 host 224.0.0.18

 

LB1:

[root@mktapps-lb1 centos]# tcpdump -v -i eth0 host 224.0.0.18
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
13:47:38.506958 IP (tos 0xc0, ttl 255, id 1309, offset 0, flags [none], proto VRRP (112), length 40)
host-192-168-10-19.cisco.com > vrrp.mcast.net: vrrp host-192-168-10-19.cisco.com > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 101, authtype none, intvl 1s, length 20, addrs: mktapps-lb1.novalocal
13:47:39.507504 IP (tos 0xc0, ttl 255, id 1310, offset 0, flags [none], proto VRRP (112), length 40)
host-192-168-10-19.cisco.com > vrrp.mcast.net: vrrp host-192-168-10-19.cisco.com > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 101, authtype none, intvl 1s, length 20, addrs: mktapps-lb1.novalocal
13:47:40.508707 IP (tos 0xc0, ttl 255, id 1311, offset 0, flags [none], proto VRRP (112), length 40)
host-192-168-10-19.cisco.com > vrrp.mcast.net: vrrp host-192-168-10-19.cisco.com > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 101, authtype none, intvl 1s, length 20, addrs: mktapps-lb1.novalocal
13:47:41.509283 IP (tos 0xc0, ttl 255, id 1312, offset 0, flags [none], proto VRRP (112), length 40)
host-192-168-10-19.cisco.com > vrrp.mcast.net: vrrp host-192-168-10-19.cisco.com > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 101, authtype none, intvl 1s, length 20, addrs: mktapps-lb1.novalocal
13:47:42.510433 IP (tos 0xc0, ttl 255, id 1313, offset 0, flags [none], proto VRRP (112), length 40)
host-192-168-10-19.cisco.com > vrrp.mcast.net: vrrp host-192-168-10-19.cisco.com > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 101, authtype none, intvl 1s, length 20, addrs: mktapps-lb1.novalocal
13:47:43.511006 IP (tos 0xc0, ttl 255, id 1314, offset 0, flags [none], proto VRRP (112), length 40)
host-192-168-10-19.cisco.com > vrrp.mcast.net: vrrp host-192-168-10-19.cisco.com > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 101, authtype none, intvl 1s, length 20, addrs: mktapps-lb1.novalocal
^C
6 packets captured
6 packets received by filter
0 packets dropped by kernel
[root@mktapps-lb1 centos]# tcpdump -vvv -n -i eth0 host 224.0.0.18
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
13:47:56.522951 IP (tos 0xc0, ttl 255, id 1327, offset 0, flags [none], proto VRRP (112), length 40)
192.168.10.19 > 224.0.0.18: vrrp 192.168.10.19 > 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 101, authtype none, intvl 1s, length 20, addrs: 192.168.10.25
13:47:57.523510 IP (tos 0xc0, ttl 255, id 1328, offset 0, flags [none], proto VRRP (112), length 40)
192.168.10.19 > 224.0.0.18: vrrp 192.168.10.19 > 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 101, authtype none, intvl 1s, length 20, addrs: 192.168.10.25
13:47:58.524739 IP (tos 0xc0, ttl 255, id 1329, offset 0, flags [none], proto VRRP (112), length 40)
192.168.10.19 > 224.0.0.18: vrrp 192.168.10.19 > 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 101, authtype none, intvl 1s, length 20, addrs: 192.168.10.25
13:47:59.525325 IP (tos 0xc0, ttl 255, id 1330, offset 0, flags [none], proto VRRP (112), length 40)
192.168.10.19 > 224.0.0.18: vrrp 192.168.10.19 > 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 101, authtype none, intvl 1s, length 20, addrs: 192.168.10.25
^C
4 packets captured
4 packets received by filter
0 packets dropped by kernel
[root@mktapps-lb1 centos]#

 

LB2:

[root@mktapps-lb2 centos]# tcpdump -v -i eth0 host 224.0.0.18
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
13:47:06.324102 IP (tos 0xc0, ttl 255, id 3, offset 0, flags [none], proto VRRP (112), length 40)
host-192-168-10-20.cisco.com > vrrp.mcast.net: vrrp host-192-168-10-20.cisco.com > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 100, authtype none, intvl 1s, length 20, addrs: mktapps-lb2.novalocal
13:47:07.324438 IP (tos 0xc0, ttl 255, id 4, offset 0, flags [none], proto VRRP (112), length 40)
host-192-168-10-20.cisco.com > vrrp.mcast.net: vrrp host-192-168-10-20.cisco.com > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 100, authtype none, intvl 1s, length 20, addrs: mktapps-lb2.novalocal
13:47:08.325564 IP (tos 0xc0, ttl 255, id 5, offset 0, flags [none], proto VRRP (112), length 40)
host-192-168-10-20.cisco.com > vrrp.mcast.net: vrrp host-192-168-10-20.cisco.com > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 100, authtype none, intvl 1s, length 20, addrs: mktapps-lb2.novalocal
13:47:09.325779 IP (tos 0xc0, ttl 255, id 6, offset 0, flags [none], proto VRRP (112), length 40)
host-192-168-10-20.cisco.com > vrrp.mcast.net: vrrp host-192-168-10-20.cisco.com > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 100, authtype none, intvl 1s, length 20, addrs: mktapps-lb2.novalocal
13:47:10.325956 IP (tos 0xc0, ttl 255, id 7, offset 0, flags [none], proto VRRP (112), length 40)
host-192-168-10-20.cisco.com > vrrp.mcast.net: vrrp host-192-168-10-20.cisco.com > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 100, authtype none, intvl 1s, length 20, addrs: mktapps-lb2.novalocal
13:47:11.326285 IP (tos 0xc0, ttl 255, id 8, offset 0, flags [none], proto VRRP (112), length 40)
host-192-168-10-20.cisco.com > vrrp.mcast.net: vrrp host-192-168-10-20.cisco.com > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 100, authtype none, intvl 1s, length 20, addrs: mktapps-lb2.novalocal
^C
6 packets captured
6 packets received by filter
0 packets dropped by kernel
[root@mktapps-lb2 centos]# tcpdump -vvv -n -i eth0 host 224.0.0.18
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
13:48:07.367359 IP (tos 0xc0, ttl 255, id 64, offset 0, flags [none], proto VRRP (112), length 40)
192.168.10.20 > 224.0.0.18: vrrp 192.168.10.20 > 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 100, authtype none, intvl 1s, length 20, addrs: 192.168.10.25
13:48:08.368530 IP (tos 0xc0, ttl 255, id 65, offset 0, flags [none], proto VRRP (112), length 40)
192.168.10.20 > 224.0.0.18: vrrp 192.168.10.20 > 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 100, authtype none, intvl 1s, length 20, addrs: 192.168.10.25
13:48:09.368680 IP (tos 0xc0, ttl 255, id 66, offset 0, flags [none], proto VRRP (112), length 40)
192.168.10.20 > 224.0.0.18: vrrp 192.168.10.20 > 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 100, authtype none, intvl 1s, length 20, addrs: 192.168.10.25
13:48:10.369827 IP (tos 0xc0, ttl 255, id 67, offset 0, flags [none], proto VRRP (112), length 40)
192.168.10.20 > 224.0.0.18: vrrp 192.168.10.20 > 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 100, authtype none, intvl 1s, length 20, addrs: 192.168.10.25
^C
4 packets captured
4 packets received by filter
0 packets dropped by kernel
[root@mktapps-lb2 centos]#

 

January 30, 2015, 14:55
User photo
Rafael Oliveira

These are the log messages right after I stop/start keepalived:

LB1:
[root@mktapps-lb1 centos]# tail -f /var/log/messages
Jan 30 14:01:35 mktapps-lb1 Keepalived_vrrp[12567]: Registering Kernel netlink reflector
Jan 30 14:01:35 mktapps-lb1 Keepalived_vrrp[12567]: Registering Kernel netlink command channel
Jan 30 14:01:35 mktapps-lb1 Keepalived_vrrp[12567]: Registering gratuitous ARP shared channel
Jan 30 14:01:35 mktapps-lb1 Keepalived_vrrp[12567]: Opening file '/etc/keepalived/keepalived.conf'.
Jan 30 14:01:35 mktapps-lb1 Keepalived_vrrp[12567]: Configuration is using : 64418 Bytes
Jan 30 14:01:35 mktapps-lb1 Keepalived_vrrp[12567]: Using LinkWatch kernel netlink reflector...
Jan 30 14:01:35 mktapps-lb1 Keepalived_vrrp[12567]: VRRP sockpool: [ifindex(2), proto(112), unicast(0), fd(10,11)]
Jan 30 14:01:35 mktapps-lb1 Keepalived_healthcheckers[12566]: Opening file '/etc/keepalived/keepalived.conf'.
Jan 30 14:01:35 mktapps-lb1 Keepalived_healthcheckers[12566]: Configuration is using : 7273 Bytes
Jan 30 14:01:35 mktapps-lb1 Keepalived_healthcheckers[12566]: Using LinkWatch kernel netlink reflector...
Jan 30 14:01:36 mktapps-lb1 Keepalived_vrrp[12567]: VRRP_Instance(VI_1) Transition to MASTER STATE
Jan 30 14:01:37 mktapps-lb1 Keepalived_vrrp[12567]: VRRP_Instance(VI_1) Entering MASTER STATE
Jan 30 14:01:37 mktapps-lb1 Keepalived_vrrp[12567]: VRRP_Instance(VI_1) setting protocol VIPs.
Jan 30 14:01:37 mktapps-lb1 Keepalived_vrrp[12567]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.10.25
Jan 30 14:01:37 mktapps-lb1 Keepalived_healthcheckers[12566]: Netlink reflector reports IP 192.168.10.25 added
Jan 30 14:01:39 mktapps-lb1 ntpd[329]: Listen normally on 16 eth0 192.168.10.25 UDP 123
Jan 30 14:01:42 mktapps-lb1 Keepalived_vrrp[12567]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.10.25

 

LB2:
[root@mktapps-lb2 centos]# tail -f /var/log/messages
Jan 30 14:02:18 mktapps-lb2 Keepalived_healthcheckers[11931]: Netlink reflector reports IP fe80::f816:3eff:fe25:92c1 added
Jan 30 14:02:18 mktapps-lb2 Keepalived_healthcheckers[11931]: Registering Kernel netlink reflector
Jan 30 14:02:18 mktapps-lb2 Keepalived_healthcheckers[11931]: Registering Kernel netlink command channel
Jan 30 14:02:18 mktapps-lb2 Keepalived_healthcheckers[11931]: Opening file '/etc/keepalived/keepalived.conf'.
Jan 30 14:02:18 mktapps-lb2 Keepalived_healthcheckers[11931]: Configuration is using : 7263 Bytes
Jan 30 14:02:18 mktapps-lb2 Keepalived_vrrp[11932]: Opening file '/etc/keepalived/keepalived.conf'.
Jan 30 14:02:18 mktapps-lb2 Keepalived_vrrp[11932]: Configuration is using : 64408 Bytes
Jan 30 14:02:18 mktapps-lb2 Keepalived_vrrp[11932]: Using LinkWatch kernel netlink reflector...
Jan 30 14:02:18 mktapps-lb2 Keepalived_vrrp[11932]: VRRP sockpool: [ifindex(2), proto(112), unicast(0), fd(10,11)]
Jan 30 14:02:18 mktapps-lb2 Keepalived_healthcheckers[11931]: Using LinkWatch kernel netlink reflector...
Jan 30 14:02:19 mktapps-lb2 Keepalived_vrrp[11932]: VRRP_Instance(VI_1) Transition to MASTER STATE
Jan 30 14:02:20 mktapps-lb2 Keepalived_vrrp[11932]: VRRP_Instance(VI_1) Entering MASTER STATE
Jan 30 14:02:20 mktapps-lb2 Keepalived_vrrp[11932]: VRRP_Instance(VI_1) setting protocol VIPs.
Jan 30 14:02:20 mktapps-lb2 Keepalived_vrrp[11932]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.10.25
Jan 30 14:02:20 mktapps-lb2 Keepalived_healthcheckers[11931]: Netlink reflector reports IP 192.168.10.25 added
Jan 30 14:02:20 mktapps-lb2 avahi-daemon[325]: Registering new address record for 192.168.10.25 on eth0.IPv4.
Jan 30 14:02:21 mktapps-lb2 ntpd[382]: Listen normally on 17 eth0 192.168.10.25 UDP 123
Jan 30 14:02:25 mktapps-lb2 Keepalived_vrrp[11932]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.10.25

 

January 30, 2015, 15:05
User photo
Ashraf Sharif
Severalnines

Hi Rafael,

Did you have iptables enabled between LB1 and LB2? If yes, please add following rules:

iptables -I INPUT -i eth0 -d 224.0.0.0/8 -j ACCEPT
iptables -I INPUT -p 112 -i eth0 -j ACCEPT
iptables -I OUTPUT -p 112 -o eth0 -j ACCEPT

 

Please also send a complete log of Keepalived:

$ grep -i keepalived /var/log/messages

 

Regards,

Ashraf

January 30, 2015, 15:06
User photo
Rafael Oliveira

Hi Ashraf,

I am using centOS 7 in both machines and I haven't configured anything on iptables. Before running iptables commands, do I need to enable/start any service?

Keepalived logs attached.

 

January 30, 2015, 15:20
User photo
Rafael Oliveira

I have installed iptables-services:

# yum -y install iptables-services

And ran the commmands for iptables Ashraf posted.

However, after restarting keepalived, still get same results (VIP being assigned to both machines at same time).

January 30, 2015, 15:28
User photo
Ashraf Sharif
Severalnines

Hi Rafael,

I would say run the iptables command first and restart keepalived on both hosts. Ensure selinux is turned off or set to permissive mode. We haven't really test the Keepalived deployment on CentOS 7 at the moment. If the problem still persists, please attach the output of "iptables -L -n" and keepalived.conf from both hosts.

 

Regards,

Ashraf

January 30, 2015, 15:29
User photo
Rafael Oliveira

Hi Ashraf,

SElinux is disabled on both servers:

---

[root@mktapps-lb1 centos]# sestatus
SELinux status: disabled
[root@mktapps-lb1 centos]#

---

“iptables -L -n" from both servers (same values):

Chain INPUT (policy ACCEPT)
target prot opt source destination
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED
ACCEPT icmp -- 0.0.0.0/0 0.0.0.0/0
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:22
REJECT all -- 0.0.0.0/0 0.0.0.0/0 reject-with icmp-host-prohibited

Chain FORWARD (policy ACCEPT)
target prot opt source destination
REJECT all -- 0.0.0.0/0 0.0.0.0/0 reject-with icmp-host-prohibited

Chain OUTPUT (policy ACCEPT)
target prot opt source destination
[root@mktapps-lb2 centos]#

 

LB1 keepalived.conf:

! Configuration File for keepalived
global_defs {
notification_email {
raolivei@domain.com
}
notification_email_from LB_NODE1@domain.com
smtp_server sjc-mail-00.domain.com
smtp_connect_timeout 30
}
vrrp_script chk_haproxy {
script "killall -0 haproxy" # verify the pid existance
interval 2 # check every 2 seconds
weight 2 # add 2 points of prio if OK
}

vrrp_instance VI_1 {
interface eth0 # interface to monitor
state MASTER
virtual_router_id 51 # Assign one ID for this route
priority 101 # 101 on master, 100 on backup
virtual_ipaddress {
192.168.10.25 # the virtual IP
}
track_script {
chk_haproxy
}
}

 

LB2 keepalived.conf:

! Configuration File for keepalived
global_defs {
notification_email {
raolivei@domain.com
}
notification_email_from LB_NODE@domain.com
smtp_server sjc-mail-00.domain.com
smtp_connect_timeout 30
}
vrrp_script chk_haproxy {
script "killall -0 haproxy" # verify the pid existance
interval 2 # check every 2 seconds
weight 2 # add 2 points of prio if OK
}

vrrp_instance VI_1 {
interface eth0 # interface to monitor
state MASTER
virtual_router_id 51 # Assign one ID for this route
priority 100 # 101 on master, 100 on backup
virtual_ipaddress {
192.168.10.25 # the virtual IP
}
track_script {
chk_haproxy

}

}

January 30, 2015, 15:48
User photo
Ashraf Sharif
Severalnines

Hi Rafael,

It seems multicast environment doesn't really work for you. I would suggest you to disable iptables completely and try with unicast instead. On keepalived.conf apply following config:

LB1 (192.168.10.19):

vrrp_script chk_haproxy {
   script "killall -0 haproxy"   # verify the pid existance
   interval 2                    # check every 2 seconds
   weight 2                      # add 2 points of prio if OK
}
 
vrrp_instance VI_1 {
   interface eth0                # interface to monitor
   state MASTER
   virtual_router_id 51          # Assign one ID for this route
   priority 101                  # 101 on master, 100 on backup
   virtual_ipaddress {
	192.168.10.25		# the virtual IP
   }
   unicast_src_ip 192.168.10.19
   unicast_peer {
	192.168.10.20
   }
   track_script {
       chk_haproxy
   }
}

LB2 (192.168.10.20):

vrrp_script chk_haproxy {
   script "killall -0 haproxy"   # verify the pid existance
   interval 2                    # check every 2 seconds
   weight 2                      # add 2 points of prio if OK
}
 
vrrp_instance VI_1 {
   interface eth0                # interface to monitor
   state MASTER
   virtual_router_id 51          # Assign one ID for this route
   priority 100                  # 101 on master, 100 on backup
   virtual_ipaddress {
       192.168.10.25            # the virtual IP
   }
   unicast_src_ip 192.168.10.20
   unicast_peer {
	192.168.10.19
   }
   track_script {
       chk_haproxy
   }
}

 

Notice the unicast_src_ip and unicast_peer added to the config. 

Regards,

Ashraf

January 30, 2015, 17:28
User photo
Mauricio Querves

Hello All...

 

I've installed keepalived on a RHEL 7 and had the same problem as Rafael Oliveira

What I did was to disable Firewall and it get Working ... so I have to enable the UDP port 123 on the servers that I've installed Keepalive.

I wait that this could be helpull to someone else.

Regards,

 

Mauricio.

 

May 7, 2015, 20:44
User photo
Mauricio Querves

Something esle:

I have added this rules to the FW (both keepalived nodes):

iptables -I INPUT -d 224.0.0.0/8 -j ACCEPT
iptables -I INPUT -p vrrp -j ACCEPT

May 7, 2015, 21:47