Install HAProxy and Keepalived (Virtual IP)

Follow
Avatar
Ashraf Sharif

To avoid a single point of failure with your HAProxy, one would set up two identical HAProxy instances (one active and one standby) and use Keepalived to run VRRP between them. VRRP provides a virtual IP address to the active HAProxy, and transfers the Virtual IP to the standby HAProxy in case of failure. This is seamless because the two HAProxy instances need no shared state.

In this example, we are using 2 nodes to act as the load balancer with IP failover in front of our database cluster. VIP will be floating around between LB1 (master) and LB2 (backup). When LB1 is down, the VIP will be taking over by LB2 and once the LB1 up again, the VIP will be failback to LB1 since it hold the higher priority number. 

We are using following hosts/IPs:

VIP: 192.168.10.100
LB1: 192.168.10.101
LB2: 192.168.10.102

DB1: 192.168.10.111
DB2: 192.168.10.112
DB3: 192.168.10.113
ClusterControl: 192.168.10.115

You may refer to following diagram for the architecture:

haproxy_keepalived.PNG

 

Install HAproxy

1. Before we start to deploy, make sure LB1 and LB2 are accessible using passwordless SSH. Copy the SSH keys to the load balancer nodes:

$ ssh-copy-id -i ~/.ssh/id_rsa 192.168.10.101
$ ssh-copy-id -i ~/.ssh/id_rsa 192.168.10.102 

2. Install HAproxy into both nodes, select in the UI Manage -> Load Balancer

Screen_Shot_2015-06-03_at_17.38.14.png

Click "Install HAProxy" when you are happy with the settings. The HAProxy Configuration template is stored on the controller in /usr/share/cmon/templates/haproxy.cfg   and  in that directory you also have a the template for the mysqlchk script.

3. You will noticed that these 2 load balancer nodes have been installed and provisioned by ClusterControl. You can verify this by login into ClusterControl > Nodes and you should see similar screenshot as below:

haproxy.png

 

Install Keepalived

Requires that you have two load balancers installed

1. Navigate to Manage -> Load Balancer, and select the tab Keepalived.

Screen_Shot_2015-06-03_at_17.43.23.png

Installation completed! You can now access your database servers through VIP, 192.168.10.100 port 33306.

1

Comments

26 comments

Sort by Date Votes
  • Avatar
    JOE YU

    How to avoid brain split in the above keepalived configuration?

    eg. when the communication is broken between keepalived master and slave host ?

    0
    Comment actions Permalink
  • Avatar
    Ashraf Sharif

    You can refer to following pages for detailed explanation on how to avoid Keepalived split-brain:

    http://scale-out-blog.blogspot.com/2011/01/virtual-ip-addresses-and-their.html

    http://www.austintek.com/LVS/LVS-HOWTO/HOWTO/LVS-HOWTO.failover.html

    0
    Comment actions Permalink
  • Avatar
    JOE YU

    Thanks for reply.

    Have read these two docs already. Still can't get clue to solve the problem without using pacemaker or other component to make a complex architect.

    Not sure if using the same state (Like BACKUP) and same priority like (100) in both keepalived.conf  can avoid the brain split of keepalived.

     

    Thanks,

    0
    Comment actions Permalink
  • Avatar
    Aiman Farhat

    Hi Ashraf,

    Thanks for this nice article, I am having oneissue and can't figure it out. The virtual IP gets assigned to the master and on fail over the VIP get's assigned to the backup, but the issue is I can't ping the IP Address (10.134.41.180)  from the backup or externally. 

     ip addr show:

    eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:50:56:a8:0d:cf brd ff:ff:ff:ff:ff:ff
    inet 10.134.41.103/25 brd 10.134.41.127 scope global eth0
    inet 10.134.41.180/32 scope global eth0
    inet6 fe80::250:56ff:fea8:dcf/64 scope link
    valid_lft forever preferred_lft forever

     

    Thanks for your help

    Aiman

    0
    Comment actions Permalink
  • Avatar
    Ashraf Sharif

    Hi Aiman,

     

    Try to check firewall setting on backup host and ARP table on client host. You can use command "arp -an" to verify the latest virtual IP mapping. It should be mapped to the backup host's MAC address. Depending on your router or switch, you might face "arp cache problem" if the virtual IP has been failover but not updated in your client's ARP table.

    0
    Comment actions Permalink
  • Avatar
    Aiman Farhat

    Hi Ashraf,

       Thank you for the speedy reply. It turn out that the VIP I am using it is not on the same subnet. I got it working now.

    Thanks again.

    Aiman

    0
    Comment actions Permalink
  • Avatar
    Shaun Botsis

    Hi Ashraf

     

    I get the following error when trying to provision HAProxy to a vanilla debian7 install.

     

    ll# ./s9s-admin/cluster/s9s_haproxy --install -i 1 -h 172.16.200.48
    cmon12341
    load opts 1
    Testing ssh to 172.16.200.48: ssh -q -p22 -o UserKnownHostsFile=/dev/null -o Str                    ictHostKeyChecking=no -oNumberOfPasswordPrompts=0 -oConnectTimeout=10 -oIdentity                    File=/root/.ssh/id_rsa -oNumberOfPasswordPrompts=0 root@172.16.200.48  ls -al /u                    sr
    [ok]
    Using loadbalancing policy 'leastconn'.
    ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/tmp/mys                    ql.sock' (2)
    No hostnames found.

    What am I doing wrong ?

    Thanks

    0
    Comment actions Permalink
  • Avatar
    Ashraf Sharif

    Hi Shaun,

    Could you update in /usr/bin/s9s_haproxy line 76 to:

    chmod 644 $MYCNF_CMON

    Then try again the HAproxy deployment. Please verify if it is working.

    Regards,
    Ashraf

    0
    Comment actions Permalink
  • Avatar
    Shaun Botsis

    hi Ashraf

    I get exactly the same response after updating

    chmod 600 $MYCNF_CMON >> chmod 644 $MYCNF_CMON

    0
    Comment actions Permalink
  • Avatar
    Ashraf Sharif

    Hi Shaun,

    Can you edit /usr/bin/s9s_haproxy starting from line 71:

    [mysql_cmon]
    user=cmon
    password=
    EOF

    to:

    [mysql_cmon]
    host=127.0.0.1
    port=3306
    user=cmon
    password=
    EOF

     

    Then save and try again the deployment. It seems like CMON didn't use correct credentials when connecting to CMON DB.

     

    Regards,

    Ashraf

    0
    Comment actions Permalink
  • Avatar
    Ashraf Sharif

    Hi Shaun,

    Can you edit /usr/bin/s9s_haproxy starting from line 71:

    [mysql_cmon]
    user=cmon
    password=
    EOF

    to:

    [mysql_cmon]
    host=127.0.0.1
    port=3306
    user=cmon
    password=
    EOF

     

    Then save and try again the deployment. It seems like CMON didn't use correct credentials when connecting to CMON DB.

     

    Regards,

    Ashraf

    0
    Comment actions Permalink
  • Avatar
    Shaun Botsis

    Thanks for the help Ashraf :)  I got it working. 

     

    There was one more issue relating to the package manager not having the HAproxy package available.

    On my Debian 7 install I had to add deb http://ftp.debian.org/debian/ wheezy-backports main  to /etc/apt/sources.list.

    0
    Comment actions Permalink
  • Avatar
    Ashraf Sharif

    Hi Shaun,

    Yes, we are aware of that. Certain package managers do not have Haproxy in their repository, so installing with "use source" option inside ClusterControl UI would solve this. 

    Regards,

    Ashraf

    0
    Comment actions Permalink
  • Avatar
    Rafael Oliveira

    Hi guys,

    After following all steps to get Keepalived installed and configured, I realized that it is assigning the VIP to my slave LB server when the master is online yet.

    I tried restarting keepalived service from both LB servers but after they are back online, VIP gets assigned to both again.

    What would be the reason for that to be happening?

    0
    Comment actions Permalink
  • Avatar
    Baptiste Assmann

    Hi Rafael,

    Can you check the message reported by keepalived in your syslog?
    There may be some interesting hints inside. Also, running a tcpdump and capture the VRRP traffic on both hosts may help.

    Baptiste

    0
    Comment actions Permalink
  • Avatar
    Rafael Oliveira

    Hi Baptiste, thank you in advance for your interest on helping. Here it goes:

    LB1:

    [root@mktapps-lb1 centos]# tail -f /var/log/messages
    Jan 30 13:25:48 mktapps-lb1 Keepalived_healthcheckers[10278]: Opening file '/etc/keepalived/keepalived.conf'.
    Jan 30 13:25:48 mktapps-lb1 Keepalived_healthcheckers[10278]: Configuration is using : 7273 Bytes
    Jan 30 13:25:48 mktapps-lb1 Keepalived_healthcheckers[10278]: Using LinkWatch kernel netlink reflector...
    Jan 30 13:25:49 mktapps-lb1 Keepalived_vrrp[10279]: VRRP_Instance(VI_1) Transition to MASTER STATE
    Jan 30 13:25:50 mktapps-lb1 Keepalived_vrrp[10279]: VRRP_Instance(VI_1) Entering MASTER STATE
    Jan 30 13:25:50 mktapps-lb1 Keepalived_vrrp[10279]: VRRP_Instance(VI_1) setting protocol VIPs.
    Jan 30 13:25:50 mktapps-lb1 Keepalived_vrrp[10279]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.10.25
    Jan 30 13:25:50 mktapps-lb1 Keepalived_healthcheckers[10278]: Netlink reflector reports IP 192.168.10.25 added
    Jan 30 13:25:52 mktapps-lb1 ntpd[329]: Listen normally on 14 eth0 192.168.10.25 UDP 123
    Jan 30 13:25:55 mktapps-lb1 Keepalived_vrrp[10279]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.10.25

     

    LB2:

    [root@mktapps-lb2 centos]# tail -f /var/log/messages
    Jan 30 13:25:46 mktapps-lb2 avahi-daemon[325]: Registering new address record for 192.168.10.25 on eth0.IPv4.
    Jan 30 13:25:47 mktapps-lb2 ntpd[382]: Listen normally on 13 eth0 192.168.10.25 UDP 123
    Jan 30 13:25:51 mktapps-lb2 Keepalived_vrrp[10226]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.10.25
    Jan 30 13:36:29 mktapps-lb2 systemd: Stopping LVS and VRRP High Availability Monitor...
    Jan 30 13:36:29 mktapps-lb2 Keepalived[10224]: Stopping Keepalived v1.2.10 (06/10,2014)
    Jan 30 13:36:29 mktapps-lb2 Keepalived_vrrp[10226]: VRRP_Instance(VI_1) sending 0 priority
    Jan 30 13:36:29 mktapps-lb2 Keepalived_vrrp[10226]: VRRP_Instance(VI_1) removing protocol VIPs.
    Jan 30 13:36:29 mktapps-lb2 systemd: Stopped LVS and VRRP High Availability Monitor.
    Jan 30 13:36:29 mktapps-lb2 avahi-daemon[325]: Withdrawing address record for 192.168.10.25 on eth0.
    Jan 30 13:36:31 mktapps-lb2 ntpd[382]: Deleting interface #13 eth0, 192.168.10.25#123, interface stats: received=0, sent=0, dropped=0, active_time=644 secs

    0
    Comment actions Permalink
  • Avatar
    Rafael Oliveira

    I found an article online that says that by default keepalived uses 224.0.0.18 IP address for VRRP (Virtual Router Redundancy Protocol) for communication between two nodes for health check. So I ran tcpdump as follows on eth0 (please correct if this information is wrong):

    Commands: tcpdump -v -i eth0 host 224.0.0.18
                           tcpdump -vvv -n -i eth0 host 224.0.0.18

     

    LB1:

    [root@mktapps-lb1 centos]# tcpdump -v -i eth0 host 224.0.0.18
    tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
    13:47:38.506958 IP (tos 0xc0, ttl 255, id 1309, offset 0, flags [none], proto VRRP (112), length 40)
    host-192-168-10-19.cisco.com > vrrp.mcast.net: vrrp host-192-168-10-19.cisco.com > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 101, authtype none, intvl 1s, length 20, addrs: mktapps-lb1.novalocal
    13:47:39.507504 IP (tos 0xc0, ttl 255, id 1310, offset 0, flags [none], proto VRRP (112), length 40)
    host-192-168-10-19.cisco.com > vrrp.mcast.net: vrrp host-192-168-10-19.cisco.com > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 101, authtype none, intvl 1s, length 20, addrs: mktapps-lb1.novalocal
    13:47:40.508707 IP (tos 0xc0, ttl 255, id 1311, offset 0, flags [none], proto VRRP (112), length 40)
    host-192-168-10-19.cisco.com > vrrp.mcast.net: vrrp host-192-168-10-19.cisco.com > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 101, authtype none, intvl 1s, length 20, addrs: mktapps-lb1.novalocal
    13:47:41.509283 IP (tos 0xc0, ttl 255, id 1312, offset 0, flags [none], proto VRRP (112), length 40)
    host-192-168-10-19.cisco.com > vrrp.mcast.net: vrrp host-192-168-10-19.cisco.com > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 101, authtype none, intvl 1s, length 20, addrs: mktapps-lb1.novalocal
    13:47:42.510433 IP (tos 0xc0, ttl 255, id 1313, offset 0, flags [none], proto VRRP (112), length 40)
    host-192-168-10-19.cisco.com > vrrp.mcast.net: vrrp host-192-168-10-19.cisco.com > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 101, authtype none, intvl 1s, length 20, addrs: mktapps-lb1.novalocal
    13:47:43.511006 IP (tos 0xc0, ttl 255, id 1314, offset 0, flags [none], proto VRRP (112), length 40)
    host-192-168-10-19.cisco.com > vrrp.mcast.net: vrrp host-192-168-10-19.cisco.com > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 101, authtype none, intvl 1s, length 20, addrs: mktapps-lb1.novalocal
    ^C
    6 packets captured
    6 packets received by filter
    0 packets dropped by kernel
    [root@mktapps-lb1 centos]# tcpdump -vvv -n -i eth0 host 224.0.0.18
    tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
    13:47:56.522951 IP (tos 0xc0, ttl 255, id 1327, offset 0, flags [none], proto VRRP (112), length 40)
    192.168.10.19 > 224.0.0.18: vrrp 192.168.10.19 > 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 101, authtype none, intvl 1s, length 20, addrs: 192.168.10.25
    13:47:57.523510 IP (tos 0xc0, ttl 255, id 1328, offset 0, flags [none], proto VRRP (112), length 40)
    192.168.10.19 > 224.0.0.18: vrrp 192.168.10.19 > 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 101, authtype none, intvl 1s, length 20, addrs: 192.168.10.25
    13:47:58.524739 IP (tos 0xc0, ttl 255, id 1329, offset 0, flags [none], proto VRRP (112), length 40)
    192.168.10.19 > 224.0.0.18: vrrp 192.168.10.19 > 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 101, authtype none, intvl 1s, length 20, addrs: 192.168.10.25
    13:47:59.525325 IP (tos 0xc0, ttl 255, id 1330, offset 0, flags [none], proto VRRP (112), length 40)
    192.168.10.19 > 224.0.0.18: vrrp 192.168.10.19 > 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 101, authtype none, intvl 1s, length 20, addrs: 192.168.10.25
    ^C
    4 packets captured
    4 packets received by filter
    0 packets dropped by kernel
    [root@mktapps-lb1 centos]#

     

    LB2:

    [root@mktapps-lb2 centos]# tcpdump -v -i eth0 host 224.0.0.18
    tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
    13:47:06.324102 IP (tos 0xc0, ttl 255, id 3, offset 0, flags [none], proto VRRP (112), length 40)
    host-192-168-10-20.cisco.com > vrrp.mcast.net: vrrp host-192-168-10-20.cisco.com > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 100, authtype none, intvl 1s, length 20, addrs: mktapps-lb2.novalocal
    13:47:07.324438 IP (tos 0xc0, ttl 255, id 4, offset 0, flags [none], proto VRRP (112), length 40)
    host-192-168-10-20.cisco.com > vrrp.mcast.net: vrrp host-192-168-10-20.cisco.com > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 100, authtype none, intvl 1s, length 20, addrs: mktapps-lb2.novalocal
    13:47:08.325564 IP (tos 0xc0, ttl 255, id 5, offset 0, flags [none], proto VRRP (112), length 40)
    host-192-168-10-20.cisco.com > vrrp.mcast.net: vrrp host-192-168-10-20.cisco.com > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 100, authtype none, intvl 1s, length 20, addrs: mktapps-lb2.novalocal
    13:47:09.325779 IP (tos 0xc0, ttl 255, id 6, offset 0, flags [none], proto VRRP (112), length 40)
    host-192-168-10-20.cisco.com > vrrp.mcast.net: vrrp host-192-168-10-20.cisco.com > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 100, authtype none, intvl 1s, length 20, addrs: mktapps-lb2.novalocal
    13:47:10.325956 IP (tos 0xc0, ttl 255, id 7, offset 0, flags [none], proto VRRP (112), length 40)
    host-192-168-10-20.cisco.com > vrrp.mcast.net: vrrp host-192-168-10-20.cisco.com > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 100, authtype none, intvl 1s, length 20, addrs: mktapps-lb2.novalocal
    13:47:11.326285 IP (tos 0xc0, ttl 255, id 8, offset 0, flags [none], proto VRRP (112), length 40)
    host-192-168-10-20.cisco.com > vrrp.mcast.net: vrrp host-192-168-10-20.cisco.com > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 100, authtype none, intvl 1s, length 20, addrs: mktapps-lb2.novalocal
    ^C
    6 packets captured
    6 packets received by filter
    0 packets dropped by kernel
    [root@mktapps-lb2 centos]# tcpdump -vvv -n -i eth0 host 224.0.0.18
    tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
    13:48:07.367359 IP (tos 0xc0, ttl 255, id 64, offset 0, flags [none], proto VRRP (112), length 40)
    192.168.10.20 > 224.0.0.18: vrrp 192.168.10.20 > 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 100, authtype none, intvl 1s, length 20, addrs: 192.168.10.25
    13:48:08.368530 IP (tos 0xc0, ttl 255, id 65, offset 0, flags [none], proto VRRP (112), length 40)
    192.168.10.20 > 224.0.0.18: vrrp 192.168.10.20 > 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 100, authtype none, intvl 1s, length 20, addrs: 192.168.10.25
    13:48:09.368680 IP (tos 0xc0, ttl 255, id 66, offset 0, flags [none], proto VRRP (112), length 40)
    192.168.10.20 > 224.0.0.18: vrrp 192.168.10.20 > 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 100, authtype none, intvl 1s, length 20, addrs: 192.168.10.25
    13:48:10.369827 IP (tos 0xc0, ttl 255, id 67, offset 0, flags [none], proto VRRP (112), length 40)
    192.168.10.20 > 224.0.0.18: vrrp 192.168.10.20 > 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 100, authtype none, intvl 1s, length 20, addrs: 192.168.10.25
    ^C
    4 packets captured
    4 packets received by filter
    0 packets dropped by kernel
    [root@mktapps-lb2 centos]#

     

    0
    Comment actions Permalink
  • Avatar
    Rafael Oliveira

    These are the log messages right after I stop/start keepalived:

    LB1:
    [root@mktapps-lb1 centos]# tail -f /var/log/messages
    Jan 30 14:01:35 mktapps-lb1 Keepalived_vrrp[12567]: Registering Kernel netlink reflector
    Jan 30 14:01:35 mktapps-lb1 Keepalived_vrrp[12567]: Registering Kernel netlink command channel
    Jan 30 14:01:35 mktapps-lb1 Keepalived_vrrp[12567]: Registering gratuitous ARP shared channel
    Jan 30 14:01:35 mktapps-lb1 Keepalived_vrrp[12567]: Opening file '/etc/keepalived/keepalived.conf'.
    Jan 30 14:01:35 mktapps-lb1 Keepalived_vrrp[12567]: Configuration is using : 64418 Bytes
    Jan 30 14:01:35 mktapps-lb1 Keepalived_vrrp[12567]: Using LinkWatch kernel netlink reflector...
    Jan 30 14:01:35 mktapps-lb1 Keepalived_vrrp[12567]: VRRP sockpool: [ifindex(2), proto(112), unicast(0), fd(10,11)]
    Jan 30 14:01:35 mktapps-lb1 Keepalived_healthcheckers[12566]: Opening file '/etc/keepalived/keepalived.conf'.
    Jan 30 14:01:35 mktapps-lb1 Keepalived_healthcheckers[12566]: Configuration is using : 7273 Bytes
    Jan 30 14:01:35 mktapps-lb1 Keepalived_healthcheckers[12566]: Using LinkWatch kernel netlink reflector...
    Jan 30 14:01:36 mktapps-lb1 Keepalived_vrrp[12567]: VRRP_Instance(VI_1) Transition to MASTER STATE
    Jan 30 14:01:37 mktapps-lb1 Keepalived_vrrp[12567]: VRRP_Instance(VI_1) Entering MASTER STATE
    Jan 30 14:01:37 mktapps-lb1 Keepalived_vrrp[12567]: VRRP_Instance(VI_1) setting protocol VIPs.
    Jan 30 14:01:37 mktapps-lb1 Keepalived_vrrp[12567]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.10.25
    Jan 30 14:01:37 mktapps-lb1 Keepalived_healthcheckers[12566]: Netlink reflector reports IP 192.168.10.25 added
    Jan 30 14:01:39 mktapps-lb1 ntpd[329]: Listen normally on 16 eth0 192.168.10.25 UDP 123
    Jan 30 14:01:42 mktapps-lb1 Keepalived_vrrp[12567]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.10.25

     

    LB2:
    [root@mktapps-lb2 centos]# tail -f /var/log/messages
    Jan 30 14:02:18 mktapps-lb2 Keepalived_healthcheckers[11931]: Netlink reflector reports IP fe80::f816:3eff:fe25:92c1 added
    Jan 30 14:02:18 mktapps-lb2 Keepalived_healthcheckers[11931]: Registering Kernel netlink reflector
    Jan 30 14:02:18 mktapps-lb2 Keepalived_healthcheckers[11931]: Registering Kernel netlink command channel
    Jan 30 14:02:18 mktapps-lb2 Keepalived_healthcheckers[11931]: Opening file '/etc/keepalived/keepalived.conf'.
    Jan 30 14:02:18 mktapps-lb2 Keepalived_healthcheckers[11931]: Configuration is using : 7263 Bytes
    Jan 30 14:02:18 mktapps-lb2 Keepalived_vrrp[11932]: Opening file '/etc/keepalived/keepalived.conf'.
    Jan 30 14:02:18 mktapps-lb2 Keepalived_vrrp[11932]: Configuration is using : 64408 Bytes
    Jan 30 14:02:18 mktapps-lb2 Keepalived_vrrp[11932]: Using LinkWatch kernel netlink reflector...
    Jan 30 14:02:18 mktapps-lb2 Keepalived_vrrp[11932]: VRRP sockpool: [ifindex(2), proto(112), unicast(0), fd(10,11)]
    Jan 30 14:02:18 mktapps-lb2 Keepalived_healthcheckers[11931]: Using LinkWatch kernel netlink reflector...
    Jan 30 14:02:19 mktapps-lb2 Keepalived_vrrp[11932]: VRRP_Instance(VI_1) Transition to MASTER STATE
    Jan 30 14:02:20 mktapps-lb2 Keepalived_vrrp[11932]: VRRP_Instance(VI_1) Entering MASTER STATE
    Jan 30 14:02:20 mktapps-lb2 Keepalived_vrrp[11932]: VRRP_Instance(VI_1) setting protocol VIPs.
    Jan 30 14:02:20 mktapps-lb2 Keepalived_vrrp[11932]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.10.25
    Jan 30 14:02:20 mktapps-lb2 Keepalived_healthcheckers[11931]: Netlink reflector reports IP 192.168.10.25 added
    Jan 30 14:02:20 mktapps-lb2 avahi-daemon[325]: Registering new address record for 192.168.10.25 on eth0.IPv4.
    Jan 30 14:02:21 mktapps-lb2 ntpd[382]: Listen normally on 17 eth0 192.168.10.25 UDP 123
    Jan 30 14:02:25 mktapps-lb2 Keepalived_vrrp[11932]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.10.25

     

    0
    Comment actions Permalink
  • Avatar
    Ashraf Sharif

    Hi Rafael,

    Did you have iptables enabled between LB1 and LB2? If yes, please add following rules:

    iptables -I INPUT -i eth0 -d 224.0.0.0/8 -j ACCEPT
iptables -I INPUT -p 112 -i eth0 -j ACCEPT
iptables -I OUTPUT -p 112 -o eth0 -j ACCEPT

     

    Please also send a complete log of Keepalived:

    $ grep -i keepalived /var/log/messages

     

    Regards,

    Ashraf

    0
    Comment actions Permalink
  • Avatar
    Rafael Oliveira

    Hi Ashraf,

    I am using centOS 7 in both machines and I haven't configured anything on iptables. Before running iptables commands, do I need to enable/start any service?

    Keepalived logs attached.

     

    0
    Comment actions Permalink
  • Avatar
    Rafael Oliveira

    I have installed iptables-services:

    # yum -y install iptables-services

    And ran the commmands for iptables Ashraf posted.

    However, after restarting keepalived, still get same results (VIP being assigned to both machines at same time).

    0
    Comment actions Permalink
  • Avatar
    Ashraf Sharif

    Hi Rafael,

    I would say run the iptables command first and restart keepalived on both hosts. Ensure selinux is turned off or set to permissive mode. We haven't really test the Keepalived deployment on CentOS 7 at the moment. If the problem still persists, please attach the output of "iptables -L -n" and keepalived.conf from both hosts.

     

    Regards,

    Ashraf

    0
    Comment actions Permalink
  • Avatar
    Rafael Oliveira

    Hi Ashraf,

    SElinux is disabled on both servers:

    ---

    [root@mktapps-lb1 centos]# sestatus
    SELinux status: disabled
    [root@mktapps-lb1 centos]#

    ---

    “iptables -L -n" from both servers (same values):

    Chain INPUT (policy ACCEPT)
    target prot opt source destination
    ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED
    ACCEPT icmp -- 0.0.0.0/0 0.0.0.0/0
    ACCEPT all -- 0.0.0.0/0 0.0.0.0/0
    ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:22
    REJECT all -- 0.0.0.0/0 0.0.0.0/0 reject-with icmp-host-prohibited

    Chain FORWARD (policy ACCEPT)
    target prot opt source destination
    REJECT all -- 0.0.0.0/0 0.0.0.0/0 reject-with icmp-host-prohibited

    Chain OUTPUT (policy ACCEPT)
    target prot opt source destination
    [root@mktapps-lb2 centos]#

     

    LB1 keepalived.conf:

    ! Configuration File for keepalived
    global_defs {
    notification_email {
    raolivei@domain.com
    }
    notification_email_from LB_NODE1@domain.com
    smtp_server sjc-mail-00.domain.com
    smtp_connect_timeout 30
    }
    vrrp_script chk_haproxy {
    script "killall -0 haproxy" # verify the pid existance
    interval 2 # check every 2 seconds
    weight 2 # add 2 points of prio if OK
    }

    vrrp_instance VI_1 {
    interface eth0 # interface to monitor
    state MASTER
    virtual_router_id 51 # Assign one ID for this route
    priority 101 # 101 on master, 100 on backup
    virtual_ipaddress {
    192.168.10.25 # the virtual IP
    }
    track_script {
    chk_haproxy
    }
    }

     

    LB2 keepalived.conf:

    ! Configuration File for keepalived
    global_defs {
    notification_email {
    raolivei@domain.com
    }
    notification_email_from LB_NODE@domain.com
    smtp_server sjc-mail-00.domain.com
    smtp_connect_timeout 30
    }
    vrrp_script chk_haproxy {
    script "killall -0 haproxy" # verify the pid existance
    interval 2 # check every 2 seconds
    weight 2 # add 2 points of prio if OK
    }

    vrrp_instance VI_1 {
    interface eth0 # interface to monitor
    state MASTER
    virtual_router_id 51 # Assign one ID for this route
    priority 100 # 101 on master, 100 on backup
    virtual_ipaddress {
    192.168.10.25 # the virtual IP
    }
    track_script {
    chk_haproxy

    }

    }

    0
    Comment actions Permalink
  • Avatar
    Ashraf Sharif

    Hi Rafael,

    It seems multicast environment doesn't really work for you. I would suggest you to disable iptables completely and try with unicast instead. On keepalived.conf apply following config:

    LB1 (192.168.10.19):

    vrrp_script chk_haproxy {
   script "killall -0 haproxy"   # verify the pid existance
   interval 2                    # check every 2 seconds
   weight 2                      # add 2 points of prio if OK
}
 
vrrp_instance VI_1 {
   interface eth0                # interface to monitor
   state MASTER
   virtual_router_id 51          # Assign one ID for this route
   priority 101                  # 101 on master, 100 on backup
   virtual_ipaddress {
	192.168.10.25		# the virtual IP
   }
   unicast_src_ip 192.168.10.19
   unicast_peer {
	192.168.10.20
   }
   track_script {
       chk_haproxy
   }
}

    LB2 (192.168.10.20):

    vrrp_script chk_haproxy {
   script "killall -0 haproxy"   # verify the pid existance
   interval 2                    # check every 2 seconds
   weight 2                      # add 2 points of prio if OK
}
 
vrrp_instance VI_1 {
   interface eth0                # interface to monitor
   state MASTER
   virtual_router_id 51          # Assign one ID for this route
   priority 100                  # 101 on master, 100 on backup
   virtual_ipaddress {
       192.168.10.25            # the virtual IP
   }
   unicast_src_ip 192.168.10.20
   unicast_peer {
	192.168.10.19
   }
   track_script {
       chk_haproxy
   }
}

     

    Notice the unicast_src_ip and unicast_peer added to the config. 

    Regards,

    Ashraf

    0
    Comment actions Permalink
  • Avatar
    Mauricio Querves

    Hello All...

     

    I've installed keepalived on a RHEL 7 and had the same problem as Rafael Oliveira

    What I did was to disable Firewall and it get Working ... so I have to enable the UDP port 123 on the servers that I've installed Keepalive.

    I wait that this could be helpull to someone else.

    Regards,

     

    Mauricio.

     

    0
    Comment actions Permalink
  • Avatar
    Mauricio Querves

    Something esle:

    I have added this rules to the FW (both keepalived nodes):

    iptables -I INPUT -d 224.0.0.0/8 -j ACCEPT
    iptables -I INPUT -p vrrp -j ACCEPT

    0
    Comment actions Permalink

Please sign in to leave a comment.

Didn't find what you were looking for?

New post
Powered by Zendesk