SST with Xtrabackup

Comments

7 comments

  • Avatar
    Johan

    Hi,

    Never seen the wsrep_sst_common.

    If you are using the Codership galera for MySQL (not PXC), then you must download:

    https://launchpad.net/codership-mysql/5.5/5.5.28-23.7/+download/wsrep_sst_xtrabackup

    And place it in /usr/local/mysql/bin  (or whatever your mysql bin directory is).

    Then , are you sure you spelled it out correctly in my.cnf?

    wsrep_sst_method=xtrabackup

    and it looks like you have set "wsrep_sst_extrabackup" (an additional 'e') .

     

    Best regards,

    Johan

     

    0
    Comment actions Permalink
  • Avatar
    Jdd92

    That was a mistype on my part, "wsrep_sst_xtrabackup" was indeed the value I had set in my.cnf. Here is what I see in the logs after stopping the MySQL daemon and letting CMON do its thing:

     

    130122 11:10:50 [Note] WSREP: Setting initial position to 00000000-0000-0000-0000-000000000000:-1
    130122 11:10:50 [Note] WSREP: protonet asio version 0
    130122 11:10:50 [Note] WSREP: backend: asio
    130122 11:10:50 [Note] WSREP: GMCast version 0
    130122 11:10:50 [Note] WSREP: (abd3b13c-64b6-11e2-0800-c8990a35fc8a, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567
    130122 11:10:50 [Note] WSREP: (abd3b13c-64b6-11e2-0800-c8990a35fc8a, 'tcp://0.0.0.0:4567') multicast: , ttl: 1
    130122 11:10:50 [Note] WSREP: EVS version 0
    130122 11:10:50 [Note] WSREP: PC version 0
    130122 11:10:50 [Note] WSREP: gcomm: connecting to group 'my_wsrep_cluster', peer '10.26.138.203:4567'
    130122 11:10:50 [Note] WSREP: (abd3b13c-64b6-11e2-0800-c8990a35fc8a, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://10.26.138.204:4567
    130122 11:10:50 [Note] WSREP: (abd3b13c-64b6-11e2-0800-c8990a35fc8a, 'tcp://0.0.0.0:4567') turning message relay requesting off
    130122 11:10:50 [Note] WSREP: declaring 178d3958-6295-11e2-0800-a811d80dd113 stable
    130122 11:10:50 [Note] WSREP: declaring a8305e44-629f-11e2-0800-6963d41743cb stable
    130122 11:10:50 [Note] WSREP: view(view_id(PRIM,178d3958-6295-11e2-0800-a811d80dd113,89) memb {
    178d3958-6295-11e2-0800-a811d80dd113,
    a8305e44-629f-11e2-0800-6963d41743cb,
    abd3b13c-64b6-11e2-0800-c8990a35fc8a,
    } joined {
    } left {
    } partitioned {
    })
    130122 11:10:51 [Note] WSREP: gcomm: connected
    130122 11:10:51 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636
    130122 11:10:51 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0)
    130122 11:10:51 [Note] WSREP: Opened channel 'my_wsrep_cluster'
    130122 11:10:51 [Note] WSREP: Waiting for SST to complete.
    130122 11:10:51 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 2, memb_num = 3
    130122 11:10:51 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
    130122 11:10:51 [Note] WSREP: STATE EXCHANGE: sent state msg: ac22709d-64b6-11e2-0800-5bf7f496cccf
    130122 11:10:51 [Note] WSREP: STATE EXCHANGE: got state msg: ac22709d-64b6-11e2-0800-5bf7f496cccf from 0 (db-4.staging.local)
    130122 11:10:51 [Note] WSREP: STATE EXCHANGE: got state msg: ac22709d-64b6-11e2-0800-5bf7f496cccf from 1 (db-3.staging.local)
    130122 11:10:51 [Note] WSREP: STATE EXCHANGE: got state msg: ac22709d-64b6-11e2-0800-5bf7f496cccf from 2 (db-2.staging.local)
    130122 11:10:51 [Note] WSREP: Quorum results:
    version = 2,
    component = PRIMARY,
    conf_id = 32,
    members = 2/3 (joined/total),
    act_id = 127,
    last_appl. = -1,
    protocols = 0/4/2 (gcs/repl/appl),
    group UUID = 68e3a055-6193-11e2-0800-bfadb8a01a7f
    130122 11:10:51 [Note] WSREP: Flow-control interval: [14, 28]
    130122 11:10:51 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 127)
    130122 11:10:51 [Note] WSREP: State transfer required:
    Group state: 68e3a055-6193-11e2-0800-bfadb8a01a7f:127
    Local state: 68e3a055-6193-11e2-0800-bfadb8a01a7f:-1
    130122 11:10:51 [Note] WSREP: New cluster view: global state: 68e3a055-6193-11e2-0800-bfadb8a01a7f:127, view# 33: Primary, number of nodes: 3, my index: 2, protocol version 2
    130122 11:10:51 [Warning] WSREP: Gap in state sequence. Need state transfer.
    130122 11:10:53 [Note] WSREP: Running: 'wsrep_sst_xtrabackup 'joiner' '10.26.138.202' 'root:<somepassword>' '/gato/mysql/' '/etc/my.cnf' '10164' 2>sst.err'
    130122 11:10:53 [ERROR] WSREP: Failed to read 'ready <addr>' from: wsrep_sst_xtrabackup 'joiner' '10.26.138.202' 'root:<somepassword>' '/gato/mysql/' '/etc/my.cnf' '10164' 2>sst.err
    Read: '(null)'
    130122 11:10:53 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup 'joiner' '10.26.138.202' 'root:<somepassword>' '/gato/mysql/' '/etc/my.cnf' '10164' 2>sst.err: 13 (Permission denied)
    130122 11:10:53 [ERROR] WSREP: Failed to prepare for 'xtrabackup' SST. Unrecoverable.
    130122 11:10:53 [ERROR] Aborting

    Initially I presumed that this could be a file permissions issue but even after making mysql the owner of the file and setting the execution permissions to be quite permissive I still see this error.

    Looking at the sst.err log file, all I see is:

    /usr//bin/wsrep_sst_xtrabackup: line 21: /usr//bin/wsrep_sst_common: No such file or directory

    0
    Comment actions Permalink
  • Avatar
    Johan

    In the .tar.gz there is a wsrep_sst_common file:

    tar tfvz mysql-5.5.28_wsrep_23.7-linux-x86_64.tar.gz |grep wsrep_sst_common
    -rwxr-xr-x devel/devel 2467 2012-11-28 22:07 mysql-5.5.28_wsrep_23.7-linux-x86_64/bin/wsrep_sst_common

     

    The RPM based too:

    MySQL-server-5.5.28_wsrep_23.7-1.rhel5.x86_64.rpm  :  ./usr/bin/wsrep_sst_common

     

    What version are you using of Galera?

    Best regards,

    Johan

    0
    Comment actions Permalink
  • Avatar
    Jdd92

    rpm -qa | grep MySQL
    MySQL-server-5.5.23_wsrep_23.6-1.rhel5.x86_64
    MySQL-client-5.5.28-1.el6.x86_64

    Server version: 5.5.23 MySQL Community Server (GPL), wsrep_23.6.r3755, wsrep_23.6.r3755

    Looks like this file is missing from my install.

    [root@db-monitor][/root/s9s-galera-2.1.0-rpm/mysql/repo]-> rpm -qlp MySQL-server-5.5.23_wsrep_23.6-1.rhel5.x86_64.rpm | grep wsrep
    /etc/wsrep.cnf
    /usr/bin/wsrep_sst_mysqldump
    /usr/bin/wsrep_sst_rsync
    /usr/bin/wsrep_sst_rsync_wan
    /usr/share/doc/MySQL-server-5.5.23_wsrep_23.6
    /usr/share/doc/MySQL-server-5.5.23_wsrep_23.6/COPYING
    /usr/share/doc/MySQL-server-5.5.23_wsrep_23.6/ChangeLog
    /usr/share/doc/MySQL-server-5.5.23_wsrep_23.6/INFO_BIN
    /usr/share/doc/MySQL-server-5.5.23_wsrep_23.6/INFO_SRC
    /usr/share/doc/MySQL-server-5.5.23_wsrep_23.6/README
    /usr/share/doc/MySQL-server-5.5.23_wsrep_23.6/README-wsrep
    /usr/share/doc/MySQL-server-5.5.23_wsrep_23.6/my-huge.cnf
    /usr/share/doc/MySQL-server-5.5.23_wsrep_23.6/my-innodb-heavy-4G.cnf
    /usr/share/doc/MySQL-server-5.5.23_wsrep_23.6/my-large.cnf
    /usr/share/doc/MySQL-server-5.5.23_wsrep_23.6/my-medium.cnf
    /usr/share/doc/MySQL-server-5.5.23_wsrep_23.6/my-small.cnf
    /usr/share/doc/MySQL-server-5.5.23_wsrep_23.6/wsrep.cnf
    /usr/share/doc/MySQL-server-5.5.23_wsrep_23.6/wsrep_notify
    /usr/share/mysql/wsrep.cnf
    /usr/share/mysql/wsrep_notify

     

    0
    Comment actions Permalink
  • Avatar
    Johan

    Aha, it seems that this one is for Galera 2.2.0 (you are on Galera 2.1) so you need to upgrade.

    Thanks,

    Johan

    0
    Comment actions Permalink
  • Avatar
    Jdd92

    Yup, that was the ticket. I did a rolling update across the cluster and then setup xtrabackup as the SST method. Did some more testing and now it seems to be working.

    0
    Comment actions Permalink
  • Avatar
    Johan

    Awesome!

    - Johan

    0
    Comment actions Permalink

Please sign in to leave a comment.

Powered by Zendesk