Failed to add replication slave.

Follow

Michael

• July 30, 2015 14:44

Could not start '/usr/bin/innobackupex --host=127.0.0.1 --user=cmon --password=cmon --stream=xbstream /data/ 2>/home/ec2-user/s9s_tmp/1/galera/backup.log | nc 10.10.2.186 9999' on 10.10.2.25. Exit Code: 1	9613	29 Jul 2015 11:26:54
Streaming backup from 10.10.2.25 to 10.10.2.186 Exit Code: 0	9613	24 Jul 2015 11:20:30
10.10.2.25 running: '/usr/bin/innobackupex --host=127.0.0.1 --user=cmon --password=********* --stream=xbstream /data/ 2>/home/ec2-user/s9s_tmp/1/galera/backup.log | nc 10.10.2.186 9999'

backup.log starts with

 

xtrabackup version 2.2.3 based on MySQL server 5.6.17 Linux (x86_64) (revision id: )
xtrabackup: uses posix_fadvise().
xtrabackup: cd to /data
xtrabackup: open files limit requested 0, set to 1024
xtrabackup: using the following InnoDB configuration:
xtrabackup: innodb_data_home_dir = ./
xtrabackup: innodb_data_file_path = ibdata1:100M:autoextend
xtrabackup: innodb_log_group_home_dir = ./
xtrabackup: innodb_log_files_in_group = 2
xtrabackup: innodb_log_file_size = 536870912
xtrabackup: using O_DIRECT
>> log scanned up to (109068711869)
[01] Streaming ./ibdata1
>> log scanned up to (109068712431)
>> log scanned up to (109068718265)

...

....

>> log scanned up to (109069763643)
>> log scanned up to (109069763656)
>> log scanned up to (109069763656)
>> log scanned up to (109069763656)
>> log scanned up to (109069771391)
>> log scanned up to (109069776191)
xtrabackup: Error writing file 'UNOPENED' (Errcode: 32 - Broken pipe)
xb_stream_write_data() failed.
xtrabackup: Error writing file 'UNOPENED' (Errcode: 32 - Broken pipe)
[01] xtrabackup: Error: xtrabackup_copy_datafile() failed.
[01] xtrabackup: Error: failed to copy datafile.
innobackupex: Error: The xtrabackup child process has died at /usr/bin/innobackupex line 2672.

0

• 

  Johan

  • July 30, 2015 15:50

  Hi,

  do you have enough disk space on 10.10.2.186:/home/ec2-user ? 

  What AMI are you using and what instance size?
  Best regards

  Johan

   

  0

  Comment actions Permalink
• 

  Michael

  • July 30, 2015 17:10

  Should have plenty of disk space.
  
  Database is about 60GB.
  
  [ec2-user@ip-10-10-2-186 ~]$ df -h
  Filesystem Size Used Avail Use% Mounted on
  /dev/xvda1 99G 1.5G 97G 2% /
  devtmpfs 1.9G 64K 1.9G 1% /dev
  tmpfs 1.9G 0 1.9G 0% /dev/shm
  /dev/xvdb 197G 60M 187G 1% /data
  /dev/xvdc 197G 60M 187G 1% /backups
  [ec2-user@ip-10-10-2-186 ~]$
  
  
  Here is a copy of the slave configuration
  
  

  [mysqld]
  bind-address=0.0.0.0
  log_bin=binlog
  log_slave_updates=1
  expire_logs_days=7
  server_id=1001
  binlog_format=ROW
  slave_net_timeout=60
  basedir=/usr
  datadir=/data
  
  
  
  

  i am using an M3.Medium for EC2 Size.

   

  0

  Comment actions Permalink
• 

  Michael

  • July 30, 2015 17:25

  AMI is  amzn-ami-hvm-2015.03.0.x86_64-gp2 

  0

  Comment actions Permalink
• 

  Johan

  • July 30, 2015 17:37

  Thanks. will check.

  
  BR

  johan

  0

  Comment actions Permalink
• 

  Ashraf Sharif

  • July 31, 2015 07:44

  Hi Michael,

  I couldn't replicate the problem with the same setup. According to the innobackup.backup.log provided, you might encounter the same issue as described in this thread: https://www.percona.com/forums/questions-discussions/percona-xtradb-cluster/39187-nodes-don-t-join-after-upgrade-to-xtradb-cluster-5-6-22-72-0-and-xtrabackup-2-2-10

  Can you try to delete /data/.sst directory (on 10.10.2.186) and try again to add the slave from ClusterControl UI?

  Regards,

  Ashraf

  0

  Comment actions Permalink
• 

  Johan

  • July 31, 2015 09:58

  Hi Michael,

  If you on 10.10.2.186 do:

  nc -dl 9999 | xbstream -x -C /data
  
  and on 10.10.2.25 do:
  
  /usr/bin/innobackupex --host=127.0.0.1 --user=cmon --password=cmon --stream=xbstream /data/ 2>/home/ec2-user/s9s_tmp/1/galera/backup.log | nc 10.10.2.186 9999

  What do you get then?

   

  BR

  johan

  0

  Comment actions Permalink
• 

  Michael

  • July 31, 2015 12:15

  [ec2-user@ip-10-10-2-186 ~]$ nc -dl 9999 | xbstream -x -C /data
  nc: Address already in use
  
  
  

  [ec2-user@ip-10-10-2-25 ~]$ /usr/bin/innobackupex --host=127.0.0.1 --user=cmon --password=cmon --stream=xbstream /data/ 2>/home/ec2-user/s9s_tmp/1/galera/backup.log | nc 10.10.2.186 9999
  [ec2-user@ip-10-10-2-25 ~]$

   

   

  both commands run without error 

  
  
  

  0

  Comment actions Permalink
• 

  Michael

  • July 31, 2015 12:16

  Ashraf,
       The drive is empty

   

  [ec2-user@ip-10-10-2-186 data]$ ls -la
  total 8
  drwxr-xr-x 2 mysql root 4096 Jul 30 16:59 .
  dr-xr-xr-x 26 root root 4096 Jul 30 16:53 ..

  0

  Comment actions Permalink

Please sign in to leave a comment.