innobackupex completes, but does not exit

Follow

Kent Williams

• April 18, 2016 11:34

I'm trialing cluster control on a two MariaDB server cluster (master + slave).

Whilst the monitoring appears to work well, I'm having trouble getting a backup to complete successfully.

I configure the backup through the web ui, selecting a full backup using xtrabackup from the slave, and storing the backup on the controller.

I get the same results if I use gzip + single copy thread, or pigz + 8 copy threads (the multi threaded version is much faster!).

innobackupex appears to complete OK which I can tell by looking at the backup.log on the MariaDB slave which has "160418 20:19:26  innobackupex: completed OK!" as the last line, however it has been over an hour since that log message was written yet the process is still sitting there.

The full command is:

Launching 'ulimit -n 256000 && LC_ALL=C /usr/bin/innobackupex --defaults-file=/etc/mysql/my.cnf --parallel 8 --stream=xbstream --no-timestamp . 2>/mnt/dbbackups/tmp/1/mysql_single/backup.log | pigz - | nc 192.168.190.20 9999'

The file on the controller appears to be of the expected size (~12GB zipped), so I'm unsure why the process is sitting around?

Killing the process results in a non-zero exit code, thus the backup job shows up as failed.

0

• 

  Krzysztof Ksiazek

  • April 18, 2016 12:19

  Can you please send us following data:

  1. CMON log files (/var/log/cmon*)

  2. Backup log

  3. Output of mysqldump -ucmon -p cmon cmon_job cmon_job_message mysql_backup > jobs.sql

  4. Output of cmon --version

  If you prefer, feel free to open a support ticket with us. For your information, we've got similar reports in the past and we are already investigating this type of an issue. This data will help us to identify if you are facing something we are already aware of or is it a new case.

   

  0

  Comment actions Permalink
• 

  Krzysztof Ksiazek

  • April 18, 2016 12:31

  Additionally, developers asked for this bit of information:

  1. Output of: ps tree -a 

  2. Version of your netcat package

  Some distributions allow to setup different packages to work as 'nc' - can you check which one is used in your case? Have you tried to use different ones (if possible)?

  0

  Comment actions Permalink
• 

  Kent Williams

  • April 18, 2016 14:23

  Thanks Krzysztof, I'll move this to a support ticket and copy the fix back here once the issue has been resolved.

  0

  Comment actions Permalink
• 

  Kent Williams

  • April 19, 2016 03:01

  This issue was resolved under Debian 8.4.0 amd64 by using netcat-openbsd (eg apt-get install netcat-openbsd rather than apt-get install netcat).

  Backups now complete successfully.

  Thanks to Krzystof for assisting with this problem.

  0

  Comment actions Permalink

Please sign in to leave a comment.