add new node faild due to xtrabackup error
hi
i installed a new replica
and now i have to add a new node to it
it faild on install of xtrabackup
the complete log is as below
how can i fix this ?
-
[09:38:53]:Setting up host '192.168.30.189:3306' failed: -------/usr/bin/socat
. -
[09:38:53]:Check the configuration template for non-supported variables and typos and check disk space, firewall issues, and the errors in the mysql error log. Once you have corrected the config and/or the environment you can select the job and 'Restart' it.
-
[09:38:53]:2022-03-03T09:35:43.009086Z 0 [Warning] [MY-011068] [Server] The syntax 'log_slave_updates' is deprecated and will be removed in a future release. Please use log_replica_updates instead.
2022-03-03T09:35:43.009233Z 0 [Warning] [MY-011069] [Server] The syntax '--master-info-repository' is deprecated and will be removed in a future release.
2022-03-03T09:35:43.009252Z 0 [Warning] [MY-011069] [Server] The syntax '--relay-log-info-repository' is deprecated and will be removed in a future release.
2022-03-03T09:35:43.009326Z 0 [Warning] [MY-010084] [Server] options --log-slow-admin-statements, --log-queries-not-using-indexes, --log-slow-replica-statements and --log-query-errors have no effect if --slow-query-log is not set
2022-03-03T09:35:43.009482Z 0 [Warning] [MY-010918] [Server] 'default_authentication_plugin' is deprecated and will be removed in a future release. Please use authentication_policy instead.
2022-03-03T09:35:43.009518Z 0 [System] [MY-013169] [Server] /usr/sbin/mysqld (mysqld 8.0.27-18) initializing of server in progress as process 10734
2022-03-03T09:35:43.041425Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2022-03-03T09:35:46.097475Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2022-03-03T09:35:49.269438Z 0 [Warning] [MY-013746] [Server] A deprecated TLS version TLSv1 is enabled for channel mysql_main
2022-03-03T09:35:49.269521Z 0 [Warning] [MY-013746] [Server] A deprecated TLS version TLSv1.1 is enabled for channel mysql_main
2022-03-03T09:35:49.552299Z 6 [Warning] [MY-010453] [Server] root@localhost is created with an empty password ! Please consider switching off the --initialize-insecure option.
2022-03-03T09:37:28.152006Z 0 [Warning] [MY-011068] [Server] The syntax 'log_slave_updates' is deprecated and will be removed in a future release. Please use log_replica_updates instead.
2022-03-03T09:37:28.152081Z 0 [Warning] [MY-011069] [Server] The syntax '--master-info-repository' is deprecated and will be removed in a future release.
2022-03-03T09:37:28.152091Z 0 [Warning] [MY-011069] [Server] The syntax '--relay-log-info-repository' is deprecated and will be removed in a future release.
2022-03-03T09:37:28.152145Z 0 [Warning] [MY-010084] [Server] options --log-slow-admin-statements, --log-queries-not-using-indexes, --log-slow-replica-statements and --log-query-errors have no effect if --slow-query-log is not set
2022-03-03T09:37:28.152238Z 0 [Warning] [MY-010918] [Server] 'default_authentication_plugin' is deprecated and will be removed in a future release. Please use authentication_policy instead.
2022-03-03T09:37:28.152275Z 0 [System] [MY-013169] [Server] /usr/sbin/mysqld (mysqld 8.0.27-18) initializing of server in progress as process 15087
2022-03-03T09:37:28.182953Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2022-03-03T09:37:31.572058Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2022-03-03T09:37:34.628316Z 0 [Warning] [MY-013746] [Server] A deprecated TLS version TLSv1 is enabled for channel mysql_main
2022-03-03T09:37:34.628362Z 0 [Warning] [MY-013746] [Server] A deprecated TLS version TLSv1.1 is enabled for channel mysql_main
2022-03-03T09:37:34.867615Z 6 [Warning] [MY-010453] [Server] root@localhost is created with an empty password ! Please consider switching off the --initialize-insecure option. -
[09:38:53]:192.168.30.189: All processes stopped.
-
[09:38:45]:192.168.30.189: Stopping MySQL service on unmanaged host.
-
[09:38:42]:192.168.30.189: Stopping MySQL service.
-
[09:38:42]:192.168.30.189:3306: Stopping mysqld (timeout=30, force stop after timeout=true).
-
[09:38:42]:192.168.30.189: netcat.log is empty.
-
[09:38:41]:Could not prepare backup: /usr/bin/xtrabackup --prepare --apply-log --target-dir=/var/lib/mysql 2>&1 on 192.168.30.189.
-
[09:38:41]:192.168.30.189: Command finished, last 100 lines:
xtrabackup: recognized server arguments: --innodb_checksum_algorithm=crc32 --innodb_log_checksums=1 --innodb_data_file_path=ibdata1:100M:autoextend --innodb_log_files_in_group=2 --innodb_log_file_size=536870912 --innodb_page_size=16384 --innodb_undo_directory=./ --innodb_undo_tablespaces=2 --server-id=3005 --innodb_log_checksums=ON --innodb_redo_log_encrypt=0 --innodb_undo_log_encrypt=0
xtrabackup: recognized client arguments: --prepare=1 --target-dir=/var/lib/mysql
/usr/bin/xtrabackup version 8.0.27-19 based on MySQL server 8.0.27 Linux (x86_64) (revision id: 50dbc8dadda)
xtrabackup: cd to /var/lib/mysql/
xtrabackup: This target seems to be not prepared yet.
Number of pools: 1
xtrabackup: xtrabackup_logfile detected: size=8388608, start_lsn=(242724118)
xtrabackup: using the following InnoDB configuration for recovery:
xtrabackup: innodb_data_home_dir = .
xtrabackup: innodb_data_file_path = ibdata1:100M:autoextend
xtrabackup: innodb_log_group_home_dir = .
xtrabackup: innodb_log_files_in_group = 1
xtrabackup: innodb_log_file_size = 8388608
xtrabackup: inititialize_service_handles suceeded
xtrabackup: [ERROR] unknown option '--apply-log'. -
[09:38:41]:192.168.30.189: A progress message will be written every 1 minutes
-
[09:38:40]:192.168.30.189 running: /usr/bin/xtrabackup --prepare --apply-log --target-dir=/var/lib/mysql 2>&1
-
[09:38:24]:192.168.30.225: A progress message will be written every 1 minutes
-
[09:38:22]:192.168.30.225: Checking 'socat' availibility.
-
[09:38:22]:192.168.30.225: Streaming backup to 192.168.30.189.
-
[09:38:22]:192.168.30.225: The xtrabackup version is 8.0.26 / 8000026.
-
[09:38:22]:192.168.30.225:3306: Checking xtrabackup version.
-
[09:38:19]:192.168.30.225:3306: Verifying connectivity and credentials.
-
[09:38:14]:192.168.30.225: The xtrabackup version is 8.0.26 / 8000026.
-
[09:38:14]:192.168.30.225:3306: Checking xtrabackup version.
-
[09:38:14]:192.168.30.189: Netcat started, error log is in '192.168.30.189:/tmp/netcat.log'.
-
[09:38:10]:192.168.30.189: Starting socat -u tcp-listen:9999,reuseaddr stdout 2>/tmp/netcat.log | xbstream -x -C /var/lib/mysql.
-
[09:38:10]:pgrep found no pids ('') matching the expression '(nc|socat).*9999.*'.
-
[09:38:10]:192.168.30.189: Checking 'socat' availibility.
-
[09:38:10]:192.168.30.189: Starting netcat (nc).
-
[09:38:10]:192.168.30.225: The xtrabackup version is 8.0.26 / 8000026.
-
[09:38:10]:192.168.30.225:3306: Selected xtrabackup.
-
[09:38:10]:192.168.30.225:3306: Checking for xtrabackup
-
[09:38:10]:192.168.30.225:3306: Checking for innobackupex
-
[09:38:10]:192.168.30.189:3306: Rebuilding server by streaming from 192.168.30.225:3306.
-
[09:38:08]:192.168.30.189: All processes stopped.
-
[09:38:03]:192.168.30.189: Stopping MySQL service on unmanaged host.
-
[09:37:59]:192.168.30.189: Stopping MySQL service.
-
[09:37:59]:192.168.30.189:3306: Stopping mysqld (timeout=600, force stop after timeout=false).
-
[09:37:58]:192.168.30.225:3306: Executing GRANT REPLICATION SLAVE ON *.* TO 'rpl_user'@'192.168.30.189'.
-
[09:37:57]:192.168.30.225:3306: Creating user 'rpl_user'@'192.168.30.189'.
-
[09:37:55]:192.168.30.189:3306: Ensuring the datadir '/var/lib/mysql' exists and is owned by 'mysql'.
-
[09:37:53]:192.168.30.225:3306: Executing GRANT REPLICATION SLAVE ON *.* TO 'rpl_user'@'192.168.30.189'.
-
[09:37:53]:192.168.30.225:3306: Gtid replication detected.
-
[09:37:53]:192.168.30.225:3306: Sanity checking replication master '192.168.30.225:3306[cid:139912854634499]' to be used by '192.168.30.189[cid:139912854634499]'.
-
[09:37:53]:Selected to use 'xtrabackup' (version 8000026) as the backup tool.
-
[09:37:53]:192.168.30.225: The xtrabackup version is 8.0.26 / 8000026.
-
[09:37:52]:192.168.30.225:3306: Selected xtrabackup.
-
[09:37:52]:192.168.30.225:3306: Checking for xtrabackup
-
[09:37:52]:192.168.30.225:3306: Checking for innobackupex
-
[09:37:52]:192.168.30.189: All processes stopped.
-
[09:37:46]:192.168.30.189: Stopping MySQL service on unmanaged host.
-
[09:37:41]:192.168.30.189: Stopping MySQL service.
-
[09:37:40]:192.168.30.189:3306: Installing system tables complete.
-
[09:37:40]:192.168.30.189:3306: Searching and removing '/root/.my.cnf'.
-
[09:37:40]:192.168.30.189:3306: Searching and removing '/usr/my.cnf'.
-
[09:37:40]:192.168.30.189: /usr/sbin/mysqld --initialize-insecure:
-
[09:37:27]:192.168.30.189:3306: Installing MySQL system tables.
-
[09:37:26]:192.168.30.189:3306: Cleaning up old items.
-
[09:37:26]:192.168.30.189: All processes stopped.
-
[09:37:20]:192.168.30.189: Stopping MySQL service on unmanaged host.
-
[09:37:16]:192.168.30.189: Stopping MySQL service.
-
[09:37:12]:Writing file '192.168.30.189:/etc/mysql/secrets-backup.cnf'.
-
[09:37:09]:Writing file '192.168.30.189:/etc/mysql/my.cnf'.
-
[09:37:09]:192.168.30.189:3306: /etc/mysql/my.cnf [mysqld] set: ssl_cert, ssl_key and ssl_ca values.
-
[09:37:05]:Writing file '192.168.30.189:/etc/mysql/secrets-backup.cnf'.
-
[09:37:02]:Writing file '192.168.30.189:/etc/mysql/my.cnf'.
-
[09:36:53]:192.168.30.189:3306: Deploying client certificate 'replication/cluster_3/client'
-
[09:36:53]:192.168.30.189:3306: Installed /etc/mysql/certs/server.crt, /etc/mysql/certs/server.key and /etc/mysql/certs/server_ca.crt
-
[09:36:46]:192.168.30.189:3306: Using certificate id=2.
-
[09:36:46]:192.168.30.189:3306: setting up SSL.
-
[09:36:46]:Using certificate id=2.
-
[09:36:46]:192.168.30.189:3306: Deploying SSL certificates and keys.
-
[09:36:44]:192.168.30.189:3306: Checking for SSL.
-
[09:36:41]:Writing file '192.168.30.189:/etc/mysql/secrets-backup.cnf'.
-
[09:36:37]:Writing file '192.168.30.189:/etc/mysql/my.cnf'.
-
[09:36:37]:Checking MySql configuration file(s).
-
[09:36:37]:192.168.30.189: Checking and uploading the generated MySql configuration.
-
[09:36:37]:Preparing backup secrets config file '/etc/mysql/secrets-backup.cnf'.
-
[09:36:36]:192.168.30.189: Detected memory: 7976MB.
-
[09:36:35]:192.168.30.189:3306: Setting 'innodb_data_file_path=ibdata1:100M:autoextend'.
-
[09:36:35]:192.168.30.225:3306: Detected 'innodb_data_file_path=ibdata1:100M:autoextend'.
-
[09:36:34]:192.168.30.189:3306: Preparing the MySQL configuration file.
-
[09:36:34]:Enabling 'skip_name_resolve'.
-
[09:36:34]:192.168.30.189:3306: Using dataDir: '/var/lib/mysql'
-
[09:36:34]:192.168.30.187: The xtrabackup version is 8.0.26 / 8000026.
-
[09:36:34]:192.168.30.187:3306: Checking xtrabackup version.
-
[09:36:32]:192.168.30.225: The xtrabackup version is 8.0.26 / 8000026.
-
[09:36:32]:192.168.30.225:3306: Checking xtrabackup version.
-
[09:36:31]:192.168.30.188: The xtrabackup version is 8.0.26 / 8000026.
-
[09:36:31]:192.168.30.188:3306: Checking xtrabackup version.
-
[09:36:30]:Loaded template '/usr/share/cmon/templates/my.cnf.repl80'.
-
[09:36:30]:Using config template '/usr/share/cmon/templates/my.cnf.repl80'.
-
[09:36:30]:192.168.30.189:3306: This server is a new replica.
-
[09:36:30]:192.168.30.189: Preparing MySQL Server.
-
[09:36:30]:192.168.30.189:3306: Replica is requesting to not use semi-sync irrespective of the Master's settings.
-
[09:36:30]:0TARGETDIR=/home/peiman/s9s_tmp/3/replication/restore
-
[09:36:27]:Writing file '192.168.30.189:/etc/mysql/secrets-backup.cnf'.
-
[09:36:24]:Writing file '192.168.30.189:/etc/mysql/my.cnf'.
-
[09:36:20]:Writing file '192.168.30.189:/etc/mysql/secrets-backup.cnf'.
-
[09:36:16]:Writing file '192.168.30.189:/etc/mysql/my.cnf'.
-
[09:36:13]:Writing file '192.168.30.189:/etc/mysql/secrets-backup.cnf'.
-
[09:36:09]:Writing file '192.168.30.189:/etc/mysql/my.cnf'.
-
[09:36:09]:192.168.30.189:3306: Setting server_id=3009
-
[09:36:06]:192.168.30.189: All processes stopped.
-
[09:36:00]:192.168.30.189: Stopping MySQL service on unmanaged host.
-
[09:35:56]:192.168.30.189: Stopping MySQL service.
-
[09:35:55]:192.168.30.189:3306: Installing system tables complete.
-
[09:35:55]:192.168.30.189:3306: Searching and removing '/root/.my.cnf'.
-
[09:35:55]:192.168.30.189:3306: Searching and removing '/usr/my.cnf'.
-
[09:35:55]:192.168.30.189: /usr/sbin/mysqld --initialize-insecure:
-
[09:35:42]:192.168.30.189:3306: Installing MySQL system tables.
-
[09:35:41]:192.168.30.189:3306: Cleaning up old items.
-
[09:35:41]:192.168.30.189: All processes stopped.
-
[09:35:35]:192.168.30.189: Stopping MySQL service on unmanaged host.
-
[09:35:31]:192.168.30.189: Stopping MySQL service.
-
[09:35:27]:Writing file '192.168.30.189:/etc/mysql/secrets-backup.cnf'.
-
[09:35:24]:Writing file '192.168.30.189:/etc/mysql/my.cnf'.
-
[09:35:24]:192.168.30.189:3306: /etc/mysql/my.cnf [mysqld] set: ssl_cert, ssl_key and ssl_ca values.
-
[09:35:20]:Writing file '192.168.30.189:/etc/mysql/secrets-backup.cnf'.
-
[09:35:17]:Writing file '192.168.30.189:/etc/mysql/my.cnf'.
-
[09:35:08]:192.168.30.189:3306: Deploying client certificate 'replication/cluster_3/client'
-
[09:35:08]:192.168.30.189:3306: Installed /etc/mysql/certs/server.crt, /etc/mysql/certs/server.key and /etc/mysql/certs/server_ca.crt
-
[09:35:00]:192.168.30.189:3306: Using certificate id=2.
-
[09:35:00]:192.168.30.189:3306: setting up SSL.
-
[09:35:00]:Using certificate id=2.
-
[09:35:00]:192.168.30.189:3306: Deploying SSL certificates and keys.
-
[09:34:59]:192.168.30.189:3306: Checking for SSL.
-
[09:34:55]:Writing file '192.168.30.189:/etc/mysql/secrets-backup.cnf'.
-
[09:34:52]:Writing file '192.168.30.189:/etc/mysql/my.cnf'.
-
[09:34:52]:Checking MySql configuration file(s).
-
[09:34:52]:192.168.30.189: Checking and uploading the generated MySql configuration.
-
[09:34:52]:Preparing backup secrets config file '/etc/mysql/secrets-backup.cnf'.
-
[09:34:51]:192.168.30.189: Detected memory: 7976MB.
-
[09:34:51]:192.168.30.189:3306: Setting 'innodb_data_file_path=ibdata1:100M:autoextend'.
-
[09:34:51]:192.168.30.225:3306: Detected 'innodb_data_file_path=ibdata1:100M:autoextend'.
-
[09:34:50]:192.168.30.189:3306: Preparing the MySQL configuration file.
-
[09:34:50]:Enabling 'skip_name_resolve'.
-
[09:34:50]:192.168.30.189:3306: Using dataDir: '/var/lib/mysql'
-
[09:34:46]:Writing file '192.168.30.187:/etc/mysql/secrets-backup.cnf'.
-
[09:34:43]:Writing file '192.168.30.187:/etc/mysql/my.cnf'.
-
[09:34:39]:Writing file '192.168.30.187:/etc/mysql/secrets-backup.cnf'.
-
[09:34:36]:Writing file '192.168.30.187:/etc/mysql/my.cnf'.
-
[09:34:36]:192.168.30.187: Setting [xtrabackup] databases-exclude: lost+found.
-
[09:34:36]:192.168.30.187: The xtrabackup version is 8.0.26 / 8000026.
-
[09:34:35]:192.168.30.187:3306: Checking xtrabackup version.
-
[09:34:34]:192.168.30.225: The xtrabackup version is 8.0.26 / 8000026.
-
[09:34:33]:192.168.30.225:3306: Checking xtrabackup version.
-
[09:34:32]:192.168.30.188: The xtrabackup version is 8.0.26 / 8000026.
-
[09:34:31]:192.168.30.188:3306: Checking xtrabackup version.
-
[09:34:30]:Loaded template '/usr/share/cmon/templates/my.cnf.repl80'.
-
[09:34:30]:Using config template '/usr/share/cmon/templates/my.cnf.repl80'.
-
[09:34:30]:192.168.30.189:3306: This server is a new replica.
-
[09:34:30]:-1TARGETDIR=/home/peiman/s9s_tmp/3/replication/restore
-
[09:34:30]:
SERVER HOST_STATUS RO LOGBIN
192.168.30.188 CmonHostShutDown yes
192.168.30.225 CmonHostOnline no ON
192.168.30.187 CmonHostOnline yes ON -
[09:34:30]:192.168.30.189:3306: Staging MySQL Server with data from 192.168.30.225.
-
[09:34:28]:Setting up node 192.168.30.189:3306.
-
[09:34:28]:192.168.30.189: All processes stopped.
-
[09:34:22]:192.168.30.189: Stopping MySQL service on unmanaged host.
-
[09:34:19]:192.168.30.189: Stopping MySQL service.
-
[09:34:19]:192.168.30.189:3306: Stopping mysqld on the node.
-
[09:34:19]:192.168.30.189: OS user mysql already exists.
-
[09:34:17]:192.168.30.189: Checking OS user mysql.
-
[09:34:17]:192.168.30.189:3306: Creating operating system user and group.
-
[09:34:11]:192.168.30.189:3306: Installing Software Completed.
-
[09:34:11]:192.168.30.189:3306: Installation is ok.
-
[09:34:01]:192.168.30.189:3306: Verifying installation.
-
[09:34:01]:192.168.30.189: Verifying the percona-toolkit installation.
-
[09:34:01]:192.168.30.189: Verifying the percona-xtrabackup-80 installation.
-
[09:34:01]:192.168.30.189: Verifying the percona-server-server installation.
-
[09:34:01]:192.168.30.189: Verifying the percona-server-client installation.
-
[09:33:34]:192.168.30.189: Installing percona-toolkit.
-
[09:33:19]:192.168.30.189: Installing percona-xtrabackup-80.
-
[09:32:07]:192.168.30.189: Installing percona-server-server.
-
[09:31:44]:192.168.30.189: Installing percona-server-client.
-
[09:31:34]:192.168.30.189:3306: Installing software.
-
[09:31:34]:192.168.30.189:3306: Installing replication software, vendor percona, version 8.0.
-
[09:31:33]:192.168.30.189: OS user mysql already exists.
-
[09:31:31]:192.168.30.189: Checking OS user mysql.
-
[09:30:41]:192.168.30.189: Uninstalling existing MySQL packages.
-
[09:30:41]:192.168.30.189:3306: Verifying helper packages (checking if socat is installed successfully).
-
[09:30:41]:192.168.30.189: Finished with helper packages.
-
[09:30:29]:192.168.30.189: Installing augeas-tools.
-
[09:30:28]:192.168.30.189: Installing libssl1.1.
-
[09:30:26]:192.168.30.189: Installing apt-transport-https.
-
[09:30:25]:192.168.30.189: Installing openssl.
-
[09:30:22]:192.168.30.189: Installing libatomic1.
-
[09:30:20]:192.168.30.189: Installing tar.
-
[09:30:18]:192.168.30.189: Installing iproute2.
-
[09:30:16]:192.168.30.189: Installing bzip2.
-
[09:30:14]:192.168.30.189: Installing libmecab2.
-
[09:30:12]:192.168.30.189: Installing libnuma1.
-
[09:30:11]:192.168.30.189: Installing pigz.
-
[09:30:08]:192.168.30.189: Installing curl.
-
[09:30:06]:192.168.30.189: Installing wget.
-
[09:30:05]:192.168.30.189: Installing lsb-release.
-
[09:30:03]:192.168.30.189: Installing socat.
-
[09:30:00]:192.168.30.189: Installing netcat-openbsd.
-
[09:29:59]:192.168.30.189: Installing netcat.
-
[09:29:57]:192.168.30.189: Installing libaio1.
-
[09:29:34]:192.168.30.189: Installing rsync.
-
[09:29:32]:192.168.30.189: Installing psmisc.
-
[09:29:27]:192.168.30.189:3306: Installing helper packages.
-
[09:29:07]:192.168.30.189: Using repository template file 'percona.8.0.list'.
-
[09:29:07]:192.168.30.189: Setting up repository 'percona.8.0'
-
[09:29:05]:192.168.30.189:3306: Setting up repositories.
-
[09:29:05]:192.168.30.189:3306: Installing software.
-
[09:29:05]:192.168.30.189:3306: Setting vm.swappiness = 1
-
[09:28:57]:192.168.30.189: Flushing iptables.
-
[09:28:54]:192.168.30.189: Disabling ufw.
-
[09:28:52]:192.168.30.189: Checking firewall.
-
[09:28:52]:192.168.30.189:3306: Disabling firewall.
-
[09:28:49]:192.168.30.189:3306: Setting SELinux in permissive mode.
-
[09:28:48]:192.168.30.189:3306: Setup new node (installSoftware = true).
-
[09:28:44]:192.168.30.189:3306: Installing node.
-
[09:28:43]:192.168.30.189:3306: Testing SSH to host.
-
[09:28:43]:192.168.30.189:3306: Adding host to cluster.
-
[09:28:43]:Verifying job parameters.
-
[09:28:43]:The username is 'peiman'.
-
[09:28:43]:Cluster ID is 3.
-
[09:28:43]:Using SSH credentials from cluster.
-
[09:28:43]:CMON version 1.9.1.4975.
-
Official comment
Hi rooz gar
Please try to upgrade your ClusterControl to the latest. This is a known issue because Percona XtraBackup have changed the way their xtrabackup works and this --apply-log is not known already. We have fixed this and on the latest ClusterControl, this should behave normally once you install the latest. Do not forget to restart your cmon with,
systemctl restart cmon
Comment actions
Please sign in to leave a comment.
Comments
1 comment