Falling PXC 5.6 after a successful SST

I observe a strange situation when I try to add a second node to the percona galera cluster.
After a long file upload using xtrabackup-v2 (3.5TB of data and about 1,000,000 files) and successfully passed all the stages, judging by the log, the second node falls with an absolutely uninformative error:

2017-08-29 07:24:51 14705 [Note] InnoDB: Using Linux native AIO
2017-08-29 07:24:51 14705 [Note] InnoDB: Using CPU crc32 instructions
2017-08-29 07:24:51 14705 [Warning] InnoDB: innodb_file_io_threads is deprecated. Please use innodb_read_io_threads and innodb_write_io_threads instead
2017-08-29 07:24:51 14705 [Note] InnoDB: Initializing buffer pool, size = 10.0G
2017-08-29 07:24:52 14705 [Note] InnoDB: Completed initialization of buffer pool
2017-08-29 07:24:52 14705 [Note] InnoDB: Highest supported file format is Barracuda.
2017-08-29 08:00:55 14705 [Note] WSREP: Created page /var/lib/mysql/gcache.page.000025 of size 134217728 bytes
2017-08-29 08:22:33 14705 [Note] WSREP: Created page /var/lib/mysql/gcache.page.000026 of size 134217728 bytes
2017-08-29 09:22:21 14705 [Note] WSREP: Created page /var/lib/mysql/gcache.page.000027 of size 134217728 bytes
2017-08-29 10:06:55 14705 [Note] InnoDB: 128 rollback segment (s) are active.
2017-08-29 10:06:55 14705 [Note] InnoDB: Waiting for purge to start
2017-08-29 10:06:55 14705 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.36-82.0 started; log sequence number 7347200454611
2017-08-29 10:06:55 14705 [ERROR] Aborting

2017-08-29 10:06:55 14705 [Note] WSREP: Signalling cancellation of the SST request.
2017-08-29 10:06:55 14705 [Note] WSREP: SST request was cancelled
2017-08-29 10:06:55 14705 [Note] WSREP: Closing send monitor ...
2017-08-29 10:06:55 14705 [Note] WSREP: Closed send monitor.
2017-08-29 10:06:55 14705 [Note] WSREP: gcomm: terminating thread
2017-08-29 10:06:55 14705 [Note] WSREP: gcomm: joining thread
2017-08-29 10:06:55 14705 [Note] WSREP: gcomm: closing backend
2017-08-29 10:06:57 14705 [Note] WSREP: Service disconnected.
2017-08-29 10:06:57 14705 [Note] WSREP: Waiting to close threads ......
2017-08-29 10:06:57 14705 [Note] WSREP: rollbacker thread exiting

Maybe someone came across and help with advice.

Official comment

Krzysztof Ksiazek

August 30, 2017 12:57

Hi,

It's quite hard to tell anything without all of the logs. If you happen to have ClusterControl installed, please generate an error report. If not, we'd need MySQL error logs from all Galera nodes, it'd be also great to have all innobackup.* logs from donor and joiner. Given that sharing logs in public may not be the best idea, it'd be better if you'd open a ticket related to this issue with the data we asked for and we'll continue from there.

Thanks,

Krzysztof

Stanislav Gritsenko

August 30, 2017 16:14

I have no subscription, so I think - I cant create the ticket. And I can attach log files. This form support only images ( strange at all). Where is the best place to share logs?

Johan

August 30, 2017 17:35

Run

dmesg

on the aborting node. Maybe it will tell more about the cause.

BR
johan

August 30, 2017 18:16

It's perfectly fine to open a ticket at https://severalnines.zendesk.com even if you don't have a license.

August 30, 2017 18:26

Thanks. I have created ticket

https://support.severalnines.com/hc/en-us/requests/4018

Falling PXC 5.6 after a successful SST

Comments

Didn't find what you were looking for?