Semaphore wait has lasted > 600 seconds
Hello,
when trying to transfer our base to the xtradb cluster, after a few hours we get such an error and the cluster falls apart, very sad there were high hopes for this technology
Process ID=16035, Main thread ID=140627657746176 , state=sleeping
Number of rows inserted 1033810, updated 12118864, deleted 3435052, read 1511886081
0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s
Number of system rows inserted 137400, updated 3124, deleted 137411, read 382523
0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s
----------------------------
END OF INNODB MONITOR OUTPUT
============================
InnoDB: ###### Diagnostic info printed to the standard error stream
2021-03-28T13:26:38.889046Z 0 [ERROR] [MY-012872] [InnoDB] [FATAL] Semaphore wait has lasted > 600 seconds. We intentionally crash the server because it appears to be hung.
2021-03-28T13:26:38.889144Z 0 [ERROR] [MY-013183] [InnoDB] Assertion failure: ut0ut.cc:553 thread 140646637221632
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/8.0/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
2021-03-28T13:26:38.889215Z 0 [Note] [MY-000000] [WSREP] Initiating SST cancellation
13:26:38 UTC - mysqld got signal 6 ;
Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware.
Build ID: aa2b9a4eb56db863f004935ba64544f8e74af031
Server Version: 8.0.22-13.1 Percona XtraDB Cluster (GPL), Release rel13, Revision a48e6d5, WSREP version 26.4.3, wsrep_26.4.3
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x30000
/usr/sbin/mysqld(my_print_stacktrace(unsigned char const*, unsigned long)+0x3d) [0x55a7d188056d]
/usr/sbin/mysqld(handle_fatal_signal+0x373) [0x55a7d0a05153]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7ff8fec48980]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7) [0x7ff8fcd4bfb7]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x141) [0x7ff8fcd4d921]
/usr/sbin/mysqld(ut_dbg_assertion_failed(char const*, char const*, unsigned long)+0x2fc) [0x55a7d1b92c1c]
/usr/sbin/mysqld(+0x24a2175) [0x55a7d1b96175]
/usr/sbin/mysqld(srv_error_monitor_thread()+0x84b) [0x55a7d1b28c2b]
/usr/sbin/mysqld(std::thread::_State_impl<std::thread::_Invoker<std::tuple<Runnable, void (*)()> > >::_M_run()+0xbf) [0x55a7d1a1903f]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xbd6df) [0x7ff8fd7716df]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7ff8fec3d6db]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7ff8fce2e71f]
You may download the Percona XtraDB Cluster operations manual by visiting
http://www.percona.com/software/percona-xtradb-cluster/. You may find information
in the manual which will help you identify the cause of the crash.
-
from the logs noticed:
021-03-28T11:25:00.624358Z 0 [Note] [MY-000000] [Galera] Member 0.0 (pxc-db3) synced with group.
2021-03-28T12:06:14.847604Z 80579 [ERROR] [MY-000000] [WSREP] Percona-XtraDB-Cluster prohibits use of DML command on a table (database.table) without an explicit primary key with pxc_strict_mode = ENFORCING or MASTER
2021-03-28T13:14:46.811755Z 0 [Warning] [MY-012985] [InnoDB] A long semaphore wait:
--Thread 140647833044736 has waited at trx0undo.ic line 143 for 241 seconds the semaphore:
S-lock on RW-latch at 0x7fe86e5c0128 created in file buf0buf.cc line 785
a writer (thread id 140684676830976) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file trx0undo.ic line 143
Last time write locked in file /mnt/jenkins/workspace/pxc80-autobuild-RELEASE/test/percona-xtradb-cluster-8.0.22-13/storage/innobase/include/trx0undo.ic line 129after about an hour the semaphore turned on
--------------
ROW OPERATIONS
--------------
0 queries inside InnoDB, 0 queries in queue
23 read views open inside InnoDB
34 RW transactions active inside InnoDB
---OLDEST VIEW---
Read view low limit trx n:o 66393027
Trx read view will not see trx with id >= 66393027, sees < 66392831
Read view individually stored trx ids:
Read view trx id 66392831
Read view trx id 66393016
Read view trx id 66393018
Read view trx id 66393019
Read view trx id 66393020
-----------------
Process ID=16035, Main thread ID=140627657746176 , state=sleeping
Number of rows inserted 1033805, updated 12118863, deleted 3435052, read 1511885858
0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.42 reads/s
Number of system rows inserted 137400, updated 3124, deleted 137411, read 382523
0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s
----------------------------
END OF INNODB MONITOR OUTPUT
============================
InnoDB: ###### Diagnostic info printed to the standard error stream
2021-03-28T13:15:17.812370Z 0 [Warning] [MY-012985] [InnoDB] A long semaphore wait:
--Thread 140667162797824 has waited at ha_innodb.cc line 3016 for 268 seconds the semaphore:
S-lock on RW-latch at 0x7ff8c838c230 created in file trx0purge.cc line 228
a writer (thread id 140627418212096) has reserved it in mode wait exclusive
number of readers 1, waiters flag 1, lock_word: ffffffffffffffff
Last time read locked in file ha_innodb.cc line 3016
Last time write locked in file /mnt/jenkins/workspace/pxc80-autobuild-RELEASE/test/percona-xtradb-cluster-8.0.22-13/storage/innobase/srv/srv0srv.cc line 3580
2021-03-28T13:15:17.812466Z 0 [Warning] [MY-012985] [InnoDB] A long semaphore wait:
--Thread 140665413785344 has waited at ha_innodb.cc line 3016 for 267 seconds the semaphore:
S-lock on RW-latch at 0x7ff8c838c230 created in file trx0purge.cc line 228
a writer (thread id 140627418212096) has reserved it in mode wait exclusive
number of readers 1, waiters flag 1, lock_word: ffffffffffffffff
Last time read locked in file ha_innodb.cc line 3016
Last time write locked in file /mnt/jenkins/workspace/pxc80-autobuild-RELEASE/test/percona-xtradb-cluster-8.0.22-13/storage/innobase/srv/srv0srv.cc line 3580
2021-03-28T13:15:17.812544Z 0 [Warning] [MY-012985] [InnoDB] A long semaphore wait:
--Thread 140667162593024 has waited at ha_innodb.cc line 3016 for 268 seconds the semaphore:
S-lock on RW-latch at 0x7ff8c838c230 created in file trx0purge.cc line 228
a writer (thread id 140627418212096) has reserved it in mode wait exclusive
number of readers 1, waiters flag 1, lock_word: ffffffffffffffff
Last time read locked in file ha_innodb.cc line 3016
Last time write locked in file /mnt/jenkins/workspace/pxc80-autobuild-RELEASE/test/percona-xtradb-cluster-8.0.22-13/storage/innobase/srv/srv0srv.cc line 3580
2021-03-28T13:15:17.812582Z 0 [Warning] [MY-012985] [InnoDB] A long semaphore wait:
--Thread 140647833044736 has waited at trx0undo.ic line 143 for 272 seconds the semaphore:
S-lock on RW-latch at 0x7fe86e5c0128 created in file buf0buf.cc line 785
a writer (thread id 140684676830976) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file trx0undo.ic line 143
Last time write locked in file /mnt/jenkins/workspace/pxc80-autobuild-RELEASE/test/percona-xtradb-cluster-8.0.22-13/storage/innobase/include/trx0undo.ic line 129 -
I've got the same issue with latest version xtradb 8.0.26.16.1
Issue is only present when one of the cluster node is setup as replica receiver from another cluster.
-- Restarting a node will instantly crash the cluster, unless I stop replica IO thread on the other node.
I do not know how to solve this yet. I hope someone can help.
Please sign in to leave a comment.
Comments
3 comments