Constant hanging of wsrep in pre-commit stage and Preparing for TO isolation

Comments

12 comments

  • Avatar
    Jason Mallory

    Found this in the log:

     

    2015-09-02 17:58:58 17164 [Warning] Aborted connection 376266 to db: 'SBDBRMDEV01' user: 'root' host: 'localhost' (Got an error writing communication packets)

  • Avatar
    Jason Mallory

    Can't even switch databases:

     

    mysql> use SBDBRMDEV01;
    Reading table information for completion of table and column names
    You can turn off this feature to get a quicker startup with -A

     

    Hangs for over five minutes

  • Avatar
    Jason Mallory

    Can't shut down this node either

  • Avatar
    Krzysztof Ksiazek

    It's hard to tell what exactly is happening in your cluster but the 'Preparing for TO isolation' means that a cluster is waiting for at least one of the nodes to get in sync because a query has to be executed using TOI. Frankly, I'm not sure why it stuck on the INSERT query. Can you please provide following data:

    SHOW FULL PROCESSLIST;
    SHOW GLOBAL VARIABLES\G
    SHOW STATUS LIKE 'wsrep%';
    SHOW ENGINE InnoDB STATUS\G
    Error log

    from all of the nodes. Additionally, can you please send us output of SHOW CREATE TABLE Table\G where the insert was taking place?

     

     

  • Avatar
    Krzysztof Ksiazek

    Regarding show create table, feel free to obfuscate all column names, we are interested only in their types.

  • Avatar
    Jason Mallory

    mysql> SHOW FULL PROCESSLIST;
    +------+-----------------+-------------------+------------+---------+-------+-----------------------------+-----------------------+-----------+---------------+
    | Id   | User            | Host              | db         | Command | Time  | State                       | Info                  | Rows_sent | Rows_examined |
    +------+-----------------+-------------------+------------+---------+-------+-----------------------------+-----------------------+-----------+---------------+
    |    1 | system user     |                   | NULL       | Sleep   | 16904 | NULL                        | NULL                  |         0 |             0 |
    |    2 | system user     |                   | NULL       | Sleep   | 16904 | wsrep aborter idle          | NULL                  |         0 |             0 |
    |    3 | event_scheduler | localhost         | NULL       | Daemon  |  6278 | Waiting for next activation | NULL                  |         0 |             0 |
    |    4 | system user     |                   | NULL       | Sleep   |  6278 | NULL                        | NULL                  |         0 |             0 |
    |    5 | system user     |                   | NULL       | Sleep   |  6278 | NULL                        | NULL                  |         0 |             0 |
    |    6 | system user     |                   | NULL       | Sleep   |  6278 | NULL                        | NULL                  |         0 |             0 |
    |    7 | system user     |                   | NULL       | Sleep   |  6278 | NULL                        | NULL                  |         0 |             0 |
    |    8 | system user     |                   | NULL       | Sleep   |  6278 | NULL                        | NULL                  |         0 |             0 |
    |    9 | system user     |                   | NULL       | Sleep   |  6278 | NULL                        | NULL                  |         0 |             0 |
    |   10 | system user     |                   | NULL       | Sleep   |  6278 | NULL                        | NULL                  |         0 |             0 |
    |   11 | cmon            | 10.1.105.45:51499 | gtowebii   | Sleep   |     0 |                             | NULL                  |         0 |             0 |
    |   12 | cmon            | 10.1.105.45:51500 | phpmyadmin | Sleep   |     0 |                             | NULL                  |       459 |           459 |
    | 6401 | root            | localhost         | NULL       | Query   |     0 | init                        | SHOW FULL PROCESSLIST |         0 |             0 |
    +------+-----------------+-------------------+------------+---------+-------+-----------------------------+-----------------------+-----------+---------------+

     

     

  • Avatar
    Jason Mallory

    Global variables

  • Avatar
    Jason Mallory

    innodb status

  • Avatar
    Jason Mallory

    Error

  • Avatar
    Krzysztof Ksiazek

    Is the node locked right now? I don't see any threads waiting in the 'Preparing for TO isolation' state.

  • Avatar
    Jason Mallory

    No we had to stop the servers and restart/rebuild the cluster

  • Avatar
    Krzysztof Ksiazek

    Ok, if you run into this kind of situation again, please grab that data we asked for. It may shed some light on what' happening.

Please sign in to leave a comment.

Powered by Zendesk