Failed to connect in cmon log

Follow
Avatar
Haris Zukanovic74

Hi,

I have a lot of "Failed to connect to mysql server" messages in cmon logs... I run 1.1.27
Checking the situation in my other monitoring tools there seems to have been absolutely no connection problems between Cluster Control and other Galera nodes in the cluster. What do you suggest I can check out to confirm if I have a real problem or not?

Jun 27 03:09:40 : (WARNING) Failed to connect to server : mysql2 cmon. Reason: Can't connect to MySQL server on 'mysql2' (4)
Jun 27 03:18:31 : (WARNING) Failed to connect to server : mysql2 cmon. Reason: Can't connect to MySQL server on 'mysql2' (4)
Jun 27 03:42:49 : (WARNING) Failed to connect to server : mysql2 cmon. Reason: Can't connect to MySQL server on 'mysql2' (4)
Jun 27 03:51:39 : (WARNING) Failed to connect to server : mysql2 cmon. Reason: Can't connect to MySQL server on 'mysql2' (4)
Jun 27 04:11:09 : (WARNING) Failed to connect to server : mysql1 cmon. Reason: Can't connect to MySQL server on 'mysql1' (4)
Jun 27 04:11:18 : (WARNING) Failed to connect to server : mysql1 cmon. Reason: Can't connect to MySQL server on 'mysql1' (4)
Jun 27 04:11:26 : (WARNING) Failed to connect to server : mysql1 cmon. Reason: Can't connect to MySQL server on 'mysql1' (4)
Jun 27 04:11:39 : (WARNING) Failed to connect to server : mysql1 cmon. Reason: Can't connect to MySQL server on 'mysql1' (4)
Jun 27 04:11:55 : (WARNING) Failed to connect to server : mysql1 cmon. Reason: Can't connect to MySQL server on 'mysql1' (4)
Jun 27 04:12:09 : (WARNING) Failed to connect to server : mysql1 cmon. Reason: Can't connect to MySQL server on 'mysql1' (4)
Jun 27 04:12:12 : (WARNING) Failed to connect to server : mysql1 cmon. Reason: Can't connect to MySQL server on 'mysql1' (4)
Jun 27 04:12:37 : (WARNING) Failed to connect to server : mysql1 cmon. Reason: Can't connect to MySQL server on 'mysql1' (4)
Jun 27 04:13:05 : (WARNING) Failed to connect to server : mysql1 cmon. Reason: Can't connect to MySQL server on 'mysql1' (4)
Jun 27 04:13:32 : (WARNING) Failed to connect to server : mysql1 cmon. Reason: Can't connect to MySQL server on 'mysql1' (4)
Jun 27 04:13:39 : (WARNING) Failed to connect to server : mysql2 cmon. Reason: Can't connect to MySQL server on 'mysql2' (4)
Jun 27 04:13:47 : (WARNING) Failed to connect to server : mysql2 cmon. Reason: Can't connect to MySQL server on 'mysql2' (4)
Jun 27 04:13:50 : (WARNING) Failed to connect to server : mysql1 cmon. Reason: Can't connect to MySQL server on 'mysql1' (4)
Jun 27 04:13:59 : (WARNING) Failed to connect to server : mysql1 cmon. Reason: Can't connect to MySQL server on 'mysql1' (4)
Jun 27 04:14:18 : (WARNING) Failed to connect to server : mysql1 cmon. Reason: Can't connect to MySQL server on 'mysql1' (4)
Jun 27 04:14:43 : (WARNING) Failed to connect to server : mysql1 cmon. Reason: Can't connect to MySQL server on 'mysql1' (4)
Jun 27 04:15:00 : (WARNING) Failed to connect to server : mysql2 cmon. Reason: Can't connect to MySQL server on 'mysql2' (4)
Jun 27 04:15:03 : (WARNING) Failed to connect to server : mysql1 cmon. Reason: Can't connect to MySQL server on 'mysql1' (4)
Jun 27 04:15:18 : (WARNING) Failed to connect to server : mysql1 cmon. Reason: Can't connect to MySQL server on 'mysql1' (4)
Jun 27 04:15:45 : (WARNING) Failed to connect to server : mysql1 cmon. Reason: Can't connect to MySQL server on 'mysql1' (4)
Jun 27 04:16:13 : (WARNING) Failed to connect to server : mysql1 cmon. Reason: Can't connect to MySQL server on 'mysql1' (4)
Jun 27 04:17:43 : (WARNING) Failed to connect to server : mysql1 cmon. Reason: Can't connect to MySQL server on 'mysql1' (4)
Jun 27 04:21:29 : (WARNING) Failed to connect to server : mysql2 cmon. Reason: Can't connect to MySQL server on 'mysql2' (4)
Jun 27 04:43:22 : (WARNING) Failed to connect to server : mysql2 cmon. Reason: Can't connect to MySQL server on 'mysql2' (4)
Jun 27 05:35:48 : (WARNING) Failed to connect to server : mysql2 cmon. Reason: Can't connect to MySQL server on 'mysql2' (4)
Jun 27 06:08:13 : (WARNING) Failed to connect to server : mysql2 cmon. Reason: Can't connect to MySQL server on 'mysql2' (4)
Jun 27 06:14:22 : (WARNING) Failed to connect to server : mysql2 cmon. Reason: Can't connect to MySQL server on 'mysql2' (4)
Jun 27 06:14:41 : (WARNING) Failed to connect to server : mysql2 cmon. Reason: Can't connect to MySQL server on 'mysql2' (4)
Jun 27 06:17:40 : (WARNING) Failed to connect to server : mysql2 cmon. Reason: Can't connect to MySQL server on 'mysql2' (4)
Jun 27 06:23:12 : (WARNING) Failed to connect to server : mysql2 cmon. Reason: Can't connect to MySQL server on 'mysql2' (4)
Jun 27 06:32:19 : (WARNING) Failed to connect to server : mysql2 cmon. Reason: Can't connect to MySQL server on 'mysql2' (4)
Jun 27 06:36:18 : (WARNING) Failed to connect to server : mysql2 cmon. Reason: Can't connect to MySQL server on 'mysql2' (4)
Jun 27 06:47:39 : (WARNING) Failed to connect to server : mysql2 cmon. Reason: Can't connect to MySQL server on 'mysql2' (4)
Jun 27 07:51:39 : (WARNING) Failed to connect to server : mysql2 cmon. Reason: Can't connect to MySQL server on 'mysql2' (4)
Jun 27 09:08:39 : (WARNING) Failed to connect to server : mysql1 cmon. Reason: Can't connect to MySQL server on 'mysql1' (4)
Jun 27 09:19:40 : (WARNING) Failed to connect to server : mysql2 cmon. Reason: Can't connect to MySQL server on 'mysql2' (4)
Jun 27 10:47:39 : (WARNING) Failed to connect to server : mysql2 cmon. Reason: Can't connect to MySQL server on 'mysql2' (4)
Jun 27 10:58:40 : (WARNING) Failed to connect to server : mysql2 cmon. Reason: Can't connect to MySQL server on 'mysql2' (4)
Jun 27 11:29:40 : (WARNING) Failed to connect to server : mysql1 cmon. Reason: Can't connect to MySQL server on 'mysql1' (4)
Jun 27 12:24:39 : (WARNING) Failed to connect to server : mysql2 cmon. Reason: Can't connect to MySQL server on 'mysql2' (4)

 

thank you in advance
Haris 

0

Comments

10 comments

Sort by Date Votes
  • Avatar
    Johan

    Hi,

    This can be attributed to MySQL: http://bugs.mysql.com/bug.php?id=64333.

    However, in version 1.1.32 we have changed how connections are made to the mysql server so it is very unlikely it will happen.

    I do recommend you to upgrade if you are unhappy with the warnings. What the warnings means is that you will miss a stats collection when you see this warning, because the mysql connection timed out. But, as i said we do connections in another way in 1.1.32, and then you should not see this again.

    The root of the problem however can be:
    1) server mysql1 and mysql2 are swapping

    2) the clustercontrol server is swapping

    3) the servers are running slow for some reason (high load)

    4) network problems 

    Best regards,
    Johan

    0
    Comment actions Permalink
  • Avatar
    Haris Zukanovic74

    Hi Johan,
    and thank you for the reply..
    I will try to do an upgrade.
    Do you mean that this "warning" is only connected to stats collection and that no cluster recovery is initiated from this warning?

    On the other side I am pretty confident that none of the 4 point of possible root cause are the problem. Servers have almost no load, clustercontrol has 3GB of memory on a dedicated server. The only one that is plausible is a transient network problem but certainly not as frequent as I see this warning in the logs. 

    0
    Comment actions Permalink
  • Avatar
    Haris Zukanovic74

    I believe still I am having trouble with the auto-recover system of CC. Is it possible to see in cmon log files exactly when and how it decided to preform the recovery? Maybe by enabling higher level off log information?
    One of my Galera nodes shutdown while mysqld was still up (did not crash) and it maybe due to an action requested from CC, but I cannot find it in logs...
    Would i have seen it explicitly in the log if CC did a recovery procedure on a node? The node in question has IP 176.9.107.218 and it is mentioned in CC logs.

     

    thanx in advance for any info

    This is the log of the hours when it happend:

     

    Jun 25 20:00:00 : (INFO) Maint: Checking internal schedules
Jun 25 20:00:28 : (INFO) Checking mails to send
Jun 25 20:00:28 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:00:40 : (INFO) Maint: Checking internal schedules
Jun 25 20:00:58 : (INFO) Checking mails to send
Jun 25 20:00:58 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:01:00 : (INFO) Maint: Checking internal schedules
Jun 25 20:01:28 : (INFO) Checking mails to send
Jun 25 20:01:28 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:01:40 : (INFO) Maint: Checking internal schedules
Jun 25 20:01:58 : (INFO) Checking mails to send
Jun 25 20:01:58 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:02:00 : (INFO) Maint: Checking internal schedules
Jun 25 20:02:28 : (INFO) Checking mails to send
Jun 25 20:02:28 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:02:40 : (INFO) Maint: Checking internal schedules
Jun 25 20:02:58 : (INFO) Checking mails to send
Jun 25 20:02:58 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:03:00 : (INFO) Maint: Checking internal schedules
Jun 25 20:03:28 : (INFO) Checking mails to send
Jun 25 20:03:28 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:03:40 : (INFO) Maint: Checking internal schedules
Jun 25 20:03:58 : (INFO) Checking mails to send
Jun 25 20:03:58 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:04:00 : (INFO) Maint: Checking internal schedules
Jun 25 20:04:28 : (INFO) Checking mails to send
Jun 25 20:04:28 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:04:40 : (INFO) Maint: Checking internal schedules
Jun 25 20:04:58 : (INFO) Checking mails to send
Jun 25 20:04:58 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:05:00 : (INFO) Maint: Checking internal schedules
Jun 25 20:05:28 : (INFO) Checking mails to send
Jun 25 20:05:28 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:05:40 : (INFO) Maint: Checking internal schedules
Jun 25 20:05:58 : (INFO) Checking mails to send
Jun 25 20:05:58 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:06:00 : (INFO) Maint: Checking internal schedules
Jun 25 20:06:28 : (INFO) Checking mails to send
Jun 25 20:06:28 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:06:40 : (INFO) Maint: Checking internal schedules
Jun 25 20:06:58 : (INFO) Checking mails to send
Jun 25 20:06:58 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:07:00 : (INFO) Gathering Database Stats
Jun 25 20:07:00 : (INFO) Maint: Checking internal schedules
Jun 25 20:07:28 : (INFO) Checking mails to send
Jun 25 20:07:28 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:07:40 : (INFO) Maint: Checking internal schedules
Jun 25 20:07:58 : (INFO) Checking mails to send
Jun 25 20:07:58 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:08:00 : (INFO) Maint: Checking internal schedules
Jun 25 20:08:28 : (INFO) Checking mails to send
Jun 25 20:08:28 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:08:40 : (INFO) Maint: Checking internal schedules
Jun 25 20:08:58 : (INFO) Checking mails to send
Jun 25 20:08:58 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:09:00 : (INFO) Maint: Checking internal schedules
Jun 25 20:09:28 : (INFO) Checking mails to send
Jun 25 20:09:28 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:09:40 : (INFO) Maint: Checking internal schedules
Jun 25 20:09:58 : (INFO) Checking mails to send
Jun 25 20:09:58 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:10:00 : (INFO) Maint: Checking internal schedules
Jun 25 20:10:07 : (WARNING) Could not open /proc/7755/stat file
Jun 25 20:10:07 : (WARNING) Could not open /proc/19948/stat file
Jun 25 20:10:07 : (WARNING) Could not open /proc/29462/stat file
Jun 25 20:10:07 : (WARNING) Could not open /proc/29463/stat file
Jun 25 20:10:20 : (INFO) Maint: Checking internal schedules
Jun 25 20:10:28 : (INFO) Checking mails to send
Jun 25 20:10:28 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:10:40 : (INFO) Maint: Checking internal schedules
Jun 25 20:10:58 : (INFO) Checking mails to send
Jun 25 20:10:58 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:11:00 : (INFO) Maint: Checking internal schedules
Jun 25 20:11:28 : (INFO) Checking mails to send
Jun 25 20:11:28 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:11:40 : (INFO) Maint: Checking internal schedules
Jun 25 20:11:58 : (INFO) Checking mails to send
Jun 25 20:11:58 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:12:00 : (INFO) Maint: Checking internal schedules
Jun 25 20:12:28 : (INFO) Checking mails to send
Jun 25 20:12:28 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:12:40 : (INFO) Maint: Checking internal schedules
Jun 25 20:12:58 : (INFO) Checking mails to send
Jun 25 20:12:58 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:13:00 : (INFO) Maint: Checking internal schedules
Jun 25 20:13:28 : (INFO) Checking mails to send
Jun 25 20:13:28 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:13:40 : (INFO) Maint: Checking internal schedules
Jun 25 20:13:58 : (INFO) Checking mails to send
Jun 25 20:13:58 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:14:00 : (INFO) Maint: Checking internal schedules
Jun 25 20:14:28 : (INFO) Checking mails to send
Jun 25 20:14:28 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:14:40 : (INFO) Maint: Checking internal schedules
Jun 25 20:14:58 : (INFO) Checking mails to send
Jun 25 20:14:58 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:15:00 : (INFO) Maint: Checking internal schedules
Jun 25 20:15:28 : (INFO) Checking mails to send
Jun 25 20:15:28 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:15:40 : (INFO) Maint: Checking internal schedules
Jun 25 20:15:58 : (INFO) Checking mails to send
Jun 25 20:15:58 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:16:00 : (INFO) Maint: Checking internal schedules
Jun 25 20:16:28 : (INFO) Checking mails to send
Jun 25 20:16:28 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:16:40 : (INFO) Maint: Checking internal schedules
Jun 25 20:16:58 : (INFO) Checking mails to send
Jun 25 20:16:58 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:17:00 : (INFO) Maint: Checking internal schedules
Jun 25 20:17:28 : (INFO) Checking mails to send
Jun 25 20:17:28 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:17:40 : (INFO) Maint: Checking internal schedules
Jun 25 20:17:58 : (INFO) Checking mails to send
Jun 25 20:17:58 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:18:00 : (INFO) Maint: Checking internal schedules
Jun 25 20:18:28 : (INFO) Checking mails to send
Jun 25 20:18:28 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:18:41 : (INFO) Maint: Checking internal schedules
Jun 25 20:18:55 : (INFO) Gathering Database Stats
Jun 25 20:18:58 : (INFO) Checking mails to send
Jun 25 20:18:58 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:19:01 : (INFO) Maint: Checking internal schedules
Jun 25 20:19:28 : (INFO) Checking mails to send
Jun 25 20:19:28 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:19:41 : (INFO) Maint: Checking internal schedules
Jun 25 20:19:58 : (INFO) Checking mails to send
Jun 25 20:19:58 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:20:01 : (INFO) Maint: Checking internal schedules
Jun 25 20:20:28 : (INFO) Checking mails to send
Jun 25 20:20:28 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:20:41 : (INFO) Maint: Checking internal schedules
Jun 25 20:20:58 : (INFO) Checking mails to send
Jun 25 20:20:58 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:21:01 : (INFO) Maint: Checking internal schedules
Jun 25 20:21:28 : (INFO) Checking mails to send
Jun 25 20:21:28 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:21:41 : (INFO) Maint: Checking internal schedules
Jun 25 20:21:58 : (INFO) Checking mails to send
Jun 25 20:21:58 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:22:01 : (INFO) Maint: Checking internal schedules
Jun 25 20:22:28 : (INFO) Checking mails to send
Jun 25 20:22:28 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:22:41 : (INFO) Maint: Checking internal schedules
Jun 25 20:22:58 : (INFO) Checking mails to send
Jun 25 20:22:58 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:23:01 : (INFO) Maint: Checking internal schedules
Jun 25 20:23:28 : (INFO) Checking mails to send
Jun 25 20:23:28 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:23:41 : (INFO) Maint: Checking internal schedules
Jun 25 20:23:58 : (INFO) Checking mails to send
Jun 25 20:23:58 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:24:01 : (INFO) Maint: Checking internal schedules
Jun 25 20:24:28 : (INFO) Checking mails to send
Jun 25 20:24:28 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:24:41 : (INFO) Maint: Checking internal schedules
Jun 25 20:24:58 : (INFO) Checking mails to send
Jun 25 20:24:58 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:25:01 : (INFO) Maint: Checking internal schedules
Jun 25 20:25:28 : (INFO) Checking mails to send
Jun 25 20:25:28 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:25:41 : (INFO) Maint: Checking internal schedules
Jun 25 20:25:58 : (INFO) Checking mails to send
Jun 25 20:25:58 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:26:01 : (INFO) Maint: Checking internal schedules
Jun 25 20:26:28 : (INFO) Checking mails to send
Jun 25 20:26:28 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:26:41 : (INFO) Maint: Checking internal schedules
Jun 25 20:26:58 : (INFO) Checking mails to send
Jun 25 20:26:58 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:27:01 : (INFO) Maint: Checking internal schedules
Jun 25 20:27:28 : (INFO) Checking mails to send
Jun 25 20:27:28 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:27:41 : (INFO) Maint: Checking internal schedules
Jun 25 20:27:42 : (INFO) local command : mkdir -p /tmp//1//galera returned 0
Jun 25 20:27:42 : (INFO) local command : mkdir -p /tmp//1//galera/config/ returned 0
Jun 25 20:27:42 : (INFO) local command : mkdir -p  /tmp//1//galera/etc/ returned 0
Jun 25 20:27:42 : (INFO) local command : mkdir -p /tmp//1//galera/init.d/ returned 0
Jun 25 20:27:42 : (INFO) local command : chown -R root /tmp//1//galera returned 0
Jun 25 20:27:42 : (INFO) local command : cp /root/s9s-galera-2.0.0-rpm//mysql/config/* /tmp//1//galera/config/ returned 0
Jun 25 20:27:42 : (INFO) local command : cp /root/s9s-galera-2.0.0-rpm//mysql/init.d/cmon /tmp//1//galera/init.d/ returned 0
Jun 25 20:27:58 : (INFO) Checking mails to send
Jun 25 20:27:58 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:28:01 : (INFO) Maint: Checking internal schedules
Jun 25 20:28:28 : (INFO) Checking mails to send
Jun 25 20:28:28 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:28:41 : (INFO) Maint: Checking internal schedules
Jun 25 20:28:58 : (INFO) Checking mails to send
Jun 25 20:28:58 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:29:01 : (INFO) Maint: Checking internal schedules
Jun 25 20:29:28 : (INFO) Checking mails to send
Jun 25 20:29:28 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:29:41 : (INFO) Maint: Checking internal schedules
Jun 25 20:29:58 : (INFO) Checking mails to send
Jun 25 20:29:58 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:30:01 : (INFO) Maint: Checking internal schedules
Jun 25 20:30:04 : (WARNING) Could not open /proc/26809/stat file
Jun 25 20:30:04 : (WARNING) Could not open /proc/29462/stat file
Jun 25 20:30:04 : (WARNING) Could not open /proc/29463/stat file
Jun 25 20:30:21 : (INFO) Maint: Checking internal schedules
Jun 25 20:30:28 : (INFO) Checking mails to send
Jun 25 20:30:28 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:30:41 : (INFO) Maint: Checking internal schedules
Jun 25 20:30:54 : (INFO) Gathering Database Stats
Jun 25 20:30:58 : (INFO) Checking mails to send
Jun 25 20:30:58 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:31:01 : (INFO) Maint: Checking internal schedules
Jun 25 20:31:28 : (INFO) Checking mails to send
Jun 25 20:31:28 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:31:41 : (INFO) Maint: Checking internal schedules
Jun 25 20:31:58 : (INFO) Checking mails to send
Jun 25 20:31:58 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:32:01 : (INFO) Maint: Checking internal schedules
Jun 25 20:32:28 : (INFO) Checking mails to send
Jun 25 20:32:28 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:32:41 : (INFO) Maint: Checking internal schedules
Jun 25 20:32:58 : (INFO) Checking mails to send
Jun 25 20:32:58 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:33:01 : (INFO) Maint: Checking internal schedules
Jun 25 20:33:28 : (INFO) Checking mails to send
Jun 25 20:33:28 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:33:41 : (INFO) Maint: Checking internal schedules
Jun 25 20:33:58 : (INFO) Checking mails to send
Jun 25 20:33:58 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:34:01 : (INFO) Maint: Checking internal schedules
Jun 25 20:34:28 : (INFO) Checking mails to send
Jun 25 20:34:28 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:34:41 : (INFO) Maint: Checking internal schedules
Jun 25 20:34:58 : (INFO) Checking mails to send
Jun 25 20:34:58 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:35:01 : (INFO) Maint: Checking internal schedules
Jun 25 20:35:28 : (INFO) Checking mails to send
Jun 25 20:35:28 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:35:41 : (INFO) Maint: Checking internal schedules
Jun 25 20:35:58 : (INFO) Checking mails to send
Jun 25 20:35:58 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:36:01 : (INFO) Maint: Checking internal schedules
Jun 25 20:36:28 : (INFO) Checking mails to send
Jun 25 20:36:28 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:36:41 : (INFO) Maint: Checking internal schedules
Jun 25 20:36:58 : (INFO) Checking mails to send
Jun 25 20:36:58 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:37:01 : (INFO) Maint: Checking internal schedules
Jun 25 20:37:28 : (INFO) Checking mails to send
Jun 25 20:37:28 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:37:41 : (INFO) Maint: Checking internal schedules
Jun 25 20:37:58 : (INFO) Checking mails to send
Jun 25 20:37:58 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:38:01 : (INFO) Maint: Checking internal schedules
Jun 25 20:38:28 : (INFO) Checking mails to send
Jun 25 20:38:28 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:38:41 : (INFO) Maint: Checking internal schedules
Jun 25 20:38:58 : (INFO) Checking mails to send
Jun 25 20:38:58 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:39:01 : (INFO) Maint: Checking internal schedules
Jun 25 20:39:28 : (INFO) Checking mails to send
Jun 25 20:39:28 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:39:41 : (INFO) Maint: Checking internal schedules
Jun 25 20:39:58 : (INFO) Checking mails to send
Jun 25 20:39:58 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:40:01 : (INFO) Maint: Checking internal schedules
Jun 25 20:40:03 : (WARNING) Could not open /proc/3485/stat file
Jun 25 20:40:03 : (WARNING) Could not open /proc/6777/stat file
Jun 25 20:40:03 : (WARNING) Could not open /proc/7281/stat file
Jun 25 20:40:03 : (WARNING) Could not open /proc/7527/stat file
Jun 25 20:40:03 : (WARNING) Could not open /proc/19948/stat file
Jun 25 20:40:03 : (WARNING) Could not open /proc/29462/stat file
Jun 25 20:40:03 : (WARNING) Could not open /proc/29463/stat file
Jun 25 20:40:11 : (WARNING) Failed to connect to server : 176.9.107.218 cmon. Reason: Can't connect to MySQL server on '176.9.107.218' (4)
Jun 25 20:40:21 : (INFO) Maint: Checking internal schedules
Jun 25 20:40:22 : (WARNING) Failed to connect to server : 176.9.107.218 cmon. Reason: Can't connect to MySQL server on '176.9.107.218' (4)
Jun 25 20:40:22 : (INFO) Node Recovery will commence in 85 seconds, giving Galera itself a chance first.
Jun 25 20:40:28 : (INFO) Checking mails to send
Jun 25 20:40:28 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:40:30 : (WARNING) Failed to connect to server : 176.9.107.218 cmon. Reason: Can't connect to MySQL server on '176.9.107.218' (4)
Jun 25 20:40:31 : (INFO) Node Recovery will commence in 80 seconds, giving Galera itself a chance first.
Jun 25 20:40:39 : (WARNING) Failed to connect to server : 176.9.107.218 cmon. Reason: Can't connect to MySQL server on '176.9.107.218' (4)
Jun 25 20:40:39 : (INFO) Node Recovery will commence in 75 seconds, giving Galera itself a chance first.
Jun 25 20:40:41 : (INFO) Maint: Checking internal schedules
Jun 25 20:40:51 : (WARNING) Failed to connect to server : 176.9.107.218 cmon. Reason: Can't connect to MySQL server on '176.9.107.218' (4)
Jun 25 20:40:52 : (INFO) Node Recovery will commence in 70 seconds, giving Galera itself a chance first.
Jun 25 20:40:58 : (INFO) Checking mails to send
Jun 25 20:40:58 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:41:00 : (WARNING) Failed to connect to server : 176.9.107.218 cmon. Reason: Can't connect to MySQL server on '176.9.107.218' (4)
Jun 25 20:41:00 : (INFO) Node Recovery will commence in 65 seconds, giving Galera itself a chance first.
Jun 25 20:41:01 : (INFO) Maint: Checking internal schedules
Jun 25 20:41:08 : (WARNING) Failed to connect to server : 176.9.107.218 cmon. Reason: Can't connect to MySQL server on '176.9.107.218' (4)
Jun 25 20:41:08 : (INFO) Node Recovery will commence in 60 seconds, giving Galera itself a chance first.
Jun 25 20:41:16 : (WARNING) Failed to connect to server : 176.9.107.218 cmon. Reason: Can't connect to MySQL server on '176.9.107.218' (4)
Jun 25 20:41:17 : (INFO) Node Recovery will commence in 55 seconds, giving Galera itself a chance first.
Jun 25 20:41:21 : (INFO) Maint: Checking internal schedules
Jun 25 20:41:25 : (WARNING) Failed to connect to server : 176.9.107.218 cmon. Reason: Can't connect to MySQL server on '176.9.107.218' (4)
Jun 25 20:41:25 : (INFO) Node Recovery will commence in 50 seconds, giving Galera itself a chance first.
Jun 25 20:41:28 : (INFO) Checking mails to send
Jun 25 20:41:28 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:41:33 : (WARNING) Failed to connect to server : 176.9.107.218 cmon. Reason: Can't connect to MySQL server on '176.9.107.218' (4)
Jun 25 20:41:33 : (INFO) Node Recovery will commence in 45 seconds, giving Galera itself a chance first.
Jun 25 20:41:41 : (INFO) Maint: Checking internal schedules
Jun 25 20:41:41 : (WARNING) Failed to connect to server : 176.9.107.218 cmon. Reason: Can't connect to MySQL server on '176.9.107.218' (4)
Jun 25 20:41:42 : (INFO) Node Recovery will commence in 40 seconds, giving Galera itself a chance first.
Jun 25 20:41:50 : (WARNING) Failed to connect to server : 176.9.107.218 cmon. Reason: Can't connect to MySQL server on '176.9.107.218' (4)
Jun 25 20:41:50 : (INFO) Node Recovery will commence in 35 seconds, giving Galera itself a chance first.
Jun 25 20:41:58 : (INFO) Checking mails to send
Jun 25 20:41:58 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:41:58 : (WARNING) Failed to connect to server : 176.9.107.218 cmon. Reason: Can't connect to MySQL server on '176.9.107.218' (4)
Jun 25 20:41:58 : (INFO) Node Recovery will commence in 30 seconds, giving Galera itself a chance first.
Jun 25 20:42:01 : (INFO) Maint: Checking internal schedules
Jun 25 20:42:06 : (WARNING) Failed to connect to server : 176.9.107.218 cmon. Reason: Can't connect to MySQL server on '176.9.107.218' (4)
Jun 25 20:42:07 : (INFO) Node Recovery will commence in 25 seconds, giving Galera itself a chance first.
Jun 25 20:42:15 : (WARNING) Failed to connect to server : 176.9.107.218 cmon. Reason: Can't connect to MySQL server on '176.9.107.218' (4)
Jun 25 20:42:15 : (INFO) Node Recovery will commence in 20 seconds, giving Galera itself a chance first.
Jun 25 20:42:21 : (INFO) Maint: Checking internal schedules
Jun 25 20:42:23 : (WARNING) Failed to connect to server : 176.9.107.218 cmon. Reason: Can't connect to MySQL server on '176.9.107.218' (4)
Jun 25 20:42:23 : (INFO) Node Recovery will commence in 15 seconds, giving Galera itself a chance first.
Jun 25 20:42:28 : (INFO) ssh  -q -oUserKnownHostsFile=/root/.ssh/known_hosts -oNumberOfPasswordPrompts=0  -oIdentityFile=/root/.ssh/id_rsa root@176.9.107.218 "/sbin/service cmon restart"
Jun 25 20:42:28 : (INFO) Checking mails to send
Jun 25 20:42:28 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:42:33 : (WARNING) Host 176.9.107.218 is down / not responding to ping after 3 cycles.
Jun 25 20:42:41 : (INFO) Maint: Checking internal schedules
Jun 25 20:42:49 : (INFO) Failed to execute Command. Retcode=255 Command: /sbin/service cmon restart (646, Communication.cpp)
Jun 25 20:42:52 : (INFO) ssh  -q -oUserKnownHostsFile=/root/.ssh/known_hosts -oNumberOfPasswordPrompts=0  -oIdentityFile=/root/.ssh/id_rsa root@176.9.107.218 "sync"
Jun 25 20:42:58 : (INFO) Checking mails to send
Jun 25 20:42:58 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:43:01 : (INFO) Maint: Checking internal schedules
Jun 25 20:43:01 : (INFO) Node Recovery will commence in 10 seconds, giving Galera itself a chance first.
Jun 25 20:43:07 : (INFO) Node Recovery will commence in 5 seconds, giving Galera itself a chance first.
Jun 25 20:43:21 : (INFO) Maint: Checking internal schedules
Jun 25 20:43:28 : (INFO) Checking mails to send
Jun 25 20:43:28 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:43:41 : (INFO) Maint: Checking internal schedules
Jun 25 20:43:58 : (INFO) Checking mails to send
Jun 25 20:43:58 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:44:01 : (INFO) Maint: Checking internal schedules
Jun 25 20:44:28 : (INFO) Checking mails to send
Jun 25 20:44:28 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:44:41 : (INFO) Maint: Checking internal schedules
Jun 25 20:44:58 : (INFO) Checking mails to send
Jun 25 20:44:58 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:45:01 : (INFO) Maint: Checking internal schedules
Jun 25 20:45:28 : (INFO) Checking mails to send
Jun 25 20:45:28 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:45:41 : (INFO) Maint: Checking internal schedules
Jun 25 20:45:58 : (INFO) Checking mails to send
Jun 25 20:45:58 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:46:01 : (INFO) Maint: Checking internal schedules
Jun 25 20:46:28 : (INFO) Checking mails to send
Jun 25 20:46:28 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:46:41 : (INFO) Maint: Checking internal schedules
Jun 25 20:46:58 : (INFO) Checking mails to send
Jun 25 20:46:58 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:47:01 : (INFO) Maint: Checking internal schedules
Jun 25 20:47:28 : (INFO) Checking mails to send
Jun 25 20:47:28 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:47:41 : (INFO) Maint: Checking internal schedules
Jun 25 20:47:58 : (INFO) Checking mails to send
Jun 25 20:47:58 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:48:01 : (INFO) Maint: Checking internal schedules
Jun 25 20:48:28 : (INFO) Checking mails to send
Jun 25 20:48:28 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:48:41 : (INFO) Maint: Checking internal schedules
Jun 25 20:48:58 : (INFO) Checking mails to send
Jun 25 20:48:58 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:49:01 : (INFO) Maint: Checking internal schedules
Jun 25 20:49:28 : (INFO) Checking mails to send
Jun 25 20:49:28 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:49:41 : (INFO) Maint: Checking internal schedules
Jun 25 20:49:58 : (INFO) Checking mails to send
Jun 25 20:49:58 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:50:01 : (INFO) Maint: Checking internal schedules
Jun 25 20:50:28 : (INFO) Checking mails to send
Jun 25 20:50:28 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:50:41 : (INFO) Maint: Checking internal schedules
Jun 25 20:50:58 : (INFO) Checking mails to send
Jun 25 20:50:58 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:51:01 : (INFO) Maint: Checking internal schedules
Jun 25 20:51:28 : (INFO) Checking mails to send
Jun 25 20:51:28 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:51:41 : (INFO) Maint: Checking internal schedules
Jun 25 20:51:58 : (INFO) Checking mails to send
Jun 25 20:51:58 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:52:01 : (INFO) Maint: Checking internal schedules
Jun 25 20:52:28 : (INFO) Checking mails to send
Jun 25 20:52:28 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:52:41 : (INFO) Maint: Checking internal schedules
Jun 25 20:52:58 : (INFO) Checking mails to send
Jun 25 20:52:58 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:53:01 : (INFO) Maint: Checking internal schedules
Jun 25 20:53:28 : (INFO) Checking mails to send
Jun 25 20:53:28 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:53:41 : (INFO) Maint: Checking internal schedules
Jun 25 20:53:58 : (INFO) Checking mails to send
Jun 25 20:53:58 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:54:01 : (INFO) Maint: Checking internal schedules
Jun 25 20:54:28 : (INFO) Checking mails to send
Jun 25 20:54:28 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:54:41 : (INFO) Maint: Checking internal schedules
Jun 25 20:54:58 : (INFO) Checking mails to send
Jun 25 20:54:58 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:55:01 : (INFO) Maint: Checking internal schedules
Jun 25 20:55:28 : (INFO) Checking mails to send
Jun 25 20:55:28 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:55:41 : (INFO) Maint: Checking internal schedules
Jun 25 20:55:58 : (INFO) Checking mails to send
Jun 25 20:55:58 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:56:01 : (INFO) Maint: Checking internal schedules
Jun 25 20:56:28 : (INFO) Checking mails to send
Jun 25 20:56:28 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:56:41 : (INFO) Maint: Checking internal schedules
Jun 25 20:56:58 : (INFO) Checking mails to send
Jun 25 20:56:58 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:57:01 : (INFO) Maint: Checking internal schedules
Jun 25 20:57:28 : (INFO) Checking mails to send
Jun 25 20:57:28 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:57:41 : (INFO) Maint: Checking internal schedules
Jun 25 20:57:42 : (INFO) local command : mkdir -p /tmp//1//galera returned 0
Jun 25 20:57:42 : (INFO) local command : mkdir -p /tmp//1//galera/config/ returned 0
Jun 25 20:57:42 : (INFO) local command : mkdir -p  /tmp//1//galera/etc/ returned 0
Jun 25 20:57:42 : (INFO) local command : mkdir -p /tmp//1//galera/init.d/ returned 0
Jun 25 20:57:42 : (INFO) local command : chown -R root /tmp//1//galera returned 0
Jun 25 20:57:42 : (INFO) local command : cp /root/s9s-galera-2.0.0-rpm//mysql/config/* /tmp//1//galera/config/ returned 0
Jun 25 20:57:42 : (INFO) local command : cp /root/s9s-galera-2.0.0-rpm//mysql/init.d/cmon /tmp//1//galera/init.d/ returned 0
Jun 25 20:57:58 : (INFO) Checking mails to send
Jun 25 20:57:58 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:58:01 : (INFO) Maint: Checking internal schedules
Jun 25 20:58:28 : (INFO) Checking mails to send
Jun 25 20:58:28 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:58:41 : (INFO) Maint: Checking internal schedules
Jun 25 20:58:58 : (INFO) Checking mails to send
Jun 25 20:58:58 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:59:01 : (INFO) Maint: Checking internal schedules
Jun 25 20:59:28 : (INFO) Checking mails to send
Jun 25 20:59:28 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:59:41 : (INFO) Maint: Checking internal schedules
Jun 25 20:59:58 : (INFO) Checking mails to send
Jun 25 20:59:58 : (INFO) MAIL Client: No mail server configuration found
    0
    Comment actions Permalink
  • Avatar
    Johan

    Hi,

    Yes, when it fails to connect to the server it will actually be marked as disconnected. It will then run recovery on that node, so sorry my mistake (on my previous note it is only a warning..).

    Since none of  1-4 applies, do you have any packet loss (if you do ifconfig)?

    I suggest you to upgrade to 1.1.32, it fixes these problem, and I have never before seen the connection being interrupted (errno 4).

    Best regards,

    Johan

    0
    Comment actions Permalink
  • Avatar
    Haris Zukanovic74

    From Galera support (Alex) I get info that probably CC requested a node shutdown without subsequent request for restart.
    Can I in any way confirm or falsify this reading cmon logs? 

    ..and there seems not be any packet loss registered...

    eth0 Link encap:Ethernet HWaddr 00:50:56:00:00:C2
    inet addr:the IP Bcast:bcast IP Mask:255.255.255.224
    inet6 addr: fe80::250:56ff:fe00:c2/64 Scope:Link
    UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
    RX packets:173117831 errors:0 dropped:0 overruns:0 frame:0
    TX packets:181011933 errors:0 dropped:0 overruns:0 carrier:0
    collisions:0 txqueuelen:1000
    RX bytes:20922582065 (19.4 GiB) TX bytes:30688380985 (28.5 GiB)


    0
    Comment actions Permalink
  • Avatar
    Haris Zukanovic74

    For upgrade, I need only upgrade the CC, no need to upgrade nodes themselves? Upgrade would be from my current version 1.1.27

    0
    Comment actions Permalink
  • Avatar
    Johan

    HI,

    In the log output you provide:

    L server on '176.9.107.218' (4)
Jun 25 20:42:07 : (INFO) Node Recovery will commence in 25 seconds, giving Galera itself a chance first.
Jun 25 20:42:15 : (WARNING) Failed to connect to server : 176.9.107.218 cmon. Reason: Can't connect to MySQL server on '176.9.107.218' (4)
Jun 25 20:42:15 : (INFO) Node Recovery will commence in 20 seconds, giving Galera itself a chance first.
Jun 25 20:42:21 : (INFO) Maint: Checking internal schedules
Jun 25 20:42:23 : (WARNING) Failed to connect to server : 176.9.107.218 cmon. Reason: Can't connect to MySQL server on '176.9.107.218' (4)
Jun 25 20:42:23 : (INFO) Node Recovery will commence in 15 seconds, giving Galera itself a chance first.
Jun 25 20:42:28 : (INFO) ssh  -q -oUserKnownHostsFile=/root/.ssh/known_hosts -oNumberOfPasswordPrompts=0  -oIdentityFile=/root/.ssh/id_rsa root@176.9.107.218 "/sbin/service cmon restart"
Jun 25 20:42:28 : (INFO) Checking mails to send
Jun 25 20:42:28 : (INFO) MAIL Client: No mail server configuration found
Jun 25 20:42:33 : (WARNING) Host 176.9.107.218 is down / not responding to ping after 3 cycles.

    The host 176.9.107.218 is not reachable (ping failed to it), so it must have been some problem with the host.

     

     

    0
    Comment actions Permalink
  • Avatar
    Haris Zukanovic74

    Indeed. I am very sure there was some problem with the node host connectivity.
    What I am trying to understand why the node in question just shutdown the Galera replication and never came back up again until I restarted it manually.
    The theory of Alex from Galera is that the CC requested the node to shutdown but never requested a restart. Can I confirm of falsify this?
    In the log of cmon I cannot see any line that indicate that recovery actions had been undertaken for this node. Are these normally written to the log? 
    The other theory (mine :) is that the Galera node shutdown by itself without any external factors since it was disconnected from the world and could not communicate.

    What do you think? 

    0
    Comment actions Permalink
  • Avatar
    Johan

    Hi,

    it should of course recover the node, and we have improved on the recovery in 1.1.32. The recovery actions are normally written to the log (in 1.1.32 it is quite verbose when writing to the log during the recovery phase, and more silent otherwise when all is fine). Also, you should see a Job Message being created that the CMON controller is performing a recovery job.

    If the galera node shutdown itself, it should be visible in the error log of the mysql server of that node.

    Best regards,

    Johan

    0
    Comment actions Permalink
  • Avatar
    Johan

    Hi ,

    Regarding:

    "For upgrade, I need only upgrade the CC, no need to upgrade nodes themselves? Upgrade would be from my current version 1.1.27"

    I would recommend to upgrade the CMON controller and the CMON agents (on the nodes too).

    Look here:

    http://support.severalnines.com/entries/21633407-released-clustercontrol-v1-1-32

    and 
    http://support.severalnines.com/entries/21095371-cmon-1-1-32-released-upgrade-instructions

    Best regards,

    Johan

    0
    Comment actions Permalink

Please sign in to leave a comment.

Didn't find what you were looking for?

New post
Powered by Zendesk