Only one node showing up in query monitor
I have a 3 node MySQL Galera cluster and recently did a lot of tuning with the sql queries, so I used the "clear all" button in the Query Monitor so that I would only see new data. Now only one of the nodes is reporting any data at all. I have rebooted all of the nodes as well as the cmon service on the control server. Is there anything that might have caused query logging to stop.
I have turned on general query log to troubleshoot queries manually, I did not know if that would make a difference.
-
Hi,
I looked into this. Do you run the new UI or the old one?
In the old UI then if you do "clear all", then you will have this url in your browser:
http://clustercontrol-ip/cmon/quan.php?cid=1&clear=all&qid=1
And if you then reload, the same url will be posted, and the queries will be removed again!
A work around is after you have pressed "clear all", to press "Cluster" in the filter list, to reset the clear=all flag.
You will then start to see queries again.Thanks
Johan
-
We are using the old UI. When I tried to upgrade to the new UI by clicking on the "enable our new GUI", it forwards me to the cmonapi url and all I see is "{"success":false,"msg":"Could not connect to server","code":1}" I suppose this is a separate issue. I have been happy with the old GUI, although I can try to upgrade if necessary.
I did click the "clear all" button again followed by the "Cluster" button as you suggested, however it did not seem to resolve the issue. I do not see the "clear=all" in the URL , I simply see cid=1, but it still shows only data from one of the nodes.
While doing some other troubleshooting, I noticed that when restarting cmon on the db node, I see the following lines in the cmon.log file on the node that is reporting queries, however these lines to not appear in the cmon.log on the node that is not reporting queries.
Jun 13 21:33:25 : (INFO) Connected to local mysql server!
Jun 13 21:33:25 : (WARNING) Failed to unlink:/var/lib/mysql/test-ro-slow.log error:No such file or directoryI'm not sure about the failure to unlink warning, I verified that file is there.
-
Hi,
Upgrading to the new GUI is not necessary for this.
Can you go into one of the nodes that is not reporting queries and do:
ps -ef |grep cmonIf you don't see anything can you do:
rm /var/log/cmon.log
service cmon startThen if you can send the log.
Which version of CMON are you using (check under Host Mgmt, right menu in the old gui)?
-
Hi,
It would be much appreciated if you can do on the server you sent the cmon.log from, the following tests:
Test 1:
/usr/local/mysql/bin/mysql -ucmon -pcmon -h127.0.0.1/usr/local/mysql/bin/mysql -ucmon -pcmon -hXXX.XXX.XXX.XXX
Can you connect?
Test 2:
Ubuntu:
apt-get install gdbRedhat:
yum install gdb
Then do:gdb -p `pgrep cmon`
when a prompt comes up, type the following and hit return:
thread apply all backtrace
It would be much appreciated to get the printout of the above.
Then to quit gdb press 'q' and then 'y'.
Thanks a lot!
Johan
-
Mysql connection to 127.0.0.1 does not work, however using localhost does:
root@testdb:~# /usr/local/mysql/bin/mysql -ucmon -pcmon -h127.0.0.1
ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111)
root@testdb:~# /usr/local/mysql/bin/mysql -ucmon -pcmon -hlocalhost
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -AWelcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 118237
Server version: 5.5.29 Source distribution, wsrep_23.7.3.r3853Copyright (c) 2000, 2012, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql>
Below is the output from the gdb command:
root@testdb:~# gdb -p `pgrep cmon`
GNU gdb (Ubuntu/Linaro 7.4-2012.04-0ubuntu2.1) 7.4-2012.04
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
<http://bugs.launchpad.net/gdb-linaro/>.
Attaching to process 17255
Reading symbols from /usr/local/cmon-1.2.0-64bit-glibc23-mc70/sbin/cmon...done.
Reading symbols from /lib/x86_64-linux-gnu/librt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/librt.so.1
Reading symbols from /lib/x86_64-linux-gnu/libpthread.so.0...(no debugging symbols found)...done.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7f341e552700 (LWP 17470)]
[New Thread 0x7f341ed53700 (LWP 17395)]
[New Thread 0x7f341f554700 (LWP 17258)][New Thread 0x7f341ff62700 (LWP 17256)]
Loaded symbols for /lib/x86_64-linux-gnu/libpthread.so.0
Reading symbols from /lib/x86_64-linux-gnu/libcrypt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libcrypt.so.1
Reading symbols from /lib/x86_64-linux-gnu/libnsl.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libnsl.so.1
Reading symbols from /usr/lib/x86_64-linux-gnu/libstdc++.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libstdc++.so.6
Reading symbols from /lib/x86_64-linux-gnu/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libm.so.6
Reading symbols from /lib/x86_64-linux-gnu/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libgcc_s.so.1
Reading symbols from /lib/x86_64-linux-gnu/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib/x86_64-linux-gnu/libnss_files.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libnss_files.so.2
0x00007f342002283d in nanosleep () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) thread apply all backtraceThread 5 (Thread 0x7f341ff62700 (LWP 17256)):
#0 0x00007f342002283d in nanosleep () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f3420022740 in sleep () from /lib/x86_64-linux-gnu/libc.so.6
#2 0x000000000056ba61 in CmonDb::connect_local (this=0x7f34180009e0, mysql=0x7f341ff61570) at CmonDb.cpp:231
#3 0x0000000000424c5b in mysql_collector_thread (t=0x27f00c0) at mysql_coll.cpp:70
#4 0x00007f3420f8ee9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#5 0x00007f3420056cbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#6 0x0000000000000000 in ?? ()Thread 4 (Thread 0x7f341f554700 (LWP 17258)):
#0 0x00007f342002283d in nanosleep () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f3420022740 in sleep () from /lib/x86_64-linux-gnu/libc.so.6
#2 0x0000000000429c5a in host_coll_thread (t=<optimized out>) at host_coll.cpp:164
#3 0x00007f3420f8ee9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#4 0x00007f3420056cbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#5 0x0000000000000000 in ?? ()Thread 3 (Thread 0x7f341ed53700 (LWP 17395)):
#0 0x00007f342002283d in nanosleep () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f3420022740 in sleep () from /lib/x86_64-linux-gnu/libc.so.6
#2 0x000000000042927c in process_thread (t=0x27f00c0) at process_mgr.cpp:78
#3 0x00007f3420f8ee9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#4 0x00007f3420056cbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#5 0x0000000000000000 in ?? ()Thread 2 (Thread 0x7f341e552700 (LWP 17470)):
#0 0x00007f342002283d in nanosleep () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f3420022740 in sleep () from /lib/x86_64-linux-gnu/libc.so.6
#2 0x000000000043008f in bencher_run_thread (t=<optimized out>) at bencher.cpp:709
#3 0x00007f3420f8ee9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#4 0x00007f3420056cbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#5 0x0000000000000000 in ?? ()Thread 1 (Thread 0x7f34215c1740 (LWP 17255)):
#0 0x00007f342002283d in nanosleep () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f3420050774 in usleep () from /lib/x86_64-linux-gnu/libc.so.6
#2 0x000000000041979f in main (argc=<optimized out>, argv=<optimized out>) at cmon.cpp:1532
(gdb) -
Hi,
Verify the following:
mysql_hostname=<address of the mysql server holding the cmon db>
hostname=<address of *this* server, not 127.0.0.1 or locatost>
Then do:mysql -uroot -p -h127.0.0.1
ormysql -uroot -p -hlocalhost
GRANT ALL ON *.* TO 'cmon'@'<address of *this* server>' IDENTIFIED BY 'cmon';
GRANT ALL ON *.* TO 'cmon'@'127.0.0.1' IDENTIFIED BY 'cmon';service cmon restart
If still problems, do:
Can you send me the cmon.cnf file and the my.cnf of the mysql server ( if you want, you can send it directly to johan@severalnines.com)?
Do you happen to have bind-address=... set in my.cnf?
If so, can you comment it out, kill the mysql server, and let CC recover it?
BR
johan
-
Hi Johan,
The issue seems to have been resolved when I removed the bind-address= setting from my.cnf
Our systems have three network interfaces, one to a public Internet accessible network, one to a private network, and the localhost interface. I prefer to have MySQL bind to the interface on the private network and I set the hostname= in /etc/cmon.cnf to the same interface, however I did see the following line in the general query log which makes me think that cmon is connecting to the localhost interface instead of the private subnet: "Connect cmon@127.0.0.1 on".
Is there another place to configure which interface cmon should use besides in /etc/cmon.cnf
Thanks again for your help on this.
-
I am looking on becoming a paid subscription customer...I have just deployed a MySQL Replication cluster and everything was ok but now I receive the error:
{"success":false,"msg":"Could not connect to server","code":1}This happened right when I was try to show my boss Cluster control...I cant possibly sell my management the several nines solution with this bug. Can you help me please
-
I am also receiving this strange message when I try to restart cmon:
root@clustercontrol:/var/lib/mysql/mysql# service cmon restart
Stopping cmon : ok
Starting cmon --config-file=/etc/cmon.cnf :cat: /var/run//cmon.pid: No such file or directory
failed
root@clustercontrol:/var/lib/mysql/mysql# -
Going here:
http://192.168.3.42/cmon/quan.php?cid=1&clear=all&qid=1
I receive the error:
Could not connect to mysql server:
- Hostname: 127.0.0.1
- Port: 3306
- Username: cmon
- Password: cmon
Error message:Mysql error: Can't connect to MySQL server on '127.0.0.1' (4) Mysql errno: 2003
Could not select cmon. If you are using another database than 'cmon' you need to change this in 'cmon_db_conf.inc.php'!Also, check your connections and/or run setup.
-
I turned the firewall of. It was running - rookie mistake. The output of ps is:
root@clustercontrol:/var/lib/mysql/mysql# ps -ef |grep mysqld
root 14547 1 0 15:13 pts/0 00:00:00 /bin/sh /usr/local/mysql//bin/mysqld_safe --datadir=/var/lib/mysql/ --pid-file=/var/lib/mysql//mysqld.pid
mysql 15087 14547 0 15:13 pts/0 00:00:14 /usr/local/mysql/bin/mysqld --basedir=/usr/local/mysql/ --datadir=/var/lib/mysql/ --plugin-dir=/usr/local/mysql//lib/plugin --user=mysql --log-error=/var/lib/mysql//error.log --pid-file=/var/lib/mysql//mysqld.pid --socket=/var/lib/mysql/mysql.sock --port=3306
root 23621 6355 0 15:58 pts/0 00:00:00 grep mysqldI restarted cmon and nothing :(
Please sign in to leave a comment.
Comments
16 comments