Failed connect to 127.0.0.1:9500 after every reboot.
Centos 7 Clean installation using install.sh
After reboot Failed connect to 127.0.0.1:9500
- [root@cc ~]# systemctl status cmon
● cmon.service - LSB: cmon - clustercontrol-controller
Loaded: loaded (/etc/rc.d/init.d/cmon; bad; vendor preset: disabled)
Active: active (exited) since Fri 2019-08-23 14:37:11 HKT; 16min ago
Docs: man:systemd-sysv-generator(8)
Aug 23 14:37:09 cc systemd[1]: Starting LSB: cmon - clustercontrol-controller...
Aug 23 14:37:10 cc cmon[1357]: Schema : cmon: Could not connect to cmon database (config '/etc/cmon.cnf'): Can't connect to MySQL server on '127.0.0.1' (115) (errno: 2002)
Aug 23 14:37:10 cc cmon[1357]: Exiting...
Aug 23 14:37:10 cc cmon[1437]: DIGEST-MD5 common mech free
Aug 23 14:37:10 cc cmon[1357]: The Cmon Controller is initializing.
Aug 23 14:37:10 cc cmon[1357]: Going to daemonize.. - cmon will write a log in /var/log/cmon.log from now
Aug 23 14:37:10 cc cmon[1562]: DIGEST-MD5 common mech free
Aug 23 14:37:11 cc cmon[1357]: Starting cmon --rpc-port=9500 --events-client='http://127.0.0.1:9510' --cloud-service='http://127.0.0.1:9518' : ok
Aug 23 14:37:11 cc systemd[1]: Started LSB: cmon - clustercontrol-controller.
Aug 23 14:37:15 cc cmon[1588]: DIGEST-MD5 common mech free
Hint: Some lines were ellipsized, use -l to show in full.
- [root@cc ~]#systemctl enable cmon
cmon.service is not a native service, redirecting to /sbin/chkconfig.
Executing /sbin/chkconfig cmon on
- [root@cc ~]# /sbin/chkconfig --list
cmon 0:off 1:off 2:on 3:on 4:on 5:on 6:off
- [root@cc ~]#service cmon status
cmon is NOT running. ??
- [root@cc ~]# journalctl -u cmon
Aug 23 14:37:10 cc cmon[1357]: Schema : cmon: Could not connect to cmon database (config '/etc/cmon.cnf'): Can't connect to MySQL server on '127.0.0.1' (115) (errno: 2002)
At this point it's clear why CMON cannot start so
[root@cc ~]# journalctl -u mariadb
Aug 23 14:37:09 cc systemd[1]: Starting MariaDB database server...
Aug 23 14:37:10 cc mariadb-prepare-db-dir[1351]: Database MariaDB is probably initialized in /var/lib/mysql already, nothing is done.
Aug 23 14:37:10 cc mariadb-prepare-db-dir[1351]: If this is not the case, make sure the /var/lib/mysql is empty before running mariadb-prepare-db-dir.
Aug 23 14:37:10 cc mysqld_safe[1464]: 190823 14:37:10 mysqld_safe Logging to '/var/lib/mysql/cc.err'.
Aug 23 14:37:10 cc mysqld_safe[1464]: 190823 14:37:10 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
Aug 23 14:37:12 cc systemd[1]: Started MariaDB database server.
- 14:37:10 cmon failed to connect
- 14:37:12 mariadb started
Issue is this 2 seconds. Now how do I make cmon to wait for mariadb to start?
-
Official comment
Hi,
By default, cmon is installed with a cronjob to recover the service, it's located at /etc/cron.d/cmon. In case if MySQL is inaccessible, it keeps trying to connect to the MySQL server every minute right after that. Check /var/log/cmon.log and /var/log/cron for details.
So the answer is, you don't have to wait for MariaDB to start first. As long as MySQL/MariaDB is going to be started, cmon will recover the service by itself.
Regards,
AshrafComment actions -
Not auto relaunching cmon. after first failure waiting for mariadb
file exists under /etc/cron.d/cmon but it is completely empty!
Aug 27 11:00:01 ha1 crond[1047]: (*system*) RELOAD (/etc/cron.d/cmon)
cmon log desn't say anything four days empty while
2019-08-23 07:05:28 : (INFO) License is enterprise version.
2019-08-23 07:05:28 : (INFO) CmonCommandHandler started, polling for commands.
2019-08-23 07:05:28 : (INFO) Aborting DEQUEUED, RUNNING and DEFINED jobs
2019-08-27 02:59:23 : (INFO) Additional RPC URL for events: 'http://127.0.0.1:9510'
2019-08-27 02:59:23 : (INFO) Configuration loaded.
2019-08-27 02:59:23 : (INFO) Loaded configuration file '/etc/cmon.cnf'.
2019-08-27 02:59:23 : (INFO) Initializing the user manager.
2019-08-27 02:59:23 : (INFO) User manager is connecting the Cmon Database.
2019-08-27 02:59:23 : (INFO) Checking that the system users exist.
2019-08-27 02:59:23 : (INFO) Starting user manager thread.
2019-08-27 02:59:23 : (INFO) Server started at tcp://127.0.0.1:9500
2019-08-27 02:59:23 : (INFO) Server started at tls://127.0.0.1:9501
2019-08-27 02:59:23 : (INFO) Initializing controller manager.
2019-08-27 02:59:23 : (INFO) Cmon HA is disabled, running in single mode.
2019-08-27 02:59:23 : (INFO) Checking command handler.
2019-08-27 02:59:23 : (INFO) CDT entry '/.runtime/controller_clock' found.
2019-08-27 02:59:23 : (INFO) Starting the command handler.
2019-08-27 02:59:23 : (INFO) Loading configuration file '/etc/cmon.d/cmon_2.cnf'.
2019-08-27 02:59:23 : (INFO) cmon 1.7.3.3374
2019-08-27 02:59:23 : (INFO) CmonDb database is 'cmon' with schema version 107030.
2019-08-27 02:59:23 : (INFO) Running cmon schema hot-fixes.
2019-08-27 02:59:23 : (INFO) Applying modifications from 'cmon_db_mods_hotfix.sql'.
2019-08-27 02:59:23 : (INFO) Loaded configuration file '/etc/cmon.d/cmon_2.cnf'.
2019-08-27 02:59:23 : (INFO) Starting main loop.
2019-08-27 02:59:23 : (INFO) Schema auto-upgrade succeed (version 107030).
2019-08-27 02:59:23 : (INFO) Applying modifications from 'cmon_db_mods_hotfix.sql'.
2019-08-27 02:59:23 : (INFO) Checked tables - everything is ok.
2019-08-27 02:59:23 : (WARNING) Alarm raised: License expiration.
2019-08-27 02:59:23 : (INFO) License is enterprise version.
2019-08-27 02:59:23 : (INFO) CmonCommandHandler started, polling for commands. -
Hi,
We found a bug in the cmon service to properly handle cmon process recovery in case the MySQL/MariaDB is not started. This has been fixed in the latest release clustercontrol-controller-1.7.3-3388. Details at https://severalnines.com/docs/changelog.html
Please upgrade to the latest version and try again. Upgrade instructions at https://severalnines.com/docs/administration.html#upgrading-clustercontrol
Regards,
Ashraf
Please sign in to leave a comment.
Comments
3 comments