cmon extremely high CPU usage

Hi there,

Recently I've updated ClusterControl Community edition from version 1.5.1-4395.x86_64 to 1.7.0-5455.x86_64. 

After the reboot CC UI seemed perfectly fine, and the server was generally responsive. 

However, couple days later we started seeing increased CPU usage that gets to the point where the system is fully unresponsive. Neither SSH nor Web interfaces are operational and we have to hard reset the VM. Then the cycle repeats, good for some time, but the CPU usage keeps growing until it crashes the system.

I've searched through community posts and documentation. I found it's recommended to make sure PTR records exist for all cluster nodes and that /var/log/wtmp is kept small (rotated daily).

Does anyone know what could be the issue? What else can we try to resolve the issue? 

I wasn't sure if schema upgrade completed successfully, because I see the following the lines in the log every time cmon is (re)started.
2018-12-13 17:17:59 : (INFO) Running cmon schema hot-fixes.
2018-12-13 17:18:00 : (INFO) Schema auto-upgrade succeed (version 107003).

Here is some information about our system:

OS: CentOS Linux release 7.6.1810 (Core) - latest updates

YUM history:
Updated clustercontrol-1.5.1-4395.x86_64 @s9s-repo
Update 1.7.0-5455.x86_64 @s9s-repo
Updated clustercontrol-cloud-1.5.0-31.x86_64 @s9s-repo
Update 1.7.0-154.x86_64 @s9s-repo
Updated clustercontrol-clud-1.5.0-31.x86_64 @s9s-repo
Update 1.7.0-154.x86_64 @s9s-repo
Updated clustercontrol-cmonapi-1.5.0-290.x86_64 @s9s-repo
Update 1.7.0-333.x86_64 @s9s-repo
Updated clustercontrol-controller-1.5.1-2390.x86_64 @s9s-repo
Update 1.7.0-2904.x86_64 @s9s-repo
Updated clustercontrol-notifications-1.5.0-73.x86_64 @s9s-repo
Update 1.7.0-153.x86_64 @s9s-repo
Updated clustercontrol-ssh-1.5.0-39.x86_64 @s9s-repo
Update 1.7.0-66.x86_64 @s9s-repo

Log lines with Debug on:
2018-12-13 17:17:54 : (INFO) Server started at tcp://127.0.0.1:9500
2018-12-13 17:17:54 : (INFO) Server started at tls://127.0.0.1:9501
2018-12-13 17:17:59 : (INFO) Found 'cmon' schema version 107003.
2018-12-13 17:17:59 : (DEBUG) cmon_db.sql found 107003 expected 107003.
2018-12-13 17:17:59 : (DEBUG) cmon_db_mods-1.6.2-1.7.0.sql found 107001 expected 107001.
2018-12-13 17:17:59 : (INFO) Running cmon schema hot-fixes.
2018-12-13 17:18:00 : (INFO) Schema auto-upgrade succeed (version 107003).
2018-12-13 17:18:00 : (DEBUG) CmonDb executed: SELECT TABLE_NAME FROM information_schema.tables WHERE TABLE_SCHEMA='cmon' and TABLE_NAME='email_notification'
2018-12-13 17:18:00 : (DEBUG) CmonDb executed: SELECT TABLE_NAME FROM information_schema.tables WHERE TABLE_SCHEMA='cmon' and TABLE_NAME='cluster_log'
2018-12-13 17:18:00 : (DEBUG) CmonDb executed: SELECT TABLE_NAME FROM information_schema.tables WHERE TABLE_SCHEMA='cmon' and TABLE_NAME='node_state'
2018-12-13 17:18:00 : (DEBUG) CmonDb executed: SELECT TABLE_NAME FROM information_schema.tables WHERE TABLE_SCHEMA='cmon' and TABLE_NAME='mysql_server'
2018-12-13 17:18:00 : (DEBUG) CmonDb executed: SELECT TABLE_NAME FROM information_schema.tables WHERE TABLE_SCHEMA='cmon' and TABLE_NAME='mysql_backup'
2018-12-13 17:18:00 : (DEBUG) CmonDb executed: SELECT TABLE_NAME FROM information_schema.tables WHERE TABLE_SCHEMA='cmon' and TABLE_NAME='mysql_processlist'
2018-12-13 17:18:00 : (DEBUG) CmonDb executed: SELECT TABLE_NAME FROM information_schema.tables WHERE TABLE_SCHEMA='cmon' and TABLE_NAME='mysql_performance_meta'
2018-12-13 17:18:00 : (DEBUG) CmonDb executed: SELECT TABLE_NAME FROM information_schema.tables WHERE TABLE_SCHEMA='cmon' and TABLE_NAME='mysql_performance_probes'
2018-12-13 17:18:00 : (DEBUG) CmonDb executed: SELECT TABLE_NAME FROM information_schema.tables WHERE TABLE_SCHEMA='cmon' and TABLE_NAME='mysql_performance_results'
2018-12-13 17:18:00 : (DEBUG) CmonDb executed: SELECT TABLE_NAME FROM information_schema.tables WHERE TABLE_SCHEMA='cmon' and TABLE_NAME='simple_alarm'
2018-12-13 17:18:00 : (DEBUG) CmonDb executed: SELECT TABLE_NAME FROM information_schema.tables WHERE TABLE_SCHEMA='cmon' and TABLE_NAME='cluster'
2018-12-13 17:18:00 : (DEBUG) CmonDb executed: SELECT TABLE_NAME FROM information_schema.tables WHERE TABLE_SCHEMA='cmon' and TABLE_NAME='mailserver'
2018-12-13 17:18:00 : (DEBUG) CmonDb executed: SELECT TABLE_NAME FROM information_schema.tables WHERE TABLE_SCHEMA='cmon' and TABLE_NAME='cmon_configuration'
2018-12-13 17:18:00 : (DEBUG) CmonDb executed: SELECT TABLE_NAME FROM information_schema.tables WHERE TABLE_SCHEMA='cmon' and TABLE_NAME='cmon_job'
2018-12-13 17:18:00 : (DEBUG) CmonDb executed: SELECT TABLE_NAME FROM information_schema.tables WHERE TABLE_SCHEMA='cmon' and TABLE_NAME='cmon_host_log'
2018-12-13 17:18:00 : (DEBUG) CmonDb executed: SELECT TABLE_NAME FROM information_schema.tables WHERE TABLE_SCHEMA='cmon' and TABLE_NAME='cmon_job_message'
2018-12-13 17:18:00 : (DEBUG) CmonDb executed: SELECT TABLE_NAME FROM information_schema.tables WHERE TABLE_SCHEMA='cmon' and TABLE_NAME='backup_schedule'
2018-12-13 17:18:00 : (DEBUG) CmonDb executed: SELECT TABLE_NAME FROM information_schema.tables WHERE TABLE_SCHEMA='cmon' and TABLE_NAME='hosts'
2018-12-13 17:18:00 : (DEBUG) CmonDb executed: SELECT TABLE_NAME FROM information_schema.tables WHERE TABLE_SCHEMA='cmon' and TABLE_NAME='processes'
2018-12-13 17:18:00 : (DEBUG) CmonDb executed: SELECT TABLE_NAME FROM information_schema.tables WHERE TABLE_SCHEMA='cmon' and TABLE_NAME='mysql_slow_queries'
2018-12-13 17:18:00 : (DEBUG) CmonDb executed: SELECT TABLE_NAME FROM information_schema.tables WHERE TABLE_SCHEMA='cmon' and TABLE_NAME='mysql_explains'
2018-12-13 17:18:00 : (DEBUG) CmonDb executed: SELECT TABLE_NAME FROM information_schema.tables WHERE TABLE_SCHEMA='cmon' and TABLE_NAME='cmon_uploads'
2018-12-13 17:18:00 : (DEBUG) CmonDb executed: SELECT TABLE_NAME FROM information_schema.tables WHERE TABLE_SCHEMA='cmon' and TABLE_NAME='collected_logs'
2018-12-13 17:18:00 : (DEBUG) CmonDb executed: SELECT TABLE_NAME FROM information_schema.tables WHERE TABLE_SCHEMA='cmon' and TABLE_NAME='outgoing_messages'
2018-12-13 17:18:00 : (DEBUG) CmonDb executed: SELECT TABLE_NAME FROM information_schema.tables WHERE TABLE_SCHEMA='cmon' and TABLE_NAME='message_filters'
2018-12-13 17:18:00 : (DEBUG) CmonDb executed: SELECT TABLE_NAME FROM information_schema.tables WHERE TABLE_SCHEMA='cmon' and TABLE_NAME='server_node'
2018-12-13 17:18:00 : (DEBUG) CmonDb executed: SELECT TABLE_NAME FROM information_schema.tables WHERE TABLE_SCHEMA='cmon' and TABLE_NAME='cmon_stats'
2018-12-13 17:18:00 : (DEBUG) CmonDb executed: SELECT TABLE_NAME FROM information_schema.tables WHERE TABLE_SCHEMA='cmon' and TABLE_NAME='cmon_stats_daily'
2018-12-13 17:18:00 : (DEBUG) CmonDb executed: SELECT TABLE_NAME FROM information_schema.tables WHERE TABLE_SCHEMA='cmon' and TABLE_NAME='cmon_stats_weekly'
2018-12-13 17:18:00 : (DEBUG) CmonDb executed: SELECT TABLE_NAME FROM information_schema.tables WHERE TABLE_SCHEMA='cmon' and TABLE_NAME='cmon_stats_monthly'
2018-12-13 17:18:00 : (DEBUG) CmonDb executed: SELECT TABLE_NAME FROM information_schema.tables WHERE TABLE_SCHEMA='cmon' and TABLE_NAME='cmon_stats_yearly'
2018-12-13 17:18:00 : (DEBUG) CmonDb executed: SELECT TABLE_NAME FROM information_schema.tables WHERE TABLE_SCHEMA='cmon' and TABLE_NAME='scripts'
2018-12-13 17:18:00 : (DEBUG) CmonDb executed: SELECT TABLE_NAME FROM information_schema.tables WHERE TABLE_SCHEMA='cmon' and TABLE_NAME='scripts_schedule'
2018-12-13 17:18:00 : (DEBUG) CmonDb executed: SELECT TABLE_NAME FROM information_schema.tables WHERE TABLE_SCHEMA='cmon' and TABLE_NAME='scripts_results'
2018-12-13 17:18:00 : (DEBUG) CmonDb executed: SELECT TABLE_NAME FROM information_schema.tables WHERE TABLE_SCHEMA='cmon' and TABLE_NAME='scripts_audit_log'
2018-12-13 17:18:00 : (DEBUG) CmonDb executed: SELECT TABLE_NAME FROM information_schema.tables WHERE TABLE_SCHEMA='cmon' and TABLE_NAME='opreports'
2018-12-13 17:18:00 : (DEBUG) CmonDb executed: SELECT TABLE_NAME FROM information_schema.tables WHERE TABLE_SCHEMA='cmon' and TABLE_NAME='opreports_schedule'
2018-12-13 17:18:00 : (DEBUG) CmonDb executed: SELECT TABLE_NAME FROM information_schema.tables WHERE TABLE_SCHEMA='cmon' and TABLE_NAME='certificate_data'
2018-12-13 17:18:00 : (DEBUG) CmonDb executed: SELECT TABLE_NAME FROM information_schema.tables WHERE TABLE_SCHEMA='cmon' and TABLE_NAME='cluster_events'
2018-12-13 17:18:00 : (DEBUG) CmonDb executed: SELECT TABLE_NAME FROM information_schema.tables WHERE TABLE_SCHEMA='cmon' and TABLE_NAME='node_events'
2018-12-13 17:18:00 : (DEBUG) CmonDb executed: SELECT TABLE_NAME FROM information_schema.tables WHERE TABLE_SCHEMA='cmon' and TABLE_NAME='cmondb_version'
2018-12-13 17:18:00 : (INFO) Checked tables - seems ok
2018-12-13 17:18:00 : (INFO) Community version
2018-12-13 17:18:00 : (INFO) CmonCommandHandler: started, polling for commands.
2018-12-13 17:18:00 : (INFO) Aborting DEQUEUED, RUNNING and DEFINED jobs