CMON does not pull Network, CPU, Disk stat from Nodes

Comments

13 comments

  • Avatar
    Johan

    Hi,

    What version of CMON are you using? 

    rpm -qa |grep cmon

    BR

    johan

  • Avatar
    Anushka Dissanayaka

    Hello Johan,

    Its cmon-controller-1.2.5-137.x86_64

    Thanks.

  • Avatar
    Johan

    Hi,

    And can you from the controller ssh without passwords to the other nodes?
    The best would be to the see /var/log/cmon.log from the controller.

    If you want to put it here or if you want to create a support ticket for it.

    BR
    johan

  • Avatar
    Anushka Dissanayaka

    Hi,

    Yes, I can ssh into all nodes. Log file attached.

  • Avatar
    Ashraf Sharif

    Hi,

    Can you make sure that in ClusterControl's /etc/cmon.cnf contains following line:

    agentless=1

    If it does not exist, please add it and restart cmon by using "service cmon restart".

    Regards,

    Ashraf

  • Avatar
    Anushka Dissanayaka

    Hello Ashraf,

    Thanks It worked. Is that due to a bug in installer? Because i am certain that i used agentless method.

     

  • Avatar
    Ashraf Sharif

    Hi Anushka,

    I need to double check on this. May I know what is your environment (DB cluster type/vendor, OS distribution, virtualization platform etc)? Did you use bootstrap script from http://www.severalnines.com/downloads/cmon/cc-bootstrap.tar.gz?

    Regards,

    Ashraf

  • Avatar
    Anushka Dissanayaka

    Hi Ashraf,

    Below are the information you requested.

    Cluster Type : Galera(PXC)

    OS : CentOS 6.4 x64

    Platform : On VMWare and EC2 separate clusters


    And yes i downloaded the script from the location you mentioned.

    Also we have another cluster setup on DigitalOcean and cmon deployed with agentless=1 and all worked fine. Even though i used same method for installing, on EC2 and VMware agentless=1 is not presented.

     

  • Avatar
    Chris Church

    I'm having this same problem on ubuntu 12.04, also with PXC.  This cluster was configured in the last few weeks with the latest version on severalnines.com.

    Interestingly, I can see the top for each node, just not cpu/disk/etc.  Also, the stats don't work for the controller either.

    Yes, passwordless SSH works and agentless=1.

    This was set up using the severalnines "install over an existing cluster" option (the nodes were created independently of the cluster monitoring server).

    The mysql stats all work fine, top shows up, and the cluster monitor will restart mysql if I manually stop it on a node.

     

    Was this problem resolved in March of last year?

  • Avatar
    Alex

    Hi

    If you are on 1.2.8 or older then can you please try to upgrade to our latest version 1.2.9 and see if that issue persists.

    Follow the instructions in this post, http://support.severalnines.com/entries/21095371-CMON-1-2-9-released-upgrade-instructions

  • Avatar
    Johan

    Hi Chris,

    Are you running on bare metal, or on VMs? Do you have a  /proc on the nodes (some virtualizers do not) and do you have e.g /proc/diskstats ?

    BR

    johan

  • Avatar
    Chris Church

    I'm running on VMs at a hosted datacenter.  As the cmon user, /proc/diskstats (and meminfo and cpuinfo) are all readable on the nodes.

    I'm running 1.2.8.  I can try upgrading.

  • Avatar
    Chris Church

    Looking at the fairly jenky upgrade process to go from 1.2.8 to 1.2.9, that's not something I can turn around quickly.  I was hoping that there was something in configuration or logs that could be relevant, though I see nothing in cmon.log.

    For what it's worth, I can see the following process periodically running on the cluster monitoring server:


    /bin/bash -c 'for f in /proc/[0-9][0-9][0-9]*/stat /proc/[0-9][0-9][0-9]*/statm /proc/[0-9][0-9][0-9]*/status /proc/uptime; do echo %f$f; cat $f | sed -e "s/%/%%/g"; done 2>/dev/null'

    Running this command as the cmon user produces a giant list of data.

Please sign in to leave a comment.

Powered by Zendesk