Installation on an existing Cluster

Follow

Comments

79 comments

  • Avatar
    Raul Zuleta

    Hi!

    Im trying the new 1.1.11 release with no success , i did each step of this guide and the cmon daemon seems to work , but shows primary key erros in the CMON Database, here's an example :

    Nov 08 20:41:31 : (WARNING) Query replace into memory_usage(cid,nodeid,pg_used_dm,pg_tot_dm,pg_used_im,pg_tot_im,pg_sz_dm,pg_sz_im,dm_used_mb,im_used_mb,dm_total_mb,im_total_mb,report_ts) values (1,12,37, 224000, 59, 131200, 32768, 8192,1,0,7000,1025,now()) failed: Error: Duplicate entry '1-0-2011-11-08 20:41:31' for key 'PRIMARY'

    Im using the last schema version (shown in this page) and mysql 5.1.x (MySQL-Cluster 7.1) ,

    Any hint in this issue?

    Thanks for the support!

    Raul

     

     

  • Avatar
    Johan

    Hi,

    We will fix this in the next release.

    Can you send more of your cmon log details (i would be grateful if you could log it as a Support Ticket - easier to track)?

    BR

    johan

  • Avatar
    Johan

    Hi again Raul,

    If you could:

    edit the cmon_db.sql

    * Search for 'INSERT INTO memory_usage_history'

    * replace NEW.report_ts   with NULL

    THen do:

    mysql -uroot -p  < cmon_db.sql

    to reload the triggers.

    Does this help?

    br

    Johan

  • Avatar
    Michael Sasek

    These instructions appear to be incomplete. There is an entire section about creating SSH keys that is missing. In order for this to work, you also need to set up public key access via ssh for a user on all the cluster servers and the cluster control server.

  • Avatar
    Michael Sasek

    I am having a lot of problems with these instructions. A lot of information is simply missing from this tutorial.

    I have set up everything per the instructions, but I am still getting errors. Several questions:

    1. What should the cmon_core_dir be set to? I have not installed any core scripts besides those that install with the RPMs. 

    2. Why do I see this error status in the cluster control dashboard:

     

    NO CONTACT WITH NDB_MGMD

    CMON lost contact to the management server

     

     

  • Avatar
    Johan

    Hi Michael,

    If you want to install Clustercontrol for MySQL Cluster you should add (on the controller, one of them cmons' is the boss):

    ndb-connectstring=mgmt_server_1,mgmt_server_2

    type=mysqlcluster **

    **mode=controller

    Attached is an example.

    You don't have  to set the cmon_core_dir to anything meaningful (reason for this is further down), but it must be set. Set it to e.g:

    *cmon_core_dir=/root/clustercontrol

    *

    and do:

    *mkdir -p /root/clustercontrol/cluster/config

    *

    Copy your config.ini and my.cnf that you use into this directory (with a licensed version you will be able to import the configuration files into ClusterControl and perform configuration changes.

    If you set the ndb_connectstring (and mysql_hostname=<hostname of mysqld holding cmon db, not 127.0.0.1>, and have GRANTs) correctly you should now see management servers and data nodes in the dashboard, otherwise, the /var/log/cmon.log will tell you if there are problems with the GRANTs.

    However, you should also add the agents on each host, it is the same cmon.cnf, except that you have to set:

    mode=agent

    I will update the instructions tomorrow for how to install this on MySQL Cluster, but I hope you will get going with this.

    You will also need to set GRANTs so that the CMON (controller) is allowed to connect to the MySQL servers you have (using the 'cmon' user) and also tat the agents are allowed to connect to the mysql server having the CMON DB.

    But I will write this in more detail tomorrow.

    About the cmon_core_dir:

    ClusterControl is included when you generate a deployment package at www.severalnines.com/cluster-configurator

    The cmon_core_dir points to the top of this deployment package. If you get a license for ClusterControl, then CMON will import the configuration from {cmon_core_dir}/cluster/config

    Also, the current version uses a couple of scripts (in the licensed version)  in that directory to perform rolling restarts. However, we are moving away from the scripts and have almost completed implementation of the those scripts in C++ code instead. This to avoid the dependency of the deployment package.

    I hope this helps and if not don't hesitate to contact us again.

    Thank you ,

    Johan

     

  • Avatar
    Michael Sasek

    I am installing cluster control to work with an existing cluster, so I all I need to do is configure it to connect to the other servers. It seems like it should be straightforward, but I cannot get it to work. I don't need cluster control to install the cluster, just see the existing one.

    There are the following errors constantly showing up in the cluster controller dashboard:

    Galera Cluster - 'default'

    | cluster id | cluster name | status | heartbeat | actions |
    | --- | --- | --- | --- | --- |
    |
    1
    | default
    | NO CONTACT WITH 

    NDB_MGMD
    | 2012-01-03 16:25:36
    |

     

    And:

    Database and Process Alarms

    |   | node id | component / resource | alarm | alarm count | exit message | description | recommendation | updated | dismiss |
    | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
    |

    | 0
    | CLUSTER
    | CLUSTER STATUS UNKNOWN
    | 0
    |  
    | CMON lost contact to the management server
    | Start the management servers.
    | 2012-01-03 12:37:06
    |

     

     

    I just need to know what NGB_MGMD is and where I can find it and place it properly, There is no mention of this in the tutorial or galera documentation.

     

    Next, you say:

    "Copy your config.ini and my.cnf that you use into this directory"

    Where would I find a config.ini? I don't see that anywhere in the tutorial either. 

  • Avatar
    Johan

    Hi,

    Given the error I assumed you were on MySQL Cluster for some reason (since you are on Galera the config.ini stuff does not apply at all here).

    However, can you send me the cmon.cnf file you have?

    BR

  • Avatar
    Michael Sasek

    I should have specified the platform. I forgot that this product works with so many.

    I have been slowly working out the problems, and managed to get both the controller server and one galera node to start the cmon agent and the logs say they are happily talking to each other, but no galera cluster is detected in the web interface and the status still remains reported as errors.

     

    I no longer see any errors in the cmon.log regarding connectivity, and they seem to be connected, just not showing up without erros in the dashboard or hosts section of the web interface. The current error for the galera node is that it is not pingable, even though it is if I ping from the controller server (using host name, fqdn and IP, all pingable in both directions). I also keep seeing the same error about losing connection to the controller even though it is up and running.

     

     

    Posted below are my cmon.cnf files for the controller and one of the galera nodes.

     

    controller conf:

    #cmon config file

    id and name of cluster that this cmon agent is monitoring.

    Must be unique for each monitored cluster, like server-id in mysql

    cluster_id=1

    name=default

    mode=controller, agent, or dual (if the cmon db is on a mysql server connected to MySQL Cluster)

    mode=controller

    this is the controller and we monitor mysql cluster (can be set to 'replication' for mysql repl)

    type=galera

    #config for mysqld holding the cmon database (owned by the 'cmon' mysql user)

    mysql_port=3306

    mysql_hostname=ccontrol

    mysql_password=password

    #how the agent should connect to the local mysql server

    local_mysql_port=3306

    local_mysql_password=password

    local_mysql_hostname=ccontrol

    #connectstring to mysql cluster (comma separate the hosts, if you have more than one management server)

    ndb_connectstring=ccontrol

    #port CMON is listening on for incoming commands (like stop node, start node etc)

    cmon_port=50001

    #The user who owns the core scripts specified below

    osuser=root

    #location of Configurator script root, e.g /root/mysqlcluster-71/

    cmon_core_dir=/root/ccontrol/

    #location of cmon.pid file. The pidfile is written in /tmp/ by default

    pidfile=/var/run/

    #logfile is default to syslog.

    logfile=/var/log/cmon.log

    #BUG right now - start/stop nodes/getting logs/start backups etc does not work in daemon mode

    at this point

    nodaemon=1

    #save clusterlog and stats events for 24 hours

    savetime=24

    RRD SECTION - graphs ######

    mysql_bindir=/usr/bin/

    imagedir=/var/www/html/cmon/graphs/

    rrdtool=/usr/bin/rrdtool

    rrd_datadir=/var/lib/cmon/

     

    galera node conf:

    #cmon config file

    id of cluster that this cmon agent is monitoring.

    Must be unique for each monitored cluster, like server-id in mysql

    cluster_id=1

    #type (mysqlcluster, replication)

    mode=agent

    type=galera

    name=default

    #port of mysql server holding cmon database

    mysql_port=3306

    #hostname/ip of mysql server holding cmon database

    mysql_hostname=ccontrol

    #password for 'cmon' user on the 'mysql_hostname'

    mysql_password=password

    #how the agent should connect to the local mysql server

    local_mysql_port=3306

    local_mysql_password=password

    local_mysql_hostname=lb-mysql1

    #connectstring to mysql cluster (comma separate the hosts, if you have more than one management server)

    ndb_connectstring=ccontrol

    #port CMON is listening on for incoming commands (like stop node, start node etc)

    cmon_port=50001

    #location of Configurator script root, e.g /root/mysqlcluster-71/

    cmon_core_dir=/root/mysqlcluster-71/

    #The user who owns the core scripts

    osuser=root

    #logfile is default to syslog.

    logfile=/var/log/cmon.log

    #location of cmon.pid file. The pidfile is written in /tmp/ by default

    pidfile=/var/run/

    #cmon will daemonize when started, uncomment nodaemon if you want it to be in foreground

    #BUG right now - start/stop nodes/getting logs/start backups etc does not work in daemon mode

    at this point

    nodaemon=1

    #save clusterlog and stats events for 24 hours

    savetime=24

    RRD SECTION - graphs ######

    mysql_bindir=/usr/local/mysql/bin/

    imagedir=/var/www/cmon/graphs/

    rrdtool=/usr/local/bin/rrdtool

    rrd_datadir=/var/lib/cmon/

  • Avatar
    Michael Sasek

    I think I have found the final issue. I needed to modify the /etc/hosts file. I already had the hosts names on both servers, but on the controller, it was resolving its host name to a private IP. I corrected that and now both controller and galera node are showing up as operational. I am connecting the other nodes in the cluster now, and it looks like smooth sailing for the rest of the configuration. 

    Thanks for taking the time to explain things to me. It was very helpful and pointed me to where my mistakes were!

  • Avatar
    Michael Sasek

    One problem remains. Although everyhthing is now working for the most part, the controller still reports the cluster status as yellow with the status of:

    NO CONTACT WITH 

    NDB_MGMD

    Any idea why I still get this?

  • Avatar
    Johan

    Hi,

    Did you ever start with "type=mysqlcluster" in /etc/cmon.cnf (because it is MySQL Cluster specific)?

    There could then be some old data laying around.

    You could do (since you know have it more or less working):

    drop database cmon;

    source cmon_db...sql

    source cmon_data...sql

    restart the cmons'.

    We are about to release a new version tomorrow that fixes many issues and make installation a lot easier.

    Thank you,

    johan

  • Avatar
    Michael Sasek

    I dropped the db and recreated it. No change in the error status. It detects the nodes but says there is no cluster, even though it sees all the nodes and is displaying all the cluster graphs. Odd indeed.  I will wait for the next release and upgrade, maybe that will fix my issue.

  • Avatar
    kerlion.he

    Hi Johan

         You did not mention about creating SSH keys, do you need to setup SSH keys, does the CMON controller need to execute OS commands remotely via passwordless SSH?

     Kerlion He

  • Avatar
    Johan

    Hi Kertion,

    Yes to be able to do management (start/stop/install/export configs etc) then passwordless ssh is needed.

    Moreover, we are simplifying the installation of this, and I hope to update the article later today/tomorrow.

    Best regards,

    Johah

  • Avatar
    Paul Baclace

    The cmon overview should mention how the cmon agents gather and report information.  I'm trying this out now, and noticed that the cmon agent tries to directly contact the mysql running where the cmon reports are made available via apache2.   It looks like a monitoring system that is "push" based rather than "pull".  A dedicated mysql (single machine, not replicated) collects the state from all the cmon agents and then php is used to generate reports from that.  Options to have some network segments be WAN instead of LAN would be nice. 

    Is that correct?

    Unless I missed it, the cmon installation (as part of mysql replication) did not supply an apache2 config, so I'm at a loss to see it working since I have no knowledge of php configuration best practices.

  • Avatar
    Severalnines Support

    Paul, your description is correct.

    Regarding the apache config, this is also automated.

    Did the deploy.sh raise any error during the final stages (that is when apache + php is installed) ?

    Best regards,

    Vinay

  • Avatar
    Johan Wikström

    Hi there,

    I must be missing something, I have tried these instructions on my debian based MySQL cluster, both manually and then I found the cmon_install_agent.sh in the bin dir,

    so I tried that as well. However, I get on the SQL server where the agent is located:

    Feb 12 11:06:17 : (INFO) IPv4 address: IP

    Feb 12 11:06:17 : (INFO) Testing connection to mysqld..

    Feb 12 11:06:17 : (INFO) IPv4 address: IP

    Feb 12 11:06:17 : (INFO) Setting up threads to handle monitoring

    Feb 12 11:06:17 : (INFO) Starting MySQLCollectorThread

    Feb 12 11:06:17 : (INFO) Connected to MySQL Server @ 127.0.0.1

    Feb 12 11:06:17 : (INFO) Hostname=IP, ip=IP, skip_name_resolve=1

    Feb 12 11:06:17 : (INFO) Connected to local mysql server!

    Feb 12 11:06:17 : (ERROR) Critical error (mysql error code -1) occured - shutting down

     

    The grants seem to be OK, the cmon user can connect between the nodes. Do I need to create the cmon database in the monitored cluster as well?

    It should only be created on the Cluster Control server, right? Is there any way to debug the command that leads to mysql error code -1, I must have missed

    something very basic.

     

    Best regards,

    Johan

  • Avatar
    Johan

    Hi Johan

    The cmon db should only be created on the ClusterControl server. 

    Can you paste your cmon.cnf file of the failing agent here (take out the passwords + ips if you want)?

    Did you start up the 'controller' node?

    I will look on my side too.

    Best regards

    Johan

  • Avatar
    Johan Wikström

    Hi Johan,

     

    thank you very much, it was very basic indeed, I didn't have the controller node running. Now it starts fine, I'm going to add the other nodes and try out your product.

    It looks very promising!

     

    Best regards,

    Johan

  • Avatar
    Johan

    Hi, 

    That is great! Yes, so the first time the controller needs to be started. Then, in subsequent restarts, it is not so important in which order the nodes are started.

    Good luck and happy clustering!

    BR,

    Johan.

  • Avatar
    Johan

    Note: Updated the instructions to make it easier to install.

  • Avatar
    Daniël van Eeden
    If SELinux is active the following is needed to get httpd/php to connect to MySQL:  
    

    setsebool -P httpd_can_network_connect_db=1

    The filenames are /usr/share/cmon/cmon_{db,data}.sql (there are no files with 1.2.0 in the filename)

  • Avatar
    Johan

    Thanks Daniël, updated the article with your information!

  • Avatar
    Johan

    Just another thing, if you use e.g DSA, then you can set :

    ssh_identity=/path/to/id_dsa

    in /etc/cmon.cnf

    and then restart the controller.

    By default, RSA is assumed.

  • Avatar
    Scott Haas

    I've been working through the installation.  I have cmon started on the management node and the agents on the monitored nodes. 

    One question: I am seeing the following in /var/log/cmon.log (on management node) every 30 seconds

    Apr 09 10:28:20 : (WARNING) STAGING_DIR not set - set in 'Cluster Setup', using default '/tmp/'

    Apr 09 10:28:21 : (INFO) Cluster Started

    Apr 09 10:28:50 : (WARNING) STAGING_DIR not set - set in 'Cluster Setup', using default '/tmp/'

    Apr 09 10:28:52 : (INFO) Cluster Started

    When I selected information from cmon_configuration, there is no param/value entry for TMPDIR.

    Is /tmp the preferred value for TMPDIR?  Is there a preferred method of inserting TMPDIR parameter?

    Thanks,

    Scott

  • Avatar
    Scott Haas

    I was able to update "cluster config" through the web page on the management node.  I see that it inserted a row into cmon_configuration.

  • Avatar
    Johan

    Hi Scott, 

    Yes in our new gui (http://<ccserver>/clustercontrol), it seems to be missing to set this parameter, however, in the old gui (http://<ccserver>/cmon ) it is possible to set in Cluster Settings, scroll down to "Staging area:".

    You can also set it in cmon.cnf on the controller, e.g:

    staging_dir=/my/staging/dir

    Thanks for pointing it out.

    Best regards

    Johan

  • Avatar
    Sanket Gupta

    Hi

    I am trying to manually setup the controller on an Ubuntu 12.04 server to monitor an existing galera cluster

    I have copied the relevant files and created the db as well.The permissions have also been granted

    Unfortunately the monitor fails to start

    Tried the following

    #$  /etc/init.d/cmon start

    Starting cmon --config-file=/etc/cmon.cnf : failed

    #$ /usr/local/cmon/sbin/cmon --config-file /etc/cmon.cnf

    Invalid 'os' specified. os=redhat|debian

    Here is my config file

    ********************************************************************************************

    #cmon config file

    id and name of cluster that this cmon agent is monitoring.

    Must be unique for each monitored cluster, like server-id in mysql

    cluster_id=1

    name=my_cluster

    os = [redhat|debian] type of OS

    os=debian # skip_name_resolve = [0|1] - set 1 if you use ip addresses only everywhere

    skip_name_resolve=1 # mode = [controller|agent|dual]

    mode=controller

    type = [mysqlcluster|replication|galera]

    type=galera

    ndb_connectstring, comment it out if you don't use MySQL Cluster:

    #ndb_connectstring=mgm1,mgm2

    #port of mysql server holding cmon database

    mysql_port=3306

    #hostname/ip of mysql server holding cmon database

    mysql_hostname=127.0.0.1

    #password for 'cmon' user on  the 'mysql_hostname'

    mysql_password=cmon

    local_mysql_port=3306

    local_mysql_password=cmon

    #location of Configurator script root, e.g /root/mysqlcluster-71/

    cmon_core_dir=/root/cmon_coredir/

    #osuser - the user that can ssh from this host to the other hosts w/o passwords

    osuser=root

    #logfile is default to syslog.

    logfile=/var/log/cmon.log

    #location of cmon.pid file. The pidfile is written in /tmp/ by default

    pidfile=/var/run

    nodaemon=0

    db_stats_collection_interval=30

    host_stats_collection_interval=30

    mysql_basedir /usr/local/mysql (common location for mysql tgz install  or /usr  (common location for RPM)

    mysql_basedir=/usr

    NODE ADDRESSES:

    mysql servers in the cluster. "," or " " sep. list

    mysql_server_addresses=xx.xx.xx.xx, xx.xx.xx.yy,xx.xx.xx.zz

    mgm and data nodes are only used to MySQL Cluster "," or " " sep. list

    datanode_addresses=

    mgmnode_addresses=

    wwwroot: on RH/Centos use /var/www/html. Debian/Ubuntu use /var/www/

    wwwroot=/var/www/cluster.mydomain.in/

    ##Pay attention to the bold test above! These settings must be the changed.

    *************************************************************************************

  • Avatar
    Johan

    Hi,

    I see in what you pasted above:

    os=debian # skip_name_resolve = [0|1] - set 1 if you use ip addresses only everywhere

    skip_name_resolve=1 # mode = [controller|agent|dual]

    You are not allowed to have #comments on the same line as an conf variable. Please change to:

    os=debian **

    **# skip_name_resolve = [0|1] - set 1 if you use ip addresses only everywhere


    skip_name_resolve=1 

    **
    # mode = [controller|agent|dual]**

    mode=controller

    ....

    Also since you don't use mysql cluster, please remove the following lines:

    # mgm and data nodes are only used to MySQL Cluster "," or " " sep. list

    datanode_addresses=

    mgmnode_addresses=

     

    What happens now?

    BR

    johan 

Please sign in to leave a comment.

Powered by Zendesk