If you have not used the Configurator to setup your cluster you can still install ClusterControl on your existing database cluster. Make sure the database cluster is running in full capacity prior to this installation.
Please follow the instructions here in our Getting Started Guide.
Comments
82 comments
Hi!
Im trying the new 1.1.11 release with no success , i did each step of this guide and the cmon daemon seems to work , but shows primary key erros in the CMON Database, here's an example :
Nov 08 20:41:31 : (WARNING) Query replace into memory_usage(cid,nodeid,pg_used_dm,pg_tot_dm,pg_used_im,pg_tot_im,pg_sz_dm,pg_sz_im,dm_used_mb,im_used_mb,dm_total_mb,im_total_mb,report_ts) values (1,12,37, 224000, 59, 131200, 32768, 8192,1,0,7000,1025,now()) failed: Error: Duplicate entry '1-0-2011-11-08 20:41:31' for key 'PRIMARY'
Im using the last schema version (shown in this page) and mysql 5.1.x (MySQL-Cluster 7.1) ,
Any hint in this issue?
Thanks for the support!
Raul
Hi,
We will fix this in the next release.
Can you send more of your cmon log details (i would be grateful if you could log it as a Support Ticket - easier to track)?
BR
johan
Hi again Raul,
If you could:
edit the cmon_db.sql
* Search for 'INSERT INTO memory_usage_history'
* replace NEW.report_ts with NULL
THen do:
mysql -uroot -p < cmon_db.sql
to reload the triggers.
Does this help?
br
Johan
These instructions appear to be incomplete. There is an entire section about creating SSH keys that is missing. In order for this to work, you also need to set up public key access via ssh for a user on all the cluster servers and the cluster control server.
I am having a lot of problems with these instructions. A lot of information is simply missing from this tutorial.
I have set up everything per the instructions, but I am still getting errors. Several questions:
What should the cmon_core_dir be set to? I have not installed any core scripts besides those that install with the RPMs.
Why do I see this error status in the cluster control dashboard:
NO CONTACT WITH NDB_MGMD
CMON lost contact to the management server
Hi Michael,
If you want to install Clustercontrol for MySQL Cluster you should add (on the controller, one of them cmons' is the boss):
ndb-connectstring=mgmt_server_1,mgmt_server_2
type=mysqlcluster **
**mode=controller
Attached is an example.
You don't have to set the cmon_core_dir to anything meaningful (reason for this is further down), but it must be set. Set it to e.g:
*cmon_core_dir=/root/clustercontrol
*
and do:
*mkdir -p /root/clustercontrol/cluster/config
*
Copy your config.ini and my.cnf that you use into this directory (with a licensed version you will be able to import the configuration files into ClusterControl and perform configuration changes.
If you set the ndb_connectstring (and mysql_hostname=<hostname of mysqld holding cmon db, not 127.0.0.1>, and have GRANTs) correctly you should now see management servers and data nodes in the dashboard, otherwise, the /var/log/cmon.log will tell you if there are problems with the GRANTs.
However, you should also add the agents on each host, it is the same cmon.cnf, except that you have to set:
mode=agent
I will update the instructions tomorrow for how to install this on MySQL Cluster, but I hope you will get going with this.
You will also need to set GRANTs so that the CMON (controller) is allowed to connect to the MySQL servers you have (using the 'cmon' user) and also tat the agents are allowed to connect to the mysql server having the CMON DB.
But I will write this in more detail tomorrow.
About the cmon_core_dir:
ClusterControl is included when you generate a deployment package at www.severalnines.com/cluster-configurator
The cmon_core_dir points to the top of this deployment package. If you get a license for ClusterControl, then CMON will import the configuration from {cmon_core_dir}/cluster/config
Also, the current version uses a couple of scripts (in the licensed version) in that directory to perform rolling restarts. However, we are moving away from the scripts and have almost completed implementation of the those scripts in C++ code instead. This to avoid the dependency of the deployment package.
I hope this helps and if not don't hesitate to contact us again.
Thank you ,
Johan
I am installing cluster control to work with an existing cluster, so I all I need to do is configure it to connect to the other servers. It seems like it should be straightforward, but I cannot get it to work. I don't need cluster control to install the cluster, just see the existing one.
There are the following errors constantly showing up in the cluster controller dashboard:
Galera Cluster - 'default'
| cluster id | cluster name | status | heartbeat | actions |
| --- | --- | --- | --- | --- |
|
1
| default
| NO CONTACT WITH
NDB_MGMD
| 2012-01-03 16:25:36
|
And:
Database and Process Alarms
| | node id | component / resource | alarm | alarm count | exit message | description | recommendation | updated | dismiss |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|
| 0
| CLUSTER
| CLUSTER STATUS UNKNOWN
| 0
|
| CMON lost contact to the management server
| Start the management servers.
| 2012-01-03 12:37:06
|
I just need to know what NGB_MGMD is and where I can find it and place it properly, There is no mention of this in the tutorial or galera documentation.
Next, you say:
"Copy your config.ini and my.cnf that you use into this directory"
Where would I find a config.ini? I don't see that anywhere in the tutorial either.
Hi,
Given the error I assumed you were on MySQL Cluster for some reason (since you are on Galera the config.ini stuff does not apply at all here).
However, can you send me the cmon.cnf file you have?
BR
I should have specified the platform. I forgot that this product works with so many.
I have been slowly working out the problems, and managed to get both the controller server and one galera node to start the cmon agent and the logs say they are happily talking to each other, but no galera cluster is detected in the web interface and the status still remains reported as errors.
I no longer see any errors in the cmon.log regarding connectivity, and they seem to be connected, just not showing up without erros in the dashboard or hosts section of the web interface. The current error for the galera node is that it is not pingable, even though it is if I ping from the controller server (using host name, fqdn and IP, all pingable in both directions). I also keep seeing the same error about losing connection to the controller even though it is up and running.
Posted below are my cmon.cnf files for the controller and one of the galera nodes.
controller conf:
#cmon config file
id and name of cluster that this cmon agent is monitoring.
Must be unique for each monitored cluster, like server-id in mysql
cluster_id=1
name=default
mode=controller, agent, or dual (if the cmon db is on a mysql server connected to MySQL Cluster)
mode=controller
this is the controller and we monitor mysql cluster (can be set to 'replication' for mysql repl)
type=galera
#config for mysqld holding the cmon database (owned by the 'cmon' mysql user)
mysql_port=3306
mysql_hostname=ccontrol
mysql_password=password
#how the agent should connect to the local mysql server
local_mysql_port=3306
local_mysql_password=password
local_mysql_hostname=ccontrol
#connectstring to mysql cluster (comma separate the hosts, if you have more than one management server)
ndb_connectstring=ccontrol
#port CMON is listening on for incoming commands (like stop node, start node etc)
cmon_port=50001
#The user who owns the core scripts specified below
osuser=root
#location of Configurator script root, e.g /root/mysqlcluster-71/
cmon_core_dir=/root/ccontrol/
#location of cmon.pid file. The pidfile is written in /tmp/ by default
pidfile=/var/run/
#logfile is default to syslog.
logfile=/var/log/cmon.log
#BUG right now - start/stop nodes/getting logs/start backups etc does not work in daemon mode
at this point
nodaemon=1
#save clusterlog and stats events for 24 hours
savetime=24
RRD SECTION - graphs ######
mysql_bindir=/usr/bin/
imagedir=/var/www/html/cmon/graphs/
rrdtool=/usr/bin/rrdtool
rrd_datadir=/var/lib/cmon/
galera node conf:
#cmon config file
id of cluster that this cmon agent is monitoring.
Must be unique for each monitored cluster, like server-id in mysql
cluster_id=1
#type (mysqlcluster, replication)
mode=agent
type=galera
name=default
#port of mysql server holding cmon database
mysql_port=3306
#hostname/ip of mysql server holding cmon database
mysql_hostname=ccontrol
#password for 'cmon' user on the 'mysql_hostname'
mysql_password=password
#how the agent should connect to the local mysql server
local_mysql_port=3306
local_mysql_password=password
local_mysql_hostname=lb-mysql1
#connectstring to mysql cluster (comma separate the hosts, if you have more than one management server)
ndb_connectstring=ccontrol
#port CMON is listening on for incoming commands (like stop node, start node etc)
cmon_port=50001
#location of Configurator script root, e.g /root/mysqlcluster-71/
cmon_core_dir=/root/mysqlcluster-71/
#The user who owns the core scripts
osuser=root
#logfile is default to syslog.
logfile=/var/log/cmon.log
#location of cmon.pid file. The pidfile is written in /tmp/ by default
pidfile=/var/run/
#cmon will daemonize when started, uncomment nodaemon if you want it to be in foreground
#BUG right now - start/stop nodes/getting logs/start backups etc does not work in daemon mode
at this point
nodaemon=1
#save clusterlog and stats events for 24 hours
savetime=24
RRD SECTION - graphs ######
mysql_bindir=/usr/local/mysql/bin/
imagedir=/var/www/cmon/graphs/
rrdtool=/usr/local/bin/rrdtool
rrd_datadir=/var/lib/cmon/
I think I have found the final issue. I needed to modify the /etc/hosts file. I already had the hosts names on both servers, but on the controller, it was resolving its host name to a private IP. I corrected that and now both controller and galera node are showing up as operational. I am connecting the other nodes in the cluster now, and it looks like smooth sailing for the rest of the configuration.
Thanks for taking the time to explain things to me. It was very helpful and pointed me to where my mistakes were!
One problem remains. Although everyhthing is now working for the most part, the controller still reports the cluster status as yellow with the status of:
NO CONTACT WITH
NDB_MGMD
Any idea why I still get this?
Hi,
Did you ever start with "type=mysqlcluster" in /etc/cmon.cnf (because it is MySQL Cluster specific)?
There could then be some old data laying around.
You could do (since you know have it more or less working):
drop database cmon;
source cmon_db...sql
source cmon_data...sql
restart the cmons'.
We are about to release a new version tomorrow that fixes many issues and make installation a lot easier.
Thank you,
johan
I dropped the db and recreated it. No change in the error status. It detects the nodes but says there is no cluster, even though it sees all the nodes and is displaying all the cluster graphs. Odd indeed. I will wait for the next release and upgrade, maybe that will fix my issue.
Hi Johan
You did not mention about creating SSH keys, do you need to setup SSH keys, does the CMON controller need to execute OS commands remotely via passwordless SSH?
Kerlion He
Hi Kertion,
Yes to be able to do management (start/stop/install/export configs etc) then passwordless ssh is needed.
Moreover, we are simplifying the installation of this, and I hope to update the article later today/tomorrow.
Best regards,
Johah
The cmon overview should mention how the cmon agents gather and report information. I'm trying this out now, and noticed that the cmon agent tries to directly contact the mysql running where the cmon reports are made available via apache2. It looks like a monitoring system that is "push" based rather than "pull". A dedicated mysql (single machine, not replicated) collects the state from all the cmon agents and then php is used to generate reports from that. Options to have some network segments be WAN instead of LAN would be nice.
Is that correct?
Unless I missed it, the cmon installation (as part of mysql replication) did not supply an apache2 config, so I'm at a loss to see it working since I have no knowledge of php configuration best practices.
Paul, your description is correct.
Regarding the apache config, this is also automated.
Did the deploy.sh raise any error during the final stages (that is when apache + php is installed) ?
Best regards,
Vinay
Hi there,
I must be missing something, I have tried these instructions on my debian based MySQL cluster, both manually and then I found the cmon_install_agent.sh in the bin dir,
so I tried that as well. However, I get on the SQL server where the agent is located:
Feb 12 11:06:17 : (INFO) IPv4 address: IP
Feb 12 11:06:17 : (INFO) Testing connection to mysqld..
Feb 12 11:06:17 : (INFO) IPv4 address: IP
Feb 12 11:06:17 : (INFO) Setting up threads to handle monitoring
Feb 12 11:06:17 : (INFO) Starting MySQLCollectorThread
Feb 12 11:06:17 : (INFO) Connected to MySQL Server @ 127.0.0.1
Feb 12 11:06:17 : (INFO) Hostname=IP, ip=IP, skip_name_resolve=1
Feb 12 11:06:17 : (INFO) Connected to local mysql server!
Feb 12 11:06:17 : (ERROR) Critical error (mysql error code -1) occured - shutting down
The grants seem to be OK, the cmon user can connect between the nodes. Do I need to create the cmon database in the monitored cluster as well?
It should only be created on the Cluster Control server, right? Is there any way to debug the command that leads to mysql error code -1, I must have missed
something very basic.
Best regards,
Johan
Hi Johan
The cmon db should only be created on the ClusterControl server.
Can you paste your cmon.cnf file of the failing agent here (take out the passwords + ips if you want)?
Did you start up the 'controller' node?
I will look on my side too.
Best regards
Johan
Hi Johan,
thank you very much, it was very basic indeed, I didn't have the controller node running. Now it starts fine, I'm going to add the other nodes and try out your product.
It looks very promising!
Best regards,
Johan
Hi,
That is great! Yes, so the first time the controller needs to be started. Then, in subsequent restarts, it is not so important in which order the nodes are started.
Good luck and happy clustering!
BR,
Johan.
Note: Updated the instructions to make it easier to install.
setsebool -P httpd_can_network_connect_db=1
The filenames are /usr/share/cmon/cmon_{db,data}.sql (there are no files with 1.2.0 in the filename)
Thanks Daniël, updated the article with your information!
Just another thing, if you use e.g DSA, then you can set :
ssh_identity=/path/to/id_dsa
in /etc/cmon.cnf
and then restart the controller.
By default, RSA is assumed.
I've been working through the installation. I have cmon started on the management node and the agents on the monitored nodes.
One question: I am seeing the following in /var/log/cmon.log (on management node) every 30 seconds
Apr 09 10:28:20 : (WARNING) STAGING_DIR not set - set in 'Cluster Setup', using default '/tmp/'
Apr 09 10:28:21 : (INFO) Cluster Started
Apr 09 10:28:50 : (WARNING) STAGING_DIR not set - set in 'Cluster Setup', using default '/tmp/'
Apr 09 10:28:52 : (INFO) Cluster Started
When I selected information from cmon_configuration, there is no param/value entry for TMPDIR.
Is /tmp the preferred value for TMPDIR? Is there a preferred method of inserting TMPDIR parameter?
Thanks,
Scott
I was able to update "cluster config" through the web page on the management node. I see that it inserted a row into cmon_configuration.
Hi Scott,
Yes in our new gui (http://<ccserver>/clustercontrol), it seems to be missing to set this parameter, however, in the old gui (http://<ccserver>/cmon ) it is possible to set in Cluster Settings, scroll down to "Staging area:".
You can also set it in cmon.cnf on the controller, e.g:
staging_dir=/my/staging/dir
Thanks for pointing it out.
Best regards
Johan
Hi
I am trying to manually setup the controller on an Ubuntu 12.04 server to monitor an existing galera cluster
I have copied the relevant files and created the db as well.The permissions have also been granted
Unfortunately the monitor fails to start
Tried the following
#$ /etc/init.d/cmon start
Starting cmon --config-file=/etc/cmon.cnf : failed
#$ /usr/local/cmon/sbin/cmon --config-file /etc/cmon.cnf
Invalid 'os' specified. os=redhat|debian
Here is my config file
********************************************************************************************
#cmon config file
id and name of cluster that this cmon agent is monitoring.
Must be unique for each monitored cluster, like server-id in mysql
cluster_id=1
name=my_cluster
os = [redhat|debian] type of OS
os=debian # skip_name_resolve = [0|1] - set 1 if you use ip addresses only everywhere
skip_name_resolve=1 # mode = [controller|agent|dual]
mode=controller
type = [mysqlcluster|replication|galera]
type=galera
ndb_connectstring, comment it out if you don't use MySQL Cluster:
#ndb_connectstring=mgm1,mgm2
#port of mysql server holding cmon database
mysql_port=3306
#hostname/ip of mysql server holding cmon database
mysql_hostname=127.0.0.1
#password for 'cmon' user on the 'mysql_hostname'
mysql_password=cmon
local_mysql_port=3306
local_mysql_password=cmon
#location of Configurator script root, e.g /root/mysqlcluster-71/
cmon_core_dir=/root/cmon_coredir/
#osuser - the user that can ssh from this host to the other hosts w/o passwords
osuser=root
#logfile is default to syslog.
logfile=/var/log/cmon.log
#location of cmon.pid file. The pidfile is written in /tmp/ by default
pidfile=/var/run
nodaemon=0
db_stats_collection_interval=30
host_stats_collection_interval=30
mysql_basedir /usr/local/mysql (common location for mysql tgz install or /usr (common location for RPM)
mysql_basedir=/usr
NODE ADDRESSES:
mysql servers in the cluster. "," or " " sep. list
mysql_server_addresses=xx.xx.xx.xx, xx.xx.xx.yy,xx.xx.xx.zz
mgm and data nodes are only used to MySQL Cluster "," or " " sep. list
datanode_addresses=
mgmnode_addresses=
wwwroot: on RH/Centos use /var/www/html. Debian/Ubuntu use /var/www/
wwwroot=/var/www/cluster.mydomain.in/
##Pay attention to the bold test above! These settings must be the changed.
*************************************************************************************
Hi,
I see in what you pasted above:
os=debian # skip_name_resolve = [0|1] - set 1 if you use ip addresses only everywhere
skip_name_resolve=1 # mode = [controller|agent|dual]
You are not allowed to have #comments on the same line as an conf variable. Please change to:
os=debian **
**# skip_name_resolve = [0|1] - set 1 if you use ip addresses only everywhere
skip_name_resolve=1
**# mode = [controller|agent|dual]**
mode=controller
....
Also since you don't use mysql cluster, please remove the following lines:
# mgm and data nodes are only used to MySQL Cluster "," or " " sep. list
datanode_addresses=
mgmnode_addresses=
What happens now?
BR
johan
Please sign in to leave a comment.