cmon cron job too slow !
Hi,
Coming with Cluster Control Monitor is a cron job setup under /etc/cron.d/cmon, this one is defaultly ran each 5 minutes.
After some problems occurs on the machine hosting Cluster Control Monitor, I've finally inspected this script and put some debug timing to see how it performs and saw that it takes sometimes more than 5 minutes to execute on a i3 2120 / 4 cores CPU !!? It takes an average of 4 to 4.30 minutes to ran, sometimes more.
This is a big problem as there is no lock mechanism on this script and so each run every 5 mins ran against the other etc ... etc ... making the machine almost unusable after an amount of time.
I tried to put the cron jobs every 10 mins rather than 5 mins with an output to a log file to check :
*/10 * * * * root test -x $CMON_BINDIR/cmon_rrd_all && $CMON_BINDIR/cmon_rrd_all $CMON_BINDIR > /var/log/cmon_cron.log 2>&1
then the script always finish before the next execution but it leads to another problem : graphs are not processes correctly because it seems to be hardened with a 5 minutes processing ...
What can we do here please ? Is it a normal behaviour or do I have to fix some things on my config to see this script ran faster ?
Thks.
BR,
Laurent
-
Hi Johan,
Thks for your answer, I did the modifications and logging the output to a logfile instead of /dev/null in cron but I now have a lot of error messages :
ERROR: /var/lib/cmon//cluster_1_stats.rrd: expected 9 data source readings (got 1) from N
ERROR: /var/lib/cmon//cluster_1_mysql_asterix|3306_stats.rrd: expected 340 data source readings (got 1) from N
/usr/bin/
ERROR: /var/lib/cmon//cluster_1_stats.rrd: expected 9 data source readings (got 1) from N
ERROR: /var/lib/cmon//cluster_1_mysql_asterix|3306_stats.rrd: expected 340 data source readings (got 1) from N
/usr/bin//cmon_create_graphs: 35: source: not found
/usr/bin/
imagedir not found - set imagedir in cmon.cnf
What is the value that should be set for imagedir in cmon.cnf please ? Which directory should be aimed ?
Why the path /usr/bin is displayed ?
What's the problem with two RRD files which are reporting errors ?
Thks !
Laurent -
Hi Johan,
I've finally tried to remove all nodes and add them back to see if it changes.Below are results of each command :
root@vmdebian3:/temp# bash /usr/bin/cmon_rrd_all
/usr/bin/
ERROR: /var/lib/cmon//cluster_1_stats.rrd: expected 9 data source readings (got 1) from N
root@vmdebian3:/temp#-----------------------
root@vmdebian3:/temp# bash /usr/bin/cmon_create_rrd
root@vmdebian3:/temp#
-----------------------root@vmdebian3:/temp# bash /usr/bin/cmon_update_rrd
ERROR: /var/lib/cmon//cluster_1_stats.rrd: expected 9 data source readings (got 1) from N
root@vmdebian3:/temp#-----------------------
root@vmdebian3:/temp# bash /usr/bin/cmon_create_graphs
/etc/cmon.cnf
no mysql_counters found - create_graph bailing out
root@vmdebian3:/temp#-----------------------
Curiously when ran from commandline, I do not see the output saying "imagedir not found - set imagedir in cmon.cnf", so I've finally divided the output of cron in two logfiles :*/5 * * * * root test -x $CMON_BINDIR/cmon_rrd_all && $CMON_BINDIR/cmon_rrd_all $CMON_BINDIR >> /var/log/cmon_cron_rrd_all.log 2>&1
*/10 * * * * root test -x $CMON_BINDIR/cmon_create_graphs && $CMON_BINDIR/cmon_create_graphs >> /var/log/cmon_cron_create_graphs.log 2>&1
So it allows me to identify that this is the cmon_create_graphs that is outputting the imagedir msg ...
By the way, i've checked my /etc/cmon.cnf and I already have the imagedir set :
###### RRD SECTION - graphs ######
mysql_bindir=/usr/local//mysql/bin//
imagedir=/var/www//cmon/graphs
rrdtool=/usr/bin/rrdtool
rrd_datadir=/var/lib/cmon/I'm using Debian 6.0.3.
BR,
Laurent -
Sorry for the late reply but you have to:
Change:
*/10 * * * * root test -x $CMON_BINDIR/cmon_create_graphs && $CMON_BINDIR/cmon_create_graphs >> /var/log/cmon_cron_create_graphs.log 2>&1
to
*/10 * * * * root test -x $CMON_BINDIR/cmon_create_graphs && $CMON_BINDIR/cmon_create_graphs /etc/cmon.cnf >> /var/log/cmon_cron_create_graphs.log 2>&1
Best regards,
Johan
Please sign in to leave a comment.
Comments
5 comments