Is ClusterControl normally this slow?
Hi,
I finally got around to installing Cluster Control for evaluation onto a box (4vCPU, 8GB RAM, 80GB SAS). I added our existing 3 node PXC test cluster for evaluation. Everything is up and running, but it takes a really long time for any of the dashboards to come up once you log in. I just now timed a fresh session, and here are the steps I am performing:
1. Login
2. Click "Overview" under the cluster name.
Here is where the problem begins. When working correctly, a Cluster Load graph is displayed, a Galera Nodes table is displayed, and a Hosts table is displayed. It takes 50 seconds for these areas to be drawn in the browser (with no data - shows a Loading gif), and it takes 1m52s to have data populated into the chart and tables. I have tried this IE, FireFox, and Chrome, all with the same issue. CPU on the ClusterControl node never rises above 11%. Ram usage is about 4GB (custom my.cnf).
I attached a zip with screenshots. The 5th screenshot is the 50 second mark and the 6th is the 1m52s mark.
What is going on here? Where do I even look for information on what is going on?
Thanks for any help,
Gabe
ccissue.zip
-
Hi Gabe,
No this does not look right.
Can you login to the controller and do (as sudo/root):
free -m
vmstat 1 30
# vmstat will run for 30 seconds
top -d1
#press capital M while the top is running, copy/paste as much as you can here.
the please do, on the controller:
mysql -ucmon -p -h127.0.0.1 cmon -e "show processlist"
Please send here the output of the commands.
If you do:
apachectl restart
Does it help?
* What version of ClusterControl are you using?
* What version of MySQL do you have?
BR
johan
-
Hi,
In the output from top (top.txt) i could not see the mysql server.
Also, i could not find any httpd/apache process.
Is the cmon controller and apache and mysqld (for the cmon database) running on the same server?
Can you create a new 'top' if they are indeed running on the same server, or create the top from the server where the apache + mysqld are running.
Besides that, there is no load on the server, and swap isn't used.
Can you do like this:
When the UI is rendering the page (and nothing really happens), could you do,
mysql -ucmon -p -h127.0.0.1 cmon -e "show full processlist; select sleep(1); show full processlist; select sleep(1); show full processlist"
If nothing there, i am thinking network issues in the area of DNS, Firewall, dropped packages.
Run on the controller (or where the apache is running):
time curl "http://127.0.0.1/cmonapi/nodes/all.json?clusterid=1"
time curl "http://127.0.0.1/cmonapi/clusters/dbload.json?clusterid=1"
What do you get? (if you get a lot of data back you can discard that, i am not interested in the actual data returned, more that it works).
Best regards
Johan
-
Johan,
I've attached a complete top output (using batch mode) and also the mysql process list while the UI is stuck. I ran your two curl commands, the first completed in .162s and the second in .154s.
Is it possible that its an issue with the nodes being added with dns names? Is it possible to reconfigure it with IP addresses for the hosts, or should I just drop the whole cluster and re-add?
Gabe
-
Hi Gabe,
Did you configure PTR record for all IP addresses in the cluster (ClusterControl and PXC nodes)? You can verify with following commands:
dig ptr <ip address of CC or PXC nodes>
Ensure you get an "ANSWER SECTION" or just use the simple host command:
host <IP address of CC or PXC nodes>
Ensure you don't get a "not found: 3(NXDOMAIN)"
If no, then you may need to use IP address instead. From the ClusterControl UI > Database Clusters > just press the Delete > tick Completely remove it from Controller. Then, add following line into /etc/my.cnf:
skip-name-resolve
Restart MySQL and then re-add again the cluster using IP address from ClusterControl UI.
Regards,
Ashraf
-
Hi Gabe / all,
Did you ever figure this out ? I've just installed CC with a 3 node cluster, all ssh keys and connectivity working fine. Then go to the UI and add the cluster, and the job runs fine. Then the UI just hangs with "Loading".
I notice a lot of tables like mysql_server and node_state are all empty.
The cluster is called default_name()
And then there is CONTROLLER: and GALERA NODES: which I'd expect to have a 1 and 3 after, but it's blanks.I thought CC would of detected and added the nodes opposed to me having to add them all individually ?
-
Hi Jeremy,
I suggest you to create a support ticket and provide us with error report as described in this page: http://support.severalnines.com/entries/21091498-Error-reporting-
Regards,
Ashraf
Please sign in to leave a comment.
Comments
12 comments