System failed to initiate a job command. You may want to try again.
when i try to add the existing mysql server or cluster with cluster controller. message box pop ups like "System failed to initiate a job command. You may want to try again." So please give me the right solution for this issue. I followed the installation steps from this link - http://support.severalnines.com/entries/20613923-Installation-on-an-existing-Cluster
Environment name is Ubuntu 14.10, RAM size - 4 GB.
clustercontroller error.png
-
Hi I am also getting same error described above , has some body have found any solution for it?
I also checked httpd logs where i don't find any error in error.log and only one warning in ssl_error_log file . Does below warning causing this issue ? if yes what is the solution for it.
tail -f ssl_error_log -n 1000
[Mon Jul 06 07:23:16 2015] [warn] RSA server certificate wildcard CommonName (CN) `*.severalnines.local' does NOT match server name!?Thanks is advance!!!
-
Hi Dinesh,
The error message showing up is when the UI is not able to push the job request to the REST API. Can you verify if you can access to http://[ClusterControl_host]/cmonapi ?
Also, what is the output if you run following command on ClusterControl:
mysql -uroot -p -e 'select url from dcps.apis'
Regards,
Ashraf
-
Hi Thanks, you very much for your quick response.
What is the job request api uri? Can I test them independently?
Yes I could able to access http://<my clustercontrol Host>/cmonapi.
I am getting below results
When I execute the "select url from dcps.apis" query .
http://127.0.0.1/cmonapi
Hi One more additional information that when I tried to login on cmonapi , I got below registration error.
Many Thanks
Dinesh Ghutake
-
hi I have made some progress on it. what i is that I have stopped cmon service and dropped the cmon database and re executed /va/www/html/clustercontrol/app/tools/setup-cc.sh which reinstalled cmon database again..
Then I again tried to add cluster node fro Cluster control UI and now I am getting below SSH error. ANy Idea how do i resolve this?
-
Hi Dinesh,
You also need to setup passwordless SSH on ClusterControl node, so it can SSH to itself passwordlessly. Run following command on ClusterControl node as user dinesh:
su - dinesh
ssh-copy-id -i /home/dinesh/.ssh/id_rsa dinesh@10.232.24.198Then, retry again to add the cluster.
Regards,
Ashraf
-
Hi Ashraf, Thank you for your kind help and support. that worked ..Now I could be able to made progress now.
Now I am getting permission denied error while adding cluster..I am looking into this issue whether mysql root user cann't access database.
15 - Message sent to controller
16 - Verifying controller host and cmon password.
17 - Verifying the SSH access to the controller.
18 - Verifying job parameters.
19 - Verifying the SSH connection to 10.247.97.147.
20 - Verifying the MySQL user/password.
21 - Can't access the MySQL server: sh: /home/chaitanya/percona/Percona-XtraDB-Cluster-5.6.22-rel72.0-25.8.978.Linux.x86_64/bin/mysql: Permission denied, command: /home/chaitanya/percona/Percona-XtraDB-Cluster-5.6.22-rel72.0-25.8.978.Linux.x86_64/bin/mysql -NAB -u'root' -pxxxxxx -e 'SELECT 1' 2>&1
22 - Can't access the MySQL server: sh: /home/chaitanya/percona/Percona-XtraDB-Cluster-5.6.22-rel72.0-25.8.978.Linux.x86_64/bin/mysql: Permission denied, command: /home/chaitanya/percona/Percona-XtraDB-Cluster-5.6.22-rel72.0-25.8.978.Linux.x86_64/bin/mysql -NAB -u'root' -h127.0.0.1 -P3306 -pxxxxxx -e 'SELECT 1' 2>&1
23 - Can't access the MySQL server: sh: /home/chaitanya/percona/Percona-XtraDB-Cluster-5.6.22-rel72.0-25.8.978.Linux.x86_64/bin/mysql: Permission denied, command: /home/chaitanya/percona/Percona-XtraDB-Cluster-5.6.22-rel72.0-25.8.978.Linux.x86_64/bin/mysql -NAB -u'root' -h10.247.97.147 -P3306 -pxxxxxx -e 'SELECT 1' 2>&1
24 - Could not connect to mysql server.
Job failed. -
Hi Dinesh,
Does this user 'dinesh' has sudo privileges to execute super user command? It says: "Permission denied". Please verify if the basedir can be accessible by the specified user. I can see the basedir location is under other user's home directory, "/home/chaitanya" but can't confirm if the user has enough privileges.
Regards,
Ashraf
-
Hi Thirviam, Thank you for your suggestion , since i am using private cloud vms i can not have ssh with roo user. As Ashraf suggested I have created user chaitanya and i could be able made progress now.
I am able to connect both of percona cluster nodes but on second node (10.247.97.146) from Cluster control node, not I am NOT able execute the queries because cluster control by default looks for mysql client on /usr/bin/mysql and that client is not available on 146 node (cluster node installation is on my home).
On 147 node,it's working fine because I had another mysql installed already therefore I could able to create soft link there and cluster control could able execute queries on vm 147 from /usr/bin/mysql.
Question is why clustercontrol always looks for /usr/bin/mysql instead of looking for given base directory for ex. /home/chaitanya/percona/<perconamysql>/bin/mysql?
what would be the possible solution for this ?
Any help is appreciated.Thanks is advance!!
215 - Message sent to controller
216 - Verifying controller host and cmon password.
217 - Verifying the SSH access to the controller.
218 - Verifying job parameters.
219 - Verifying the SSH connection to 10.247.97.147.
220 - Verifying the MySQL user/password.
221 - monitored_mysql_root_password is not set, please set it later the generated cmon.cnf
222 - Getting node list from the MySQL server.
223 - Found node: '10.247.97.146'
224 - Found node: '10.247.97.147'
225 - Found in total 2 nodes.
226 - Checking the nodes that those aren't in another cluster.
227 - Verifying the SSH connection to the nodes.
228 - Check SELinux statuses
229 - 10.247.97.146: Failed to determine skip_name_resolve: , command: /usr/bin/mysql -NAB -u'cmon' -pxxxxxx -e "SELECT VARIABLE_VALUE FROM information_schema.GLOBAL_VARIABLES WHERE VARIABLE_NAME='skip_name_resolve'" 2>/dev/null
Job failed. -
Hi Dinesh,
On job ID 229, /usr/bin/mysql is being executed on the controller node, not the target MySQL server to determine whether skip_name_resolve is enabled. So ensure on ClusterControl node, you have following lines configured under /etc/cmon.cnf:
mysql_basedir=[path to MySQL basedir on ClusterControl server, eg: /usr ]
mysql_bindir=[path to MySQL binary directory on ClusterControl server, eg: /usr/bin ]Restart CMON and try to add the node again.
Regards,
Ashraf -
Hi Ashraf, Thanks for your response and appreciated your help, I just tried as you suggested modified /etc/cmon.cnf but facing same issue.
I guess controller node is trying execute query on one of cluster node which is vm 146 and looking for mysql client on /usr/bin/mysql ,but could not find the mysql client there.
Shall I install mysql client on 146 in /usr/bin/mysql and try it again?
/etc/cmon.cnf
mysql_basedir=/var/cluster-control/mysql
mysql_bindir=/var/cluster-control/mysql/bin240 - Message sent to controller
241 - Verifying controller host and cmon password.
242 - Verifying the SSH access to the controller.
243 - Verifying job parameters.
244 - Verifying the SSH connection to 10.247.97.147.
245 - Verifying the MySQL user/password.
246 - monitored_mysql_root_password is not set, please set it later the generated cmon.cnf
247 - Getting node list from the MySQL server.
248 - Found node: '10.247.97.146'
249 - Found node: '10.247.97.147'
250 - Found in total 2 nodes.
251 - Checking the nodes that those aren't in another cluster.
252 - Verifying the SSH connection to the nodes.
253 - Check SELinux statuses
254 - 10.247.97.146: Failed to determine skip_name_resolve: , command: /usr/bin/mysql -NAB -u'cmon' -pxxxxxx -e "SELECT VARIABLE_VALUE FROM information_schema.GLOBAL_VARIABLES WHERE VARIABLE_NAME='skip_name_resolve'" 2>/dev/null
Job failed. -
Hi Dinesh,
I couldn't reproduce the problem with the latest version:
clustercontrol-controller-1.2.10-822.x86_64
clustercontrol-cmonapi-1.2.10-99.x86_64
clustercontrol-1.2.10-518.x86_64Here is my cmon.cnf (I used MariaDB 10.x client):
mysql_basedir=/root/mariadb-10.0.20-linux-x86_64/
mysql_bindir=/root/mariadb-10.0.20-linux-x86_64/binAnd the target MySQL server's basedir:
/usr/local/Percona-Server-5.6.25-rel73.1-Linux.x86_64.ssl101/If you are running on earlier version, please upgrade to at least the version I mentioned by following these steps: http://support.severalnines.com/entries/21095371
Also, examine /var/log/cmon.log and look for "ERROR" log level if it has.
Regards,
Ashraf
-
Hi Ashraf,
I could able to resolve above issue by making soft link of mysql.sock file with /var/lib/mysql/mysql.sock.
I think, I am pretty close now , getting access denied for 'cmon' user on VM 146.
Just for info while adding cluster i am using cmon as database user.
on both VM147 and VM 147 and on cluster control node i have granted cmon user which looks same all these VMs.
Cluster Control node-->
mysql> show grants for 'cmon';
+--------------------------------------------------------------------------------------------------------------------------------+
| Grants for cmon@% |
+--------------------------------------------------------------------------------------------------------------------------------+
| GRANT ALL PRIVILEGES ON *.* TO 'cmon'@'%' IDENTIFIED BY PASSWORD '*E8C5459B50EF1C73187CBEFB6D0FAF5C0F4E0812' WITH GRANT OPTION |
+--------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)VM 146-->
mysql> show grants for 'cmon';
+--------------------------------------------------------------------------------------------------------------------------------+
| Grants for cmon@% |
+--------------------------------------------------------------------------------------------------------------------------------+
| GRANT ALL PRIVILEGES ON *.* TO 'cmon'@'%' IDENTIFIED BY PASSWORD '*E8C5459B50EF1C73187CBEFB6D0FAF5C0F4E0812' WITH GRANT OPTION |
+--------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)VM 147 -->
mysql> show grants for 'cmon';
+--------------------------------------------------------------------------------------------------------------------------------+
| Grants for cmon@% |
+--------------------------------------------------------------------------------------------------------------------------------+
| GRANT ALL PRIVILEGES ON *.* TO 'cmon'@'%' IDENTIFIED BY PASSWORD '*E8C5459B50EF1C73187CBEFB6D0FAF5C0F4E0812' WITH GRANT OPTION |
+--------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.02 sec)456 - Message sent to controller
457 - Verifying controller host and cmon password.
458 - Verifying the SSH access to the controller.
459 - Verifying job parameters.
460 - Verifying the SSH connection to 10.247.97.147.
461 - Verifying the MySQL user/password.
462 - monitored_mysql_root_password is not set, please set it later the generated cmon.cnf
463 - Getting node list from the MySQL server.
464 - Found node: '10.247.97.146'
465 - Found node: '10.247.97.147'
466 - Found in total 2 nodes.
467 - Checking the nodes that those aren't in another cluster.
468 - Verifying the SSH connection to the nodes.
469 - Check SELinux statuses
470 - Detected that skip_name_resolve is not used on the target server(s).
471 - Granting the controller on the cluster.
472 - Node is Synced : 10.247.97.146
473 - 10.247.97.146: GRANT failed: Warning: Using a password on the command line interface can be insecure. ERROR 1045 (28000) at line 1: Access denied for user 'cmon'@'localhost' (using password: YES)
Job failed.Any Idea? Thanks in Advance!!
-
Hi Dinesh,
It clearly says:
ERROR 1045 (28000) at line 1: Access denied for user 'cmon'@'localhost' (using password: YES)Can you try with this command on 146?
mysql -ucmon -p -hlocalhost
Also please show the output of following statement on 146:
SHOW GRANTS FOR 'cmon'@'localhost';Regards,
Ashraf
-
Hi Dinesh,
Does the 'root' and 'cmon' passwords contain non-alphanumeric passwords?
Did you manually install Percona from tar.gz files or something else?
What cloud provider are you using?
I would like to setup a similar environment to reproduce your problems , b/c the installation process should be much smother than this and it seems the error messages are not clear enough.
BR
johan -
Hi Ashraf/John,
To answers Ashraf''s queries here is the out put-->
[chaitanya@IPPGITVERCTRL01 Percona-XtraDB-Cluster-5.6.22-rel72.0-25.8.978.Linux.x86_64]$ ./bin/mysql -ucmon -p -hlocalhost
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 1302
Server version: 5.6.22-72.0-25.8-log Percona XtraDB Cluster binary (GPL) 5.6.22-25.8, Revision 978, wsrep_25.8.r4150Copyright (c) 2000, 2015, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> SHOW GRANTS FOR 'cmon'@'localhost';
+----------------------------------------------------------------------------------------------------------------------+
| Grants for cmon@localhost |
+----------------------------------------------------------------------------------------------------------------------+
| GRANT ALL PRIVILEGES ON *.* TO 'cmon'@'localhost' IDENTIFIED BY PASSWORD '*E8C5459B50EF1C73187CBEFB6D0FAF5C0F4E0812' |
+----------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)mysql>
It seems I am missing some step here, but could not get it..
HI John,
'root' and 'cmon' passwords contains only alphabet values.
Yes we have used tar.gz to manually install Percona DB cluster since Yum repo was able on our VMs (redhat 5.7) and there were already other instance of mysql present.
Our cluster is working fine we have been executing perfomance lab on it from last 1 month.
It's our internal company private cloud.
I agree installation process was smoother for cluster control , I am getting these issues while adding existing cluster nodes on cluster control.
Thank you very much for you help.
-
/etc/cmon.cnf files on cluster controller node looks like this -->
mysql_port=3306
mysql_hostname=10.232.24.198
mysql_password=*****
hostname=10.232.24.198
mysql_basedir=/var/cluster-control/mysql
mysql_bindir=/var/cluster-control/mysql/bin
skip_name_resolve=1
mode=controller
type=galera
local_mysql_port=3306
local_mysql_password=******
os_user=root
logfile=/var/log/cmon.log
mysql_server_addresses=10.247.97.147,10.247.97.146Your help is much appreciated.
Regards,
Dinesh
Please sign in to leave a comment.
Comments
19 comments