Fail to launch VM

i’m trying to launch a tty-linux VM on a host but it get fail status even tho i can do a passwordless shh between the frontend and each of the nodes without problems.

this is a screenshot of the longs

Hello @abdul-almusaileem

Please, check that you can scp passwordless too, sometimes when you scp it’s waiting for you to write yes. So first, check that you can scp a file to /tmp in the host and then use the command onevm recover --retry to retry the last failing action.

hello @ahuertas i can do a passwordless scp without any issues


also tired the onevm recover --retry and still got the same error as it fails to launch

I’m also having the same problem.

I can ssh both ways.

I can create a host and vm on the local machine, but I can’t remotely.

New state is ACTIVE
Fri Oct 25 11:16:39 2019 [Z0][VM][I]: New LCM state is PROLOG
Fri Oct 25 11:16:41 2019 [Z0][TM][I]: Command execution failed (exit code: 1): /var/lib/one/remotes/tm/qcow2/clone ubuntu-System-Product-Name:/var/lib/one//datastores/1/8bd90de347bdf3460f17002124515e59 10.108.83.49:/var/lib/one//datastores/0/1/disk.0 1 1
Fri Oct 25 11:16:41 2019 [Z0][TM][I]: clone: Cloning /var/lib/one/datastores/1/8bd90de347bdf3460f17002124515e59 in 10.108.83.49:/var/lib/one//datastores/0/1/disk.0
Fri Oct 25 11:16:41 2019 [Z0][TM][E]: clone: Command “set -e -o pipefail
Fri Oct 25 11:16:41 2019 [Z0][TM][I]:
Fri Oct 25 11:16:41 2019 [Z0][TM][I]: cd /var/lib/one/datastores/0/1
Fri Oct 25 11:16:41 2019 [Z0][TM][I]:
Fri Oct 25 11:16:41 2019 [Z0][TM][I]: rm -rf “/var/lib/one/datastores/0/1/disk.0.snap”
Fri Oct 25 11:16:41 2019 [Z0][TM][I]:
Fri Oct 25 11:16:41 2019 [Z0][TM][I]: mkdir -p “/var/lib/one/datastores/0/1/disk.0.snap”
Fri Oct 25 11:16:41 2019 [Z0][TM][I]:
Fri Oct 25 11:16:41 2019 [Z0][TM][I]: B_FORMAT=$(qemu-img info /var/lib/one/datastores/1/8bd90de347bdf3460f17002124515e59 | grep “^file format:” | awk '{print 3}' || :) Fri Oct 25 11:16:41 2019 [Z0][TM][I]: qemu-img create -o backing_fmt={B_FORMAT:-raw} -b /var/lib/one/datastores/1/8bd90de347bdf3460f17002124515e59 -f qcow2 /var/lib/one/datastores/0/1/disk.0.snap/0
Fri Oct 25 11:16:41 2019 [Z0][TM][I]:
Fri Oct 25 11:16:41 2019 [Z0][TM][I]: rm -f “/var/lib/one/datastores/0/1/disk.0”
Fri Oct 25 11:16:41 2019 [Z0][TM][I]:
Fri Oct 25 11:16:41 2019 [Z0][TM][I]: ln -s disk.0.snap/0 /var/lib/one/datastores/0/1/disk.0
Fri Oct 25 11:16:41 2019 [Z0][TM][I]:
Fri Oct 25 11:16:41 2019 [Z0][TM][I]: cd /var/lib/one/datastores/0/1/disk.0.snap
Fri Oct 25 11:16:41 2019 [Z0][TM][I]:
Fri Oct 25 11:16:41 2019 [Z0][TM][I]: ln -s . /var/lib/one/datastores/0/1/disk.0.snap/disk.0.snap” failed: qemu-img: Could not open ‘/var/lib/one/datastores/1/8bd90de347bdf3460f17002124515e59’: Could not open ‘/var/lib/one/datastores/1/8bd90de347bdf3460f17002124515e59’: No such file or directory
Fri Oct 25 11:16:41 2019 [Z0][TM][I]: qemu-img: /var/lib/one/datastores/0/1/disk.0.snap/0: Could not open ‘/var/lib/one/datastores/1/8bd90de347bdf3460f17002124515e59’: No such file or directory
Fri Oct 25 11:16:41 2019 [Z0][TM][I]: Could not open backing image to determine size.
Fri Oct 25 11:16:41 2019 [Z0][TM][E]: Error copying ubuntu-System-Product-Name:/var/lib/one//datastores/1/8bd90de347bdf3460f17002124515e59 to 10.108.83.49:/var/lib/one//datastores/0/1/disk.0
Fri Oct 25 11:16:41 2019 [Z0][TM][E]: Error executing image transfer script: Error copying ubuntu-System-Product-Name:/var/lib/one//datastores/1/8bd90de347bdf3460f17002124515e59 to 10.108.83.49:/var/lib/one//datastores/0/1/disk.0
Fri Oct 25 11:16:41 2019 [Z0][VM][I]: New LCM state is PROLOG_FAILURE

Hello @abdul-almusaileem

The error message is in host 10.109.19.204 try to scp to that host please.

Hello @bblouin

Please check that oneadmin can read and write in all datastores:

ln -s . /var/lib/one/datastores/0/1/disk.0.snap/disk.0.snap” failed: qemu-img: Could not open ‘/var/lib/one/datastores/1/8bd90de347bdf3460f17002124515e59’: Could not open ‘/var/lib/one/datastores/1/8bd90de347bdf3460f17002124515e59’: No such file or directory

It seems that the directory could not be read by oneadmin.

hello @ahuertas i can scp files between the two machines passwordless and without any issues but i still get the same error every time i try to launch an instance

Execute the command which is failing manually and send me the output.

Make sure that you made at least once a SSH session from host A to B and one session back. And this with root and with oneadmin. So in total you need to accept the key of the other host 4 times. One of them is missing and because of this you are getting this error message:

Host key verification failed

i was able to run the command from the logs without any issue

but when i do the command that is beneath that i get this error:

The correct command is the one with full data, the second one is wrong because you are not passing the ids that the scripts needs.

the first command gave no errors when i ran it but still fails when i try to launch an instance

Hey all,

i had a similar issue regarding my first deployment. My specific problem was that i made the keyscan against the IP of the frontend not the hostname. This seems to make a difference because OpenNebula internally uses the hostname (not the IP) for the SCP command as you can see in the log.

In my case the hostname of my OpenNebula frontend is “opennebula”. So i did the following on my frontend:

ssh-keyscan opennebula >> /var/lib/one/.ssh/known_hosts

And copied the new file to my only node:

scp -rp /var/lib/one/.ssh 192.168.0.39:/var/lib/one/

This fixed it for me. Maybe this works for you aswell :slight_smile:

1 Like

Hello thnx for the response, this actually fixed my problem!

Hey @abdul-almusaileem,

i‘m glad that helped you :slight_smile:

Hello all,

I am having the same issue as well. The solution above that worked for Abdul didn’t work for me. My log is shown below. I can also scp passwordless and the host is already connected and shown on Sunstone.

oned.log:
Fri Nov 8 02:36:51 2019 [Z0][InM][D]: Host 10.0.0.188 (0) successfully monitored.
Fri Nov 8 02:36:51 2019 [Z0][VMM][D]: Message received: LOG I 19 Successfully execute transfer manager driver operation: tm_context.

Fri Nov 8 02:36:51 2019 [Z0][VMM][D]: Message received: LOG I 19 Successfully execute network driver operation: pre.

Fri Nov 8 02:36:52 2019 [Z0][VMM][D]: Message received: LOG I 19 Command execution fail: cat << EOT | /var/tmp/one/vmm/kvm/deploy ‘/var/lib/one//datastores/0/19/deployment.0’ ‘10.0.0.188’ 19 10.0.0.188

Fri Nov 8 02:36:52 2019 [Z0][VMM][D]: Message received: LOG I 19 error: Failed to create domain from /var/lib/one//datastores/0/19/deployment.0

Fri Nov 8 02:36:52 2019 [Z0][VMM][D]: Message received: LOG I 19 error: invalid argument: could not find capabilities for arch=x86_64 domaintype=kvm

Fri Nov 8 02:36:52 2019 [Z0][VMM][D]: Message received: LOG E 19 Could not create domain from /var/lib/one//datastores/0/19/deployment.0

Fri Nov 8 02:36:52 2019 [Z0][VMM][D]: Message received: LOG I 19 ExitCode: 255

Fri Nov 8 02:36:52 2019 [Z0][VMM][D]: Message received: LOG I 19 Failed to execute virtualization driver operation: deploy.

Fri Nov 8 02:36:52 2019 [Z0][VMM][D]: Message received: DEPLOY FAILURE 19 Could not create domain from /var/lib/one//datastores/0/19/deployment.0

I too getting the same error.

Thu Dec 19 20:09:49 2019 [Z0][VM][I]: New state is ACTIVE
Thu Dec 19 20:09:49 2019 [Z0][VM][I]: New LCM state is PROLOG
Thu Dec 19 20:09:51 2019 [Z0][VM][I]: New LCM state is BOOT
Thu Dec 19 20:09:51 2019 [Z0][VMM][I]: Generating deployment file: /var/lib/one/vms/21/deployment.0
Thu Dec 19 20:09:51 2019 [Z0][VMM][I]: Successfully execute transfer manager driver operation: tm_context.
Thu Dec 19 20:09:51 2019 [Z0][VMM][I]: Successfully execute network driver operation: pre.
Thu Dec 19 20:09:52 2019 [Z0][VMM][I]: Command execution fail: cat << EOT | /var/tmp/one/vmm/kvm/deploy ‘/var/lib/one//datastores/0/21/deployment.0’ ‘192.168.0.105’ 21 192.168.0.105
Thu Dec 19 20:09:52 2019 [Z0][VMM][I]: error: Failed to create domain from /var/lib/one//datastores/0/21/deployment.0
Thu Dec 19 20:09:52 2019 [Z0][VMM][I]: error: invalid argument: could not find capabilities for arch=x86_64 domaintype=kvm
Thu Dec 19 20:09:52 2019 [Z0][VMM][E]: Could not create domain from /var/lib/one//datastores/0/21/deployment.0
Thu Dec 19 20:09:52 2019 [Z0][VMM][I]: ExitCode: 255
Thu Dec 19 20:09:52 2019 [Z0][VMM][I]: Failed to execute virtualization driver operation: deploy.
Thu Dec 19 20:09:52 2019 [Z0][VMM][E]: Error deploying virtual machine: Could not create domain from /var/lib/one//datastores/0/21/deployment.0
Thu Dec 19 20:09:52 2019 [Z0][VM][I]: New LCM state is BOOT_FAILURE

Hi @gbenner and @Kanagaraj_G

Thu Dec 19 20:09:52 2019 [Z0][VMM][I]: error: invalid argument: could not find capabilities for arch=x86_64 domaintype=kvm

The issue here is the host not being able to use KVM acceleration. You need to either enable full virtualization, use LXD or get a new host.

@dann1 Add new host you meant new system with new hardware?? and how to configure with LXD and reference?

@dann1 Add new host you meant new system with new hardware??

Yes, something with full virtualization. For LXD read http://docs.opennebula.org/5.10/deployment/node_installation/lxd_node_installation.html