5.10 networks don't work anymore

#1

Hello one and all.

After upgrading to 5.10 (from 5.8.5) my networks do not work anymore. I get:

Wed Nov 27 21:48:24 2019 [Z0][VM][I]: New LCM state is HOTPLUG_NIC
Wed Nov 27 21:48:25 2019 [Z0][VMM][I]: Command execution fail: cat << EOT | /var/tmp/one/vnm/fw/pre
Wed Nov 27 21:48:25 2019 [Z0][VMM][I]: /usr/lib/ruby/2.5.0/base64.rb:59:in `decode64': undefined method `unpack1' for nil:NilClass (NoMethodError)
Wed Nov 27 21:48:25 2019 [Z0][VMM][I]: from /var/tmp/one/vnm/vnm_driver.rb:49:in `from_base64'
Wed Nov 27 21:48:25 2019 [Z0][VMM][I]: from /var/tmp/one/vnm/fw/pre:28:in `<main>'
Wed Nov 27 21:48:25 2019 [Z0][VMM][I]: ExitCode: 1
Wed Nov 27 21:48:25 2019 [Z0][VMM][I]: Failed to execute network driver operation: pre.
Wed Nov 27 21:48:25 2019 [Z0][VMM][E]: Error attaching new VM NIC: fw: -
Wed Nov 27 21:48:25 2019 [Z0][VM][I]: New LCM state is RUNNING
Thu Nov 28 08:55:34 2019 [Z0][VM][I]: New LCM state is HOTPLUG_NIC
Thu Nov 28 08:55:34 2019 [Z0][VMM][I]: Command execution fail: cat << EOT | /var/tmp/one/vnm/fw/pre
Thu Nov 28 08:55:34 2019 [Z0][VMM][I]: /usr/lib/ruby/2.5.0/base64.rb:59:in `decode64': undefined method `unpack1' for nil:NilClass (NoMethodError)
Thu Nov 28 08:55:34 2019 [Z0][VMM][I]: from /var/tmp/one/vnm/vnm_driver.rb:49:in `from_base64'
Thu Nov 28 08:55:34 2019 [Z0][VMM][I]: from /var/tmp/one/vnm/fw/pre:28:in `<main>'
Thu Nov 28 08:55:34 2019 [Z0][VMM][I]: ExitCode: 1
Thu Nov 28 08:55:34 2019 [Z0][VMM][I]: Failed to execute network driver operation: pre.
Thu Nov 28 08:55:34 2019 [Z0][VMM][E]: Error attaching new VM NIC: fw: -
Thu Nov 28 08:55:34 2019 [Z0][VM][I]: New LCM state is RUNNING

Ubuntu 18.04.

Any ideas how to fix this?

cheers, t.

#2

Hi @himbeere,

It looks like you missed to sync the hosts after the upgrade?

Try with the following command to delete any leftover files too…

su - oneadmin -c 'onehost sync --force --rsync'

Hope this helps,
Anton Todorov

#3

Thanks @atodorov_storpool

#4

I am also facing the same probelm and applied the give sync hosts command. But nothing works for me. still facing the same probem.

[Z0][VMM][I]: /usr/lib/ruby/2.5.0/base64.rb:59:in decode64': undefined method unpack1’ for nil:NilClass (NoMethodError)
[Z0][VMM][I]: from /var/tmp/one/vnm/vnm_driver.rb:49:in from_base64' [Z0][VMM][I]: from /var/tmp/one/vnm/802.1Q/pre:28:in ’
[Z0][VMM][I]: ExitCode: 1
[Z0][VMM][I]: Failed to execute network driver operation: pre.
[Z0][VMM][E]: Error deploying virtual machine: 802.1Q: -
[Z0][VM][I]: New state is POWEROFF
[Z0][VM][I]: New LCM state is LCM_INIT

#5

Did you issue the onedb upgrade command after upgrading? If you didn’t then try it then restart both opennebula and opennebula-sunstone services

#6

Yes, I do upgrade the onedb. Basically I was running 5.8 and backup database from 5.8 and install to 5.10.1 using upgrade onedb. Several times I do restart the opennebula and opennebula-sunstone. But no luck. Still facing same issues. I am afraid of others VM as if network need to be reset then cant attache anymore.

#7

Did you install 5.10 from the official OpenNebula repo? Just be sure that you have all the required ruby modules installed. Before you had to run that gem_install command, now they are all available as .rpm packages (or .deb packages). I never had such issues and I keep upgrading it since version 5.4

#8

I do all the tasks recommended by http://docs.opennebula.org/5.10/intro_release_notes/upgrades/upgrade_58.html the URL. We were in big trouble after upgrading. As per my understanding, there is a change upgrade from 5.8 to 5.10 and which is previously only “onehost sync” command is sufficient to upgrade the hypervisor but this time every individual hypervisor nodes need to install the upgrade packages of “apt-get install --only-upgrade opennebula-node”. And I find this makes the problem.

Now I do rollback the system to 5.8.5 and now everthing wokring properly.

#9

Hello @atodorov_storpool
Did you run the command “apt-get install --only-upgrade opennebula-node” on every individual hypervisor nodes while upgrading from 5.8 to 5.10??

In my system " su - oneadmin -c ‘onehost sync --force --rsync’" does not solve the network issues. I made rollback the system to 5.8.5 again.

#10

Hi @mosharaflink

I am working with CentOS but yes. According the upgrade docs

I’d check that the files are indeed in sync on the hosts after issuing onehost sync --force. The error is happening when OpenNebula 5.10+ controller is communicating with old drivers on the host.
The opennebula-node-kvm install only the dependent packages needed by the drivers that the opennebula frontend (the controller) sync from frontend:/var/lib/one/remotes/ to host:/var/tmp/one/.
So if you have updated frontend but not updated /var/tmp/one/ on the hosts - you’ll have the above error. If you have /var/tmp/one/ in sync but missed to update opennebula-node-kvm the new files in /var/tmp/one could have missing package dependencies on the host.

Hope this helps.

Best Regards,
Anton Todorov