Failed to migrate VMs on shared storage(moosefs)

Community Support
1

I am using moosefs + opennebula to construct a hci system. here is my setup:

  • 1*moosefs-master, 192.168.250.1
  • 3*moosefs-chunkserver+one-kvm-node(192.168.250.4-6)
  • 1*one-frontend VM(192.168.250.100)
onedatastores list
 117 mfs-img-qcow2       5.5T 99%   0                 0 img  fs      qcow2   on
 118 mfs-sys-qcow2       5.5T 99%   0                 0 sys  -       qcow2   on
 119 mfs-img-share       5.5T 99%   0                 1 img  fs      shared  on
 120 mfs-sys-share       5.5T 99%   0                 0 sys  -       shared  on

mount /mnt/mfs/one
su - oneadmin
ln -s /mnt/mfs/one/datastores/mfs-img-qcow2 /var/lib/one/datastores/117
ln -s /mnt/mfs/one/datastores/mfs-sys-qcow2 /var/lib/one/datastores/118
ln -s /mnt/mfs/one/datastores/mfs-img-shared /var/lib/one/datastores/119
ln -s /mnt/mfs/one/datastores/mfs-sys-shared /var/lib/one/datastores/120

ll /var/lib/one/datastores
lrwxrwxrwx  1 oneadmin oneadmin  37 Feb 13 16:22 117 -> /mnt/mfs/one/datastores/mfs-img-qcow2
lrwxrwxrwx  1 oneadmin oneadmin  37 Feb 13 16:22 118 -> /mnt/mfs/one/datastores/mfs-sys-qcow2
lrwxrwxrwx  1 oneadmin oneadmin  38 Feb 13 16:22 119 -> /mnt/mfs/one/datastores/mfs-img-shared
lrwxrwxrwx  1 oneadmin oneadmin  38 Feb 13 16:26 120 -> /mnt/mfs/one/datastores/mfs-sys-shared

onevm list
    ID USER     GROUP    NAME            STAT UCPU    UMEM HOST             TIME
    61 oneadmin oneadmin nfs-q-d9-test1  runn  0.0   1024M node-5       0d 06h47
    65 oneadmin oneadmin mfs-s-d9-test1  runn  0.0   1024M node-4       0d 00h54

the vm is running well on the moosefs system datastore.

there is one problem I can’t fixed. that is the migrate live.

onevm migrate mfs-q-d9-test1 node-6 --live

It failed, the virsh complaint it’s unsafe to live migrate without share storage

tail -f /var/log/one/65.log
...
 18 Wed Feb 13 17:02:46 2019 [Z0][VMM][E]: migrate: Command "virsh --connect qemu+tcp://localhost/system migrate --live  on    e-65 qemu+ssh://node-6/system" failed: error: Unsafe migration: Migration without shared storage is unsafe
...

any idea how to fixed it?
it seem likely related to virsh.

2

I found out that it is a bug of libvirt/4.5(I am running qemu-kvm-ev 2.12), opennebula utilize virsh to migrate vm. before migration, libvirt first detect if the vm is running under shared storage, but moosefs/lizardfs was mounted with fuse, so It’s not easy to identify the type of fs(shared or non-shared).

a workaround is add “–unsafe” option to virsh command. but finally it should be fixed on libvirt.

/var/lib/one/remotes/etc/vmm/kvm/kvmrc
MIGRATE_OPTIONS=--unsafe

and

onehost sync --force

I am not a developer, have not idea how to report bug.any one might kindly to helping report this bug to redhat?

here is the bug report of redhat/ovirt
https://bugzilla.redhat.com/show_bug.cgi?id=1632711
https://bugzilla.redhat.com/show_bug.cgi?id=1644679

3

Hi, Thanks for the update. There is a note in the documentation about this issue:

http://docs.opennebula.org/5.6/deployment/open_cloud_host_setup/kvm_driver.html#live-migration-for-other-cache-settings

4

hmm, this is another issue rather than cache, others might not dig out the /var/log/one/$vm-id.log. so to solve it with libvirt should be the ultimate solution.