Ceph creating swap disk

We are receiving the following when creating a swap partition.

Sun Apr 2 11:22:27 2017 [Z0][VM][I]: New state is ACTIVE
Sun Apr 2 11:22:27 2017 [Z0][VM][I]: New LCM state is PROLOG
Sun Apr 2 11:22:29 2017 [Z0][TM][I]: Command execution fail: /var/lib/one/remotes/tm/ceph/mkswap 4000 koko-kvm1:/var/lib/one//datastores/100/179/disk.1 179 100
Sun Apr 2 11:22:29 2017 [Z0][TM][I]: mkimage: Making volatile disk of 4000M at koko-kvm1:/var/lib/one//datastores/100/179/disk.1
Sun Apr 2 11:22:29 2017 [Z0][TM][E]: mkimage: Command " export PATH=/usr/sbin:/sbin:$PATH
Sun Apr 2 11:22:29 2017 [Z0][TM][I]: rbd --id libvirt create --image-format 2 one/one-sys-179-1 --size 4000 || exit $?
Sun Apr 2 11:22:29 2017 [Z0][TM][I]: 
Sun Apr 2 11:22:29 2017 [Z0][TM][I]: if [ "swap" = "swap" ]; then
Sun Apr 2 11:22:29 2017 [Z0][TM][I]: sudo rbd --id libvirt map one/one-sys-179-1 || exit $?
Sun Apr 2 11:22:29 2017 [Z0][TM][I]: sudo mkswap -L swap /dev/rbd/one/one-sys-179-1
Sun Apr 2 11:22:29 2017 [Z0][TM][I]: sudo rbd --id libvirt unmap /dev/rbd/one/one-sys-179-1
Sun Apr 2 11:22:29 2017 [Z0][TM][I]: fi" failed: rbd: sysfs write failed
Sun Apr 2 11:22:29 2017 [Z0][TM][I]: rbd: map failed: (6) No such device or address
Sun Apr 2 11:22:29 2017 [Z0][TM][E]: Error creating volatile disk.1 (one/one-sys-179-1) in koko-kvm1 into pool one.
Sun Apr 2 11:22:29 2017 [Z0][TM][I]: ExitCode: 6
Sun Apr 2 11:22:29 2017 [Z0][TM][E]: Error executing image transfer script: Error creating volatile disk.1 (one/one-sys-179-1) in koko-kvm1 into pool one.
Sun Apr 2 11:22:29 2017 [Z0][VM][I]: New LCM state is PROLOG_FAILURE

When we attempt the commands by hand on the kvm server we receive the following error. We would happily write the patch needed to correct the issue but we are not able to determine the exact features to disable and then re-enable. We are failely sure the issue is caused because the kernel driver doesn’t support the v2 features opennebula enables. (We don’t have the issue when we create the disk in v1 mode.

root@koko-kvm1:~# rbd --id libvirt map one/one-sys-180-1 
Apr  2 11:36:26 koko-kvm1 kernel: [1181938.366562] libceph: client9911700 fsid 8f6ba9d6-314d-4725-bcfa-340e500697f0
Apr  2 11:36:26 koko-kvm1 kernel: [1181938.369739] libceph: mon1 10.141.160.251:6789 session established
rbd: sysfs write failed
RBD image feature set mismatch. You can disable features unsupported by the kernel with "rbd feature disable".
In some cases useful info is found in syslog - try "dmesg | tail" or so.
rbd: map failed: (6) No such device or address

We found another post that matches this and created a diff for the file

root@koko-kvm2:/var/tmp/one/tm/ceph# diff mkimage.original mkimage
100c100
< $RBD create $FORMAT_OPT $RBD_SOURCE --size ${SIZE} || exit $?

$RBD create $FORMAT_OPT $RBD_SOURCE --size ${SIZE} --image-feature layering || exit \$?

This resolves the issue creating the swap disk but doesn’t resolve all the issues. The previously working disk disk1 now does not provision and it is a silent faiure witht no errors related to the disk failing to be made. We will investigate on our side but was hoping some one else has seen this issue.

Template:
CONTEXT = [
NETWORK = “YES”,
SSH_PUBLIC_KEY = “$USER[SSH_PUBLIC_KEY]” ]
CPU = “1”
DISK = [
IMAGE = “sl7.3-20170401.v1”,
IMAGE_UNAME = “rresnick@fau.edu” ]
DISK = [
SIZE = “4000”,
TYPE = “swap” ]
GRAPHICS = [
LISTEN = “0.0.0.0”,
TYPE = “VNC” ]
MEMORY = “4096”
NIC = [
NETWORK = “HPC Management 10.85.8.0”,
NETWORK_UNAME = “rresnick@fau.edu” ]
NIC_DEFAULT = [
MODEL = “virtio” ]
VCPU = “4”

Log:

Sun Apr 2 12:24:57 2017 [Z0][VM][I]: New state is ACTIVE
Sun Apr 2 12:24:57 2017 [Z0][VM][I]: New LCM state is PROLOG
Sun Apr 2 12:24:58 2017 [Z0][VM][I]: New LCM state is BOOT
Sun Apr 2 12:24:58 2017 [Z0][VMM][I]: Generating deployment file: /var/lib/one/vms/190/deployment.0
Sun Apr 2 12:24:59 2017 [Z0][VMM][I]: Successfully execute transfer manager driver operation: tm_context.
Sun Apr 2 12:24:59 2017 [Z0][VMM][I]: ExitCode: 0
Sun Apr 2 12:24:59 2017 [Z0][VMM][I]: Successfully execute network driver operation: pre.
Sun Apr 2 12:25:00 2017 [Z0][VMM][I]: Command execution fail: cat << EOT | /var/tmp/one/vmm/kvm/deploy '/var/lib/one//datastores/100/190/deployment.0' 'koko-kvm1' 190 koko-kvm1
Sun Apr 2 12:25:00 2017 [Z0][VMM][I]: error: Failed to create domain from /var/lib/one//datastores/100/190/deployment.0
Sun Apr 2 12:25:00 2017 [Z0][VMM][I]: error: Cannot access storage file '/var/lib/one//datastores/100/190/disk.1' (as uid:111, gid:117): No such file or directory
Sun Apr 2 12:25:00 2017 [Z0][VMM][E]: Could not create domain from /var/lib/one//datastores/100/190/deployment.0
Sun Apr 2 12:25:00 2017 [Z0][VMM][I]: ExitCode: 255
Sun Apr 2 12:25:00 2017 [Z0][VMM][I]: Failed to execute virtualization driver operation: deploy.
Sun Apr 2 12:25:00 2017 [Z0][VMM][E]: Error deploying virtual machine: Could not create domain from /var/lib/one//datastores/100/190/deployment.0
Sun Apr 2 12:25:00 2017 [Z0][VM][I]: New LCM state is BOOT_FAILURE

Thanks

This is a bug, it will be fixed here: https://dev.opennebula.org/issues/5116