LXD Support for Centos 7 hosts

(Robert Brooks) #1

Is there a timeline for supporting Centos 7 hosts with LXD installed via Snap?

I have a LXD running on a Centos 7 nodes. Our existing hypervisors are Centos 7, so it would be preferred. I can launch LXC containers manually on the host using the ceph storage backend.

After adding some sudo permissions to the oneadmin user OpenNebula is able to create a container, but it fails to launch:

lxc one-743 20190326233247.198 ERROR dir - storage/dir.c:dir_mount:198 - No such file or directory - Failed to mount “/var/snap/lxd/common/lxd/containers/one-743/rootfs” on “/var/snap/lxd/common/lxc/”

The path exists as it was created by OpenNebula and has the expected os filesystem present via a Ceph nbd device that was mounted by OpenNebula’s scripts.

Happy to provide more information if supporting Centos is being considered, or be patient if it is something that is being worked on?

0 Likes

(Daniel Clavijo Coca) #2

Hello,

Support for Centos is not currently being worked on, but it isn’t ruled out at all. If you want you can open an issue requesting the feature so it could be upvoted.

Currently we still need to polish the snap setups, since it’s full of hacks to make it run.

Could you please post the whole VM log and the output of lsblk after the failure?

0 Likes

(Anton Todorov) #3

I’ve tried unsuccessfully LxD on CentOS 7 on first 5.8 beta with intention to test/develop support for block device backed images. At the end I’ve gave up and installed an Ubuntu VM on the CentOS7 KVM host :slight_smile: For fastest networking I’ve attached some NIC VFs to the VM.

I think you could give a try to such setup but note that the contextualization on the Ubuntu VM should be updated or at least this patch applied. :wink:

Best Regards,
Anton Todorov

0 Likes

(Robert Brooks) #4

Here’s the vm log

Fri Mar 29 09:50:41 2019 [Z0][VM][I]: New state is ACTIVE
Fri Mar 29 09:50:41 2019 [Z0][VM][I]: New LCM state is PROLOG
Fri Mar 29 09:50:42 2019 [Z0][VM][I]: New LCM state is BOOT
Fri Mar 29 09:50:42 2019 [Z0][VMM][I]: Generating deployment file: /var/lib/one/vms/751/deployment.0
Fri Mar 29 09:50:43 2019 [Z0][VMM][I]: Successfully execute transfer manager driver operation: tm_context.
Fri Mar 29 09:50:43 2019 [Z0][VMM][I]: Successfully execute network driver operation: pre.
Fri Mar 29 09:50:46 2019 [Z0][VMM][I]: Command execution fail: cat << EOT | /var/tmp/one/vmm/lxd/deploy '/var/lib/one//datastores/100/751/deployment.0' 'v45' 751 v45
Fri Mar 29 09:50:46 2019 [Z0][VMM][I]: deploy: Using rbd disk mapper for
Fri Mar 29 09:50:46 2019 [Z0][VMM][I]: deploy: Mapping disk at /var/snap/lxd/common/lxd/storage-pools/default/containers/one-751/rootfs using device /dev/nbd1
Fri Mar 29 09:50:46 2019 [Z0][VMM][I]: deploy: Mounting /dev/nbd1 at /var/snap/lxd/common/lxd/storage-pools/default/containers/one-751/rootfs
Fri Mar 29 09:50:46 2019 [Z0][VMM][I]: deploy: Mapping disk at /var/lib/one/datastores/100/751/mapper/disk.1 using device /dev/loop3
Fri Mar 29 09:50:46 2019 [Z0][VMM][I]: deploy: Mounting /dev/loop3 at /var/lib/one/datastores/100/751/mapper/disk.1
Fri Mar 29 09:50:46 2019 [Z0][VMM][I]: /var/tmp/one/vmm/lxd/client.rb:98:in `wait': {"type"=>"sync", "status"=>"Success", "status_code"=>200, "operation"=>"", "error_code"=>0, "error"=>"", "metadata"=>{"id"=>"e2762b52-4060-4c9b-82ab-bbb0e3b55848", "class"=>"task", "description"=>"Starting container", "created_at"=>"2019-03-29T09:50:46.426770303-07:00", "updated_at"=>"2019-03-29T09:50:46.426770303-07:00", "status"=>"Failure", "status_code"=>400, "resources"=>{"containers"=>["/1.0/containers/one-751"]}, "metadata"=>nil, "may_cancel"=>false, "err"=>"Failed to run: /snap/lxd/current/bin/lxd forkstart one-751 /var/snap/lxd/common/lxd/containers /var/snap/lxd/common/lxd/logs/one-751/lxc.conf: "}} (LXDError)
Fri Mar 29 09:50:46 2019 [Z0][VMM][I]: from /var/tmp/one/vmm/lxd/container.rb:397:in `wait?'
Fri Mar 29 09:50:46 2019 [Z0][VMM][I]: from /var/tmp/one/vmm/lxd/container.rb:406:in `change_state'
Fri Mar 29 09:50:46 2019 [Z0][VMM][I]: from /var/tmp/one/vmm/lxd/container.rb:183:in `start'
Fri Mar 29 09:50:46 2019 [Z0][VMM][I]: from /var/tmp/one/vmm/lxd/deploy:75:in `<main>'
Fri Mar 29 09:50:46 2019 [Z0][VMM][I]: ExitCode: 1
Fri Mar 29 09:50:46 2019 [Z0][VMM][I]: Failed to execute virtualization driver operation: deploy.
Fri Mar 29 09:50:46 2019 [Z0][VMM][E]: Error deploying virtual machine
Fri Mar 29 09:50:46 2019 [Z0][VM][I]: New LCM state is BOOT_FAILURE

lsblk:

NAME          MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
loop0           7:0    0  53.8M  1 loop /var/lib/snapd/snap/lxd/10343
loop1           7:1    0  91.1M  1 loop /var/lib/snapd/snap/core/6531
loop2           7:2    0   364K  0 loop 
loop3           7:3    0   364K  0 loop /home/oneadmin/datastores/100/751/mapper/disk.1
sda             8:0    0 558.9G  0 disk 
sdb             8:16   0 558.9G  0 disk 
sdc             8:32   0 558.9G  0 disk 
sdd             8:48   0 558.9G  0 disk 
sde             8:64   0 558.9G  0 disk 
├─sde1          8:65   0 558.9G  0 part 
└─sde9          8:73   0     8M  0 part 
sdf             8:80   0 558.9G  0 disk 
├─sdf1          8:81   0 558.9G  0 part 
└─sdf9          8:89   0     8M  0 part 
sdg             8:96   0 558.4G  0 disk 
├─sdg1          8:97   0     1M  0 part 
├─sdg2          8:98   0   200M  0 part /boot
├─sdg3          8:99   0     4G  0 part /
├─sdg4          8:100  0     1K  0 part 
└─sdg5          8:101  0 554.2G  0 part 
  ├─vg01-swap 253:0    0     4G  0 lvm  [SWAP]
  ├─vg01-usr  253:1    0    10G  0 lvm  /usr
  ├─vg01-opt  253:2    0    20G  0 lvm  /opt
  ├─vg01-home 253:3    0    10G  0 lvm  /home
  ├─vg01-tmp  253:4    0     2G  0 lvm  /tmp
  └─vg01-var  253:5    0    20G  0 lvm  /var
nbd0           43:0    0     1G  0 disk 
rbd0          252:0    0   9.3G  0 disk 
nbd1           43:256  0     1G  0 disk /var/snap/lxd/common/lxd/storage-pools/default/containers/one-751/rootfs

Log from lxc:

Name: one-751
Location: none
Remote: unix://
Architecture: x86_64
Created: 2019/03/29 16:50 UTC
Status: Stopped
Type: persistent
Profiles: default

Log:

lxc one-751 20190329165046.648 ERROR    dir - storage/dir.c:dir_mount:198 - No such file or directory - Failed to mount "/var/snap/lxd/common/lxd/containers/one-751/rootfs" on "/var/snap/lxd/common/lxc/"
lxc one-751 20190329165046.648 ERROR    conf - conf.c:lxc_mount_rootfs:1351 - Failed to mount rootfs "/var/snap/lxd/common/lxd/containers/one-751/rootfs" onto "/var/snap/lxd/common/lxc/" with options "(null)"
lxc one-751 20190329165046.648 ERROR    conf - conf.c:lxc_setup_rootfs_prepare_root:3498 - Failed to setup rootfs for
lxc one-751 20190329165046.648 ERROR    conf - conf.c:lxc_setup:3551 - Failed to setup rootfs
lxc one-751 20190329165046.648 ERROR    start - start.c:do_start:1282 - Failed to setup container "one-751"
lxc one-751 20190329165046.648 ERROR    sync - sync.c:__sync_wait:62 - An error occurred in another process (expected sequence number 5)
lxc one-751 20190329165046.649 WARN     network - network.c:lxc_delete_network_priv:2589 - Operation not permitted - Failed to remove interface "eth0" with index 30
lxc one-751 20190329165046.649 ERROR    start - start.c:__lxc_start:1975 - Failed to spawn container "one-751"
lxc one-751 20190329165046.649 ERROR    lxccontainer - lxccontainer.c:wait_on_daemonized_start:864 - Received container state "ABORTING" instead of "RUNNING"
lxc 20190329165046.654 WARN     commands - commands.c:lxc_cmd_rsp_recv:132 - Connection reset by peer - Failed to receive response for command "get_state"

The image is mounted at this location though…

[root@v45 lxd]# ls -las /var/snap/lxd/common/lxd/containers/one-751/rootfs
total 92
 4 dr-xr-xr-x 20 root root  4096 Mar 29 09:50 .
 4 drwx------  3 root root  4096 Mar 29 09:50 ..
 0 lrwxrwxrwx  1 root root     7 Mar 19 00:12 bin -> usr/bin
 4 dr-xr-xr-x  2 root root  4096 Apr 10  2018 boot
 4 drwxr-xr-x  2 root root  4096 Mar 29 09:50 context
 4 drwxr-xr-x  2 root root  4096 Mar 19 00:12 dev
 4 drwxr-xr-x 59 root root  4096 Mar 19 18:08 etc
 4 drwxr-xr-x  2 root root  4096 Apr 10  2018 home
 0 lrwxrwxrwx  1 root root     7 Mar 19 00:12 lib -> usr/lib
 0 lrwxrwxrwx  1 root root     9 Mar 19 00:12 lib64 -> usr/lib64
16 drwx------  2 root root 16384 Mar 19 18:08 lost+found
 4 drwxr-xr-x  2 root root  4096 Apr 10  2018 media
 4 drwxr-xr-x  2 root root  4096 Apr 10  2018 mnt
 4 drwxr-xr-x  2 root root  4096 Apr 10  2018 opt
 4 dr-xr-xr-x  2 root root  4096 Apr 10  2018 proc
 4 dr-xr-x---  3 root root  4096 Mar 19 18:08 root
 4 drwxr-xr-x  3 root root  4096 Mar 19 00:12 run
 0 lrwxrwxrwx  1 root root     8 Mar 19 00:12 sbin -> usr/sbin
 4 drwxr-xr-x  2 root root  4096 Mar 19 00:14 selinux
 4 drwxr-xr-x  2 root root  4096 Apr 10  2018 srv
 4 dr-xr-xr-x  2 root root  4096 Apr 10  2018 sys
 4 drwxrwxrwt  2 root root  4096 Apr 10  2018 tmp
 4 drwxr-xr-x 13 root root  4096 Mar 19 00:12 usr
 4 drwxr-xr-x 18 root root  4096 Mar 19 00:12 var
0 Likes

(Robert Brooks) #5

strace shows that a mount is failing, but I have no idea why, given the path exists…

1917811 clone(child_stack=0, flags=CLONE_NEWNS|CLONE_NEWUTS|CLONE_NEWIPC|CLONE_NEWPID|CLONE_NEWNET|SIGCHLD) = 1917839
1917839 close(12 <unfinished ...>
1917811 open("/proc/1917839/ns/mnt", O_RDONLY|O_CLOEXEC <unfinished ...>
1917839 <... close resumed> )           = 0
1917839 prctl(PR_SET_PDEATHSIG, SIGKILL <unfinished ...>
1917839 <... prctl resumed> )           = 0
1917811 open("/proc/1917839/ns/pid", O_RDONLY|O_CLOEXEC <unfinished ...>
1917839 getppid( <unfinished ...>
1917839 <... getppid resumed> )         = 0
1917811 open("/proc/1917839/ns/uts", O_RDONLY|O_CLOEXEC <unfinished ...>
1917839 getuid( <unfinished ...>
1917839 <... getuid resumed> )          = 0
1917811 open("/proc/1917839/ns/ipc", O_RDONLY|O_CLOEXEC <unfinished ...>
1917839 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
1917839 read(11,  <unfinished ...>
1917811 open("/proc/1917839/ns/net", O_RDONLY|O_CLOEXEC) = 19
1917839 <... read resumed> "\0\0\0\0", 4) = 4
1917839 write(11, "\1\0\0\0", 4 <unfinished ...>
1917839 <... write resumed> )           = 4
1917839 read(11,  <unfinished ...>
1917811 write(11, "1917839", 7)         = 7
1917811 write(11, "1917839", 7)         = 7
1917811 write(11, "1917839", 7)         = 7
1917811 write(11, "1917839", 7)         = 7
1917811 write(11, "1917839", 7)         = 7
1917811 write(11, "1917839", 7)         = 7
1917811 write(11, "1917839", 7)         = 7
1917811 write(11, "1917839", 7)         = 7
1917811 write(11, "1917839", 7)         = 7
1917811 write(11, "1917839", 7)         = 7
1917811 write(11, "1917839", 7)         = 7
1917811 write(11, "1917839", 7)         = 7
1917811 open("/proc/1917839/ns/net", O_RDONLY|O_CLOEXEC) = 11
1917839 <... read resumed> "\2\0\0\0", 4) = 4
1917839 access("/var/snap/lxd/common/lxd/containers", X_OK) = 0
1917839 write(11, "\3\0\0\0", 4)        = 4
1917839 read(11,  <unfinished ...>
1917839 <... read resumed> "\4\0\0\0", 4) = 4
1917839 unshare(CLONE_NEWCGROUP)        = 0
1917839 open("/proc/self/mountinfo", O_RDONLY|O_CLOEXEC) = 12
1917839 memfd_create(".lxc_mountinfo", MFD_CLOEXEC) = 15
1917839 sendfile(15, 12, NULL, 2147479552) = 11033
1917839 close(12)                       = 0
1917839 lseek(15, 0, SEEK_SET)          = 0
1917839 fcntl(15, F_GETFL)              = 0x8002 (flags O_RDWR|O_LARGEFILE)
1917839 fstat(15, {st_dev=makedev(0, 5), st_ino=19205626, st_mode=S_IFREG|0777, st_nlink=0, st_uid=0, st_gid=0, st_blksize=4096, st_blocks=24, st_size=11033, st_atime=2019/03/26-15:26:36.935141750, st_mtime=2019/03/26-15:26:36.935141750, st_ctime=2019/03/26-15:26:36.935141750}) = 0
1917839 read(15, "217 216 7:0 / / ro,nodev,relatime master:34 - squashfs /dev/loop0 ro\n218 217 0:6 / /dev rw,nosuid master:3 - devtmpfs devtmpfs rw,size=131869172k,nr_inodes=32967293,mode=755\n219 218 0:21 / /dev/shm rw,nosuid,nodev master:4 - tmpfs tmpfs rw\n220 218 0:22 / /dev/pts rw,nosuid,noexec,relatime master:5 - devpts devpts rw,gid=5,mode=620,ptmxmode=000\n221 220 0:47 / /dev/pts rw,relatime - devpts devpts rw,gid=5,mode=620,ptmxmode=666\n222 218 0:18 / /dev/mqueue rw,relatime master:27 - mqueue mqueue rw\n223 218 0:40 / /dev/hugepages rw,relatime master:28 - hugetlbfs hugetlbfs rw,pagesize=2M\n224 218 0:47 /ptmx /dev/ptmx rw,relatime - devpts devpts rw,gid=5,mode=620,ptmxmode=666\n225 217 253:3 / /home rw,relatime master:31 - ext4 /dev/mapper/vg01-home rw\n226 225 253:5 /lib/one /home/oneadmin rw,relatime master:32 - ext4 /dev/mapper/vg01-var rw\n227 226 7:6 / /home/oneadmin/datastores/100/743/mapper/disk.1 ro,relatime master:49 - iso9660 /dev/loop6 ro,nojoliet,check=s,map=n,blocksize=2048\n228 226 7:7 /"..., 4096) = 4096
1917839 mount(NULL, "/var/snap/lxd/common/shmounts", NULL, MS_SLAVE, NULL) = 0
1917839 mount(NULL, "/var/snap/lxd/common/shmounts/lxcfs", NULL, MS_SLAVE, NULL) = 0
1917839 read(15, "ed:45 - tmpfs tmpfs rw,size=100k,mode=711\n258 254 43:2304 / /var/snap/lxd/common/lxd/storage-pools/default/containers/one-743/rootfs rw,relatime master:46 - ext4 /dev/nbd9 rw\n259 254 43:2561 / /var/snap/lxd/common/lxd/storage-pools/default/containers/one-744/rootfs rw,relatime master:53 - xfs /dev/nbd10p1 rw,attr2,inode64,noquota\n260 254 0:55 / /var/snap/lxd/common/lxd/devlxd rw,relatime - tmpfs tmpfs rw,size=100k,mode=755\n261 254 252:16 / /var/snap/lxd/common/lxd/storage-pools/default/containers/c1 rw,relatime - ext4 /dev/rbd1 rw,discard,stripe=1024\n262 254 7:6 / /var/snap/lxd/common/lxd/devices/one-743/disk.context.context ro,relatime master:49 - iso9660 /dev/loop6 ro,nojoliet,check=s,map=n,blocksize=2048\n263 254 252:0 / /var/snap/lxd/common/lxd/storage-pools/default/containers/one-743 rw,relatime - ext4 /dev/rbd0 rw,discard,stripe=1024\n264 217 253:5 /lib/snapd /var/lib/snapd rw,relatime master:32 - ext4 /dev/mapper/vg01-var rw\n265 264 7:1 / /var/lib/snapd/snap/lxd/10343 ro,nodev,rel"..., 4096) = 4096
1917839 mount(NULL, "/var/snap/lxd/common/shmounts/containers", NULL, MS_SLAVE, NULL) = 0
1917839 read(15, "419 325 0:42 / /var/lib/snapd/hostfs/mnt/auto/app rw,noatime master:37 - nfs admin1.vip.riskiq:/app rw,vers=3,rsize=32768,wsize=32768,namlen=255,soft,proto=tcp,timeo=3,retrans=5,sec=sys,mountaddr=10.8.1.4,mountvers=3,mountport=59321,mountproto=udp,local_lock=none,addr=10.8.1.4\n422 268 0:6 / /var/lib/snapd/hostfs/dev rw,nosuid master:3 - devtmpfs devtmpfs rw,size=131869172k,nr_inodes=32967293,mode=755\n423 268 0:4 / /var/lib/snapd/hostfs/proc rw,nosuid,nodev,noexec,relatime master:6 - proc proc rw\n424 268 0:20 / /var/lib/snapd/hostfs/sys rw,nosuid,nodev,noexec,relatime master:7 - sysfs sysfs rw\n425 424 0:54 / /var/lib/snapd/hostfs/sys/fs/fuse/connections rw,relatime master:274 - fusectl fusectl rw\n426 217 253:5 /tmp /var/tmp rw,relatime master:32 - ext4 /dev/mapper/vg01-var rw\n427 217 253:1 /lib/modules /lib/modules rw,relatime master:2 - ext4 /dev/mapper/vg01-usr rw\n428 217 253:1 /src /usr/src rw,relatime master:2 - ext4 /dev/mapper/vg01-usr rw\n429 217 253:5 /log /var/log rw,relatime ma"..., 4096) = 2841
1917839 mount(NULL, "/media", NULL, MS_SLAVE, NULL) = 0
1917839 read(15, "", 4096)              = 0
1917839 close(15)                       = 0
1917839 access("/var/snap/lxd/common/lxc/", F_OK) = 0
1917839 mount("/var/snap/lxd/common/lxd/containers/one-743/rootfs", "/var/snap/lxd/common/lxc/", 0x7f88fd51cb21, MS_BIND|MS_REC, NULL) = -1 ENOENT (No such file or directory)
1917839 write(7, "lxc one-743 20190326222636.937 ERROR    dir - storage/dir.c:dir_mount:198 - No such file or directory - Failed to mount \"/var/snap/lxd/common/lxd/containers/one-743/rootfs\" on \"/var/snap/lxd/common/lxc/\"\n", 204) = 204
1917839 write(7, "lxc one-743 20190326222636.938 ERROR    conf - conf.c:lxc_mount_rootfs:1351 - Failed to mount rootfs \"/var/snap/lxd/common/lxd/containers/one-743/rootfs\" onto \"/var/snap/lxd/common/lxc/\" with options \"(null)\"\n", 209) = 209
1917839 write(7, "lxc one-743 20190326222636.938 ERROR    conf - conf.c:lxc_setup_rootfs_prepare_root:3498 - Failed to setup rootfs for\n", 118) = 118
1917839 write(7, "lxc one-743 20190326222636.938 ERROR    conf - conf.c:lxc_setup:3551 - Failed to setup rootfs\n", 94) = 94
1917839 close(14)                       = 0
1917839 close(13)                       = 0
1917839 write(7, "lxc one-743 20190326222636.938 ERROR    start - start.c:do_start:1282 - Failed to setup container \"one-743\"\n", 108) = 108
1917839 write(11, "\377\377\377\377", 4) = 4
1917839 exit_group(-1)                  = ?
1917839 +++ exited with 255 +++
1917811 kill(1917839, SIGKILL <unfinished ...>
1917811 wait4(-1, [{WIFEXITED(s) && WEXITSTATUS(s) == 255}], 0, NULL) = 1917839
1917811 --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=1917839, si_uid=0, si_status=255, si_utime=0, si_stime=0} ---
0 Likes