Opennebula and SElinux

Hi,

Is it possible to make Opennebula and selinux friends with each other?
Virtual machines failed to boot/migrate/… after enforsing selinux and relabeling fs on the kvm nodes:

Tue Aug 13 09:09:48 2019 [Z0][VM][I]: New LCM state is BOOT
Tue Aug 13 09:09:48 2019 [Z0][VMM][I]: Generating deployment file: /var/lib/one/vms/260/deployment.0
Tue Aug 13 09:09:49 2019 [Z0][VMM][I]: Successfully execute transfer manager driver operation: tm_context.
Tue Aug 13 09:09:50 2019 [Z0][VMM][I]: ExitCode: 0
Tue Aug 13 09:09:50 2019 [Z0][VMM][I]: Successfully execute network driver operation: pre.
Tue Aug 13 09:09:51 2019 [Z0][VMM][I]: Command execution fail: cat << EOT | /var/tmp/one/vmm/kvm/deploy '/var/lib/one//datastores/101/260/deployment.0' 'one-kvm-node-03-int' 260 one-kvm-node-03-int
Tue Aug 13 09:09:51 2019 [Z0][VMM][I]: error: Failed to create domain from /var/lib/one//datastores/101/260/deployment.0
Tue Aug 13 09:09:51 2019 [Z0][VMM][I]: error: internal error: qemu unexpectedly closed the monitor: 2019-08-13T06:09:50.794535Z qemu-kvm: -drive file=/var/lib/one//datastores/101/260/disk.0,format=qcow2,if=none,id=drive-virtio-disk0,cache=none: Could not open '/var/lib/one//datastores/101/260/disk.0': Permission denied
Tue Aug 13 09:09:51 2019 [Z0][VMM][E]: Could not create domain from /var/lib/one//datastores/101/260/deployment.0
Tue Aug 13 09:09:51 2019 [Z0][VMM][I]: ExitCode: 255
Tue Aug 13 09:09:51 2019 [Z0][VMM][I]: Failed to execute virtualization driver operation: deploy.
Tue Aug 13 09:09:51 2019 [Z0][VMM][E]: Error deploying virtual machine: Could not create domain from /var/lib/one//datastores/101/260/deployment.0
Tue Aug 13 09:09:51 2019 [Z0][VM][I]: New LCM state is BOOT_FAILURE

grep one-260 /var/log/audit/audit.log | grep fail
type=VIRT_CONTROL msg=audit(1565676335.173:1603): pid=2303 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:virtd_t:s0-s0:c0.c1023 msg='virt=kvm op=start reason=booted vm="one-260" uuid=45e72a89-302e-49b5-98be-ab59494bde01 vm-pid=-1 exe="/usr/sbin/libvirtd" hostname=? addr=? terminal=? res=failed'
type=VIRT_CONTROL msg=audit(1565676591.013:1731): pid=2303 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:virtd_t:s0-s0:c0.c1023 msg='virt=kvm op=start reason=booted vm="one-260" uuid=df20a2b8-9bdd-43b6-8d6d-ec1965779a1c vm-pid=-1 exe="/usr/sbin/libvirtd" hostname=? addr=? terminal=? res=failed'

DS is an NFS share.

# getsebool virt_use_nfs
virt_use_nfs --> on

FS Permissions seems to be okay:

ls -lZ /var/lib/one//datastores/101/260/disk.0
lrwxrwxrwx. oneadmin oneadmin system_u:object_r:nfs_t:s0       /var/lib/one//datastores/101/260/disk.0 -> disk.0.snap/0

SElinux status:

# sestatus 
SELinux status:                 enabled
SELinuxfs mount:                /sys/fs/selinux
SELinux root directory:         /etc/selinux
Loaded policy name:             targeted
Current mode:                   enforcing
Mode from config file:          enforcing
Policy MLS status:              enabled
Policy deny_unknown status:     allowed
Max kernel policy version:      31

With selinux mode turned back to disabled everything is okay.


Hi, here is some documentation

and here is also github issue

You can of course contribute.

Hi @feldsam,

I’ve already saw this “some documentation” :slight_smile:

If the administrator isn’t experienced in the SELinux configuration, it’s recommended to disable this functionality to avoid unexpected failures. You can enable SELinux anytime later when you have the installation working.

I believed that someone has more positive experience. :wink:
In any case thanks for your time.

Did you try some toubleshoting? https://access.redhat.com/articles/2191331

Yes, I’ve tried but not hard enough.
Error messages said me nothing:

type=USER_ACCT msg=audit(1565617723.987:122): pid=3193 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=PAM:accounting grantors=? acct="oneadmin" exe="/usr/sbin/sshd" hostname=mgt-host addr=mgt-ip terminal=ssh res=failed'

type=VIRT_CONTROL msg=audit(1565675914.848:5222): pid=3260 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:virtd_t:s0-s0:c0.c1023 msg='virt=kvm op=start reason=booted vm="one-259" uuid=ea539569-a158-4b6a-b99e-c26b2c95d8b7 vm-pid=-1 exe="/usr/sbin/libvirtd" hostname=? addr=? terminal=? res=failed'

So I’ve just disabled selinux.

And ausearch shows some denieds to partitions or something else:

time->Tue Aug 13 08:58:34 2019
type=PROCTITLE msg=audit(1565675914.627:5214): proctitle=2F7573722F6C6962657865632F71656D752D6B766D002D6E616D650067756573743D6F6E652D3235392C64656275672D746872656164733D6F6E002D53002D6F626A656374007365637265742C69643D6D61737465724B6579302C666F726D61743D7261772C66696C653D2F7661722F6C69622F6C6962766972742F71656D75
type=SYSCALL msg=audit(1565675914.627:5214): arch=c000003e syscall=2 success=no exit=-13 a0=5599e2bafe60 a1=84000 a2=0 a3=5599e2c26004 items=0 ppid=1 pid=27678 auid=4294967295 uid=9869 gid=9869 euid=9869 suid=9869 fsuid=9869 egid=9869 sgid=9869 fsgid=9869 tty=(none) ses=4294967295 comm="qemu-kvm" exe="/usr/libexec/q
type=AVC msg=audit(1565675914.627:5214): avc:  denied  { read } for  pid=27678 comm="qemu-kvm" name="101" dev="sda4" ino=13109 scontext=system_u:system_r:svirt_t:s0:c348,c746 tcontext=system_u:object_r:var_lib_t:s0 tclass=lnk_file permissive=0

sda4 - part of mirror, ‘home’ partition.
That’s all I found for now.

Hello,

don’t you have NFS datastores symlinked from different mount point location somewhere into the /var/lib/one/datastores/ ?

Best regards,
Vlastimil

Hi,

Yes, I do.

I have NFS share for images and directory inside it which is symlinked
as datastore.

So did we (you indeed) find the problem?

14.08.19 17:26, Vlastimil Holer via OpenNebula Community пише:

Policy doesn’t allow access to files with SELinux type var_lib_t which are actually symlinks.

One option is to relabel the datastores directories to more suitable virt_image_t, e.g.:

semanage fcontext -a -t virt_image_t '/var/lib/one/datastores(/.*)?'
restorecon -Rv /var/lib/one/datastores/

In your case, this won’t be probably enough, so you should do the same also for the custom location you are symlinking to.

The other option is to make a custom module, which allows reading var_lib_t even if they are symlinks. For example:

# cat <<EOF >opennebula.te
module opennebula 1.0;

require {
	type svirt_t;
	type var_lib_t;
	class lnk_file read;
}

#============= svirt_t ==============

#!!!! WARNING: 'var_lib_t' is a base type.
allow svirt_t var_lib_t:lnk_file read;
EOF

# checkmodule -M -m -o opennebula.mod opennebula.te
# semodule_package -o opennebula.pp -m opennebula.mod
# semodule -i opennebula.pp

Best regards,
Vlastimil

Okay. Thanks for an explanation.
I’ll try to create an ordinary nfs share instead of symlink whereupon relabel datastores.

Confirm. Symlinc was a root cause of problem.