VMFS Datastore doesn't work correctly

Hi everyone,

I have installed ONE 4.10.2 with the ESXi5.0 on CentOS 7, basically following this installation guide. Everything gone well, however the image datastore does not shows capacity, because of this I am unable to import images or create a VM.

Bridge_list is correct. The persistent passwordless SSH connection either.

ESXi Host IP: “10.235.2.225”
ONE Front-End IP: “10.235.2.227”

I suspect that there is some problem with the VMFS DS driver.
I have created a NFS shared datastore on the ONE front-end and mapped it trough the ESXi Host.

[oneadmin@localhost ~]$ ll /var/lib/one/datastores/
total 0
drwxr-x—. 2 oneadmin oneadmin 6 Jan 15 2015 0
drwxr-x—. 2 oneadmin oneadmin 6 Oct 15 01:16 1
drwxrwxr-x. 3 oneadmin oneadmin 16 Oct 15 01:39 103
drwxr-xr-x. 2 oneadmin oneadmin 6 Oct 14 21:55 2

NFS Share Details

/var/lib/one/datastores/0 *(rw,sync,no_subtree_check,root_squash,anonuid=9869,anongid=9869)
/var/lib/one/datastores/1 *(rw,sync,no_subtree_check,root_squash,anonuid=9869,anongid=9869)

Nfs share are mounted on Esxi Host with same Datastore Id

These Datastore are accessible by the Host, but the ONE front end can’t monitor the IMAGE_DS datastore. The

Datastore information:

[oneadmin@localhost ~]$ onedatastore list
ID NAME                SIZE AVAIL     CLUSTER      IMAGES TYPE DS       TM
0 system                8.5G 80%      -                 0 sys  -        vmfs
1 default                 0M -        -                 0 img  vmfs     vmfs
2 files                 8.5G 79%      -                 0 fil  fs       ssh
103 imagens             8.5G 79%      -                 1 img  fs       ssh

DATASTORE 1 INFORMATION

ID : 1
NAME : default
USER : oneadmin
GROUP : oneadmin
CLUSTER : -
TYPE : IMAGE
DS_MAD : vmfs
TM_MAD : vmfs
BASE PATH : /vmfs/volumes/1
DISK_TYPE : FILE

DATASTORE CAPACITY
TOTAL: : 0M
FREE: : 0M
USED: : 0M
LIMIT: : -

PERMISSIONS
OWNER : um-
GROUP : u–
OTHER : u–

DATASTORE TEMPLATE
BASE_PATH=“/vmfs/volumes/”
BRIDGE_LIST=“10.235.2.225”
CLONE_TARGET=“SYSTEM”
DISK_TYPE=“FILE”
DS_MAD=“vmfs”
LN_TARGET=“NONE”
TM_MAD=“vmfs”
TYPE=“IMAGE_DS”

DATASTORE 0 INFORMATION

ID : 0
NAME : system
USER : oneadmin
GROUP : oneadmin
CLUSTER : -
TYPE : SYSTEM
DS_MAD : -
TM_MAD : vmfs
BASE PATH : /vmfs/volumes/0
DISK_TYPE : FILE

DATASTORE CAPACITY
TOTAL: : 8.5G
FREE: : 6.8G
USED: : 1.7G
LIMIT: : -

PERMISSIONS
OWNER : um-
GROUP : u–
OTHER : —

DATASTORE TEMPLATE
BASE_PATH=“/vmfs/volumes/”
SHARED=“YES”
TM_MAD=“vmfs”
TYPE=“SYSTEM_DS”

DATASTORE 103 INFORMATION

ID : 103
NAME : imagens
USER : oneadmin
GROUP : oneadmin
CLUSTER : -
TYPE : IMAGE
DS_MAD : fs
TM_MAD : ssh
BASE PATH : /var/lib/one/datastores/103/
DISK_TYPE : FILE

DATASTORE CAPACITY
TOTAL: : 8.5G
FREE: : 6.7G
USED: : 103M
LIMIT: : -

PERMISSIONS
OWNER : um-
GROUP : u–
OTHER : u–

DATASTORE TEMPLATE
BASE_PATH=“/var/lib/one/datastores”
BRIDGE_LIST=“10.235.2.227”
CLONE_TARGET=“SYSTEM”
DISK_TYPE=“FILE”
DS_MAD=“fs”
LN_TARGET=“SYSTEM”
TM_MAD=“ssh”
TYPE=“IMAGE_DS”

When I change the DS_TYPE to SYSTEM_DS, in the datastore 1, the ONE front-end monitor it normally.

Can anybody help me?

Thank you.

=====================================

ONED.LOG

Tue Oct 18 23:20:01 2016 [Z0][ONE][I]: Starting OpenNebula 4.10.2

 OpenNebula Configuration File      

AUTH_MAD=AUTHN=ssh,x509,ldap,server_cipher,server_x509,EXECUTABLE=one_auth_mad
DATASTORE_BASE_PATH=/vmfs/volumes
DATASTORE_CAPACITY_CHECK=yes
DATASTORE_LOCATION=/vmfs/volumes
DATASTORE_MAD=ARGUMENTS=-t 15 -d dummy,fs,vmfs,lvm,ceph,dev,EXECUTABLE=one_datastore
DB=BACKEND=sqlite
DEFAULT_CDROM_DEVICE_PREFIX=sd
DEFAULT_DEVICE_PREFIX=sd
DEFAULT_IMAGE_TYPE=OS
DEFAULT_UMASK=177
ENABLE_OTHER_PERMISSIONS=YES
FEDERATION=MASTER_ONED=,MODE=STANDALONE,ZONE_ID=0
HM_MAD=EXECUTABLE=one_hm
HOST_MONITORING_EXPIRATION_TIME=43200
HOST_PER_INTERVAL=15
IMAGE_RESTRICTED_ATTR=SOURCE
IM_MAD=ARGUMENTS=-p 4124 -f 5 -t 50 -i 20,EXECUTABLE=collectd,NAME=collectd
IM_MAD=ARGUMENTS=-r 3 -t 15 kvm,EXECUTABLE=one_im_ssh,NAME=kvm
IM_MAD=ARGUMENTS=-r 3 -t 15 xen4,EXECUTABLE=one_im_ssh,NAME=xen
IM_MAD=ARGUMENTS=-c -t 15 -r 0 vmware,EXECUTABLE=one_im_sh,NAME=vmware
INHERIT_DATASTORE_ATTR=CEPH_HOST
INHERIT_DATASTORE_ATTR=CEPH_SECRET
INHERIT_DATASTORE_ATTR=CEPH_USER
INHERIT_DATASTORE_ATTR=GLUSTER_HOST
INHERIT_DATASTORE_ATTR=GLUSTER_VOLUME
INHERIT_VNET_ATTR=VLAN_TAGGED_ID
INHERIT_VNET_ATTR=BRIDGE_OVS
KEEPALIVE_MAX_CONN=30
KEEPALIVE_TIMEOUT=15
LOG=DEBUG_LEVEL=3,SYSTEM=file
LOG_CALL_FORMAT=Req:%i UID:%u %m invoked %l
MAC_PREFIX=02:00
MANAGER_TIMER=15
MAX_CONN=15
MAX_CONN_BACKLOG=15
MESSAGE_SIZE=1073741824
MONITORING_INTERVAL=60
MONITORING_THREADS=50
NETWORK_SIZE=254
PORT=2633
RPC_LOG=NO
SCRIPTS_REMOTE_DIR=/var/tmp/one
SESSION_EXPIRATION_TIME=900
TIMEOUT=15
TM_MAD=ARGUMENTS=-t 15 -d dummy,lvm,shared,fs_lvm,qcow2,ssh,vmfs,ceph,dev,EXECUTABLE=one_tm
TM_MAD_CONF=CLONE_TARGET=SYSTEM,LN_TARGET=NONE,NAME=dummy,SHARED=yes
TM_MAD_CONF=CLONE_TARGET=SELF,LN_TARGET=NONE,NAME=lvm,SHARED=yes
TM_MAD_CONF=CLONE_TARGET=SYSTEM,LN_TARGET=NONE,NAME=shared,SHARED=yes
TM_MAD_CONF=CLONE_TARGET=SYSTEM,LN_TARGET=SYSTEM,NAME=fs_lvm,SHARED=yes
TM_MAD_CONF=CLONE_TARGET=SYSTEM,LN_TARGET=NONE,NAME=qcow2,SHARED=yes
TM_MAD_CONF=CLONE_TARGET=SYSTEM,LN_TARGET=SYSTEM,NAME=ssh,SHARED=no
TM_MAD_CONF=CLONE_TARGET=SYSTEM,LN_TARGET=NONE,NAME=vmfs,SHARED=yes
TM_MAD_CONF=CLONE_TARGET=SELF,LN_TARGET=NONE,NAME=ceph,SHARED=yes
TM_MAD_CONF=CLONE_TARGET=NONE,LN_TARGET=NONE,NAME=dev,SHARED=yes
VM_INDIVIDUAL_MONITORING=no
VM_MAD=ARGUMENTS=-t 15 -r 0 kvm,DEFAULT=vmm_exec/vmm_exec_kvm.conf,EXECUTABLE=one_vmm_exec,NAME=kvm,TYPE=kvm
VM_MAD=ARGUMENTS=-t 15 -r 0 xen4,DEFAULT=vmm_exec/vmm_exec_xen4.conf,EXECUTABLE=one_vmm_exec,NAME=xen,TYPE=xen
VM_MAD=ARGUMENTS=-t 15 -r 0 vmware -s sh,DEFAULT=vmm_exec/vmm_exec_vmware.conf,EXECUTABLE=one_vmm_sh,NAME=vmware,TYPE=vmware
VM_MONITORING_EXPIRATION_TIME=14400
VM_PER_INTERVAL=5
VM_RESTRICTED_ATTR=CONTEXT/FILES
VM_RESTRICTED_ATTR=NIC/MAC
VM_RESTRICTED_ATTR=NIC/VLAN_ID
VM_RESTRICTED_ATTR=NIC/BRIDGE
VM_RESTRICTED_ATTR=NIC_DEFAULT/MAC
VM_RESTRICTED_ATTR=NIC_DEFAULT/VLAN_ID
VM_RESTRICTED_ATTR=NIC_DEFAULT/BRIDGE
VM_RESTRICTED_ATTR=DISK/TOTAL_BYTES_SEC
VM_RESTRICTED_ATTR=DISK/READ_BYTES_SEC
VM_RESTRICTED_ATTR=DISK/WRITE_BYTES_SEC
VM_RESTRICTED_ATTR=DISK/TOTAL_IOPS_SEC
VM_RESTRICTED_ATTR=DISK/READ_IOPS_SEC
VM_RESTRICTED_ATTR=DISK/WRITE_IOPS_SEC
VM_SUBMIT_ON_HOLD=NO
VNC_BASE_PORT=5900
VNET_RESTRICTED_ATTR=PHYDEV
VNET_RESTRICTED_ATTR=VLAN_ID
VNET_RESTRICTED_ATTR=VLAN
VNET_RESTRICTED_ATTR=BRIDGE
VNET_RESTRICTED_ATTR=AR/PHYDEV
VNET_RESTRICTED_ATTR=AR/VLAN_ID
VNET_RESTRICTED_ATTR=AR/VLAN
VNET_RESTRICTED_ATTR=AR/BRIDGE

Tue Oct 18 23:20:01 2016 [Z0][ONE][I]: Log level:3 [0=ERROR,1=WARNING,2=INFO,3=DEBUG]
Tue Oct 18 23:20:01 2016 [Z0][ONE][I]: Support for xmlrpc-c > 1.31: yes
Tue Oct 18 23:20:01 2016 [Z0][ONE][I]: Checking database version.
Tue Oct 18 23:20:01 2016 [Z0][ONE][I]: oned is using version 4.9.80 for local_db_versioning
Tue Oct 18 23:20:01 2016 [Z0][ONE][I]: oned is using version 4.6.0 for db_versioning
Tue Oct 18 23:20:01 2016 [Z0][ACL][I]: Starting ACL Manager…
Tue Oct 18 23:20:01 2016 [Z0][ACL][I]: ACL Manager started.
Tue Oct 18 23:20:01 2016 [Z0][VMM][I]: Starting Virtual Machine Manager…
Tue Oct 18 23:20:01 2016 [Z0][LCM][I]: Starting Life-cycle Manager…
Tue Oct 18 23:20:01 2016 [Z0][InM][I]: Starting Information Manager…
Tue Oct 18 23:20:01 2016 [Z0][TrM][I]: Starting Transfer Manager…
Tue Oct 18 23:20:01 2016 [Z0][DiM][I]: Starting Dispatch Manager…
Tue Oct 18 23:20:01 2016 [Z0][HKM][I]: Starting Hook Manager…
Tue Oct 18 23:20:01 2016 [Z0][AuM][I]: Starting Auth Manager…
Tue Oct 18 23:20:01 2016 [Z0][ImM][I]: Starting Image Manager…
Tue Oct 18 23:20:02 2016 [Z0][AuM][I]: Authorization Manager started.
Tue Oct 18 23:20:02 2016 [Z0][ImM][I]: Image Manager started.
Tue Oct 18 23:20:02 2016 [Z0][InM][I]: Information Manager started.
Tue Oct 18 23:20:02 2016 [Z0][VMM][I]: Virtual Machine Manager started.
Tue Oct 18 23:20:02 2016 [Z0][TrM][I]: Transfer Manager started.
Tue Oct 18 23:20:02 2016 [Z0][DiM][I]: Dispatch Manager started.
Tue Oct 18 23:20:02 2016 [Z0][LCM][I]: Life-cycle Manager started.
Tue Oct 18 23:20:02 2016 [Z0][HKM][I]: Hook Manager started.
Tue Oct 18 23:20:02 2016 [Z0][VMM][I]: Loading Virtual Machine Manager drivers.
Tue Oct 18 23:20:02 2016 [Z0][VMM][I]: Loading driver: kvm (KVM)
Tue Oct 18 23:20:03 2016 [Z0][VMM][I]: Driver kvm loaded.
Tue Oct 18 23:20:03 2016 [Z0][VMM][I]: Loading driver: xen (XEN)
Tue Oct 18 23:20:03 2016 [Z0][VMM][I]: Driver xen loaded.
Tue Oct 18 23:20:03 2016 [Z0][VMM][I]: Loading driver: vmware (VMWARE)
Tue Oct 18 23:20:03 2016 [Z0][VMM][I]: Driver vmware loaded.
Tue Oct 18 23:20:03 2016 [Z0][InM][I]: Loading Information Manager drivers.
Tue Oct 18 23:20:03 2016 [Z0][InM][I]: Loading driver: collectd
Tue Oct 18 23:20:03 2016 [Z0][InM][I]: Driver collectd loaded
Tue Oct 18 23:20:03 2016 [Z0][InM][I]: Loading driver: kvm
Tue Oct 18 23:20:03 2016 [Z0][InM][I]: Driver kvm loaded
Tue Oct 18 23:20:03 2016 [Z0][InM][I]: Loading driver: xen
Tue Oct 18 23:20:03 2016 [Z0][InM][I]: Driver xen loaded
Tue Oct 18 23:20:03 2016 [Z0][InM][I]: Loading driver: vmware
Tue Oct 18 23:20:03 2016 [Z0][InM][I]: Driver vmware loaded
Tue Oct 18 23:20:03 2016 [Z0][TM][I]: Loading Transfer Manager driver.
Tue Oct 18 23:20:03 2016 [Z0][TM][I]: Transfer manager driver loaded
Tue Oct 18 23:20:03 2016 [Z0][HKM][I]: Loading Hook Manager driver.
Tue Oct 18 23:20:04 2016 [Z0][HKM][I]: Hook Manager loaded
Tue Oct 18 23:20:04 2016 [Z0][ImM][I]: Loading Image Manager driver.
Tue Oct 18 23:20:04 2016 [Z0][ImM][I]: Image Manager loaded
Tue Oct 18 23:20:04 2016 [Z0][AuM][I]: Loading Auth. Manager driver.
Tue Oct 18 23:20:04 2016 [Z0][AuM][I]: Auth Manager loaded
Tue Oct 18 23:20:04 2016 [Z0][ReM][I]: Starting Request Manager…
Tue Oct 18 23:20:04 2016 [Z0][ReM][I]: Starting XML-RPC server, port 2633 …
Tue Oct 18 23:20:04 2016 [Z0][ReM][I]: Request Manager started.
Tue Oct 18 23:20:06 2016 [Z0][ReM][D]: Req:4032 UID:0 SystemConfig invoked
Tue Oct 18 23:20:06 2016 [Z0][ReM][D]: Req:4032 UID:0 SystemConfig result SUCCESS, “<AUTH_MAD>…”
Tue Oct 18 23:20:16 2016 [Z0][InM][D]: Monitoring datastore default (1)
Tue Oct 18 23:20:17 2016 [Z0][InM][D]: Monitoring datastore files (2)
Tue Oct 18 23:20:17 2016 [Z0][InM][D]: Monitoring datastore imagens (103)
Tue Oct 18 23:20:17 2016 [Z0][ImM][D]: Datastore imagens (103) successfully monitored.
Tue Oct 18 23:20:17 2016 [Z0][ImM][D]: Datastore files (2) successfully monitored.
Tue Oct 18 23:20:17 2016 [Z0][ImM][D]: Datastore default (1) successfully monitored.
Tue Oct 18 23:20:32 2016 [Z0][InM][D]: Monitoring host 10.235.2.225 (0)
Tue Oct 18 23:20:34 2016 [Z0][ImM][D]: Datastore system (0) successfully monitored.
Tue Oct 18 23:20:34 2016 [Z0][InM][D]: Host 10.235.2.225 (0) successfully monitored.
Tue Oct 18 23:20:35 2016 [Z0][ReM][D]: Req:9136 UID:0 VirtualMachinePoolInfo invoked , -2, -1, -1, -1
Tue Oct 18 23:20:35 2016 [Z0][ReM][D]: Req:9136 UID:0 VirtualMachinePoolInfo result SUCCESS, “<VM_POOL></VM_POOL>”
Tue Oct 18 23:20:36 2016 [Z0][ReM][D]: Req:1152 UID:0 VirtualMachinePoolInfo invoked , -2, -1, -1, -1
Tue Oct 18 23:20:36 2016 [Z0][ReM][D]: Req:1152 UID:0 VirtualMachinePoolInfo result SUCCESS, “<VM_POOL></VM_POOL>”
Tue Oct 18 23:21:06 2016 [Z0][ReM][D]: Req:5040 UID:0 VirtualMachinePoolInfo invoked , -2, -1, -1, -1
Tue Oct 18 23:21:06 2016 [Z0][ReM][D]: Req:5040 UID:0 VirtualMachinePoolInfo result SUCCESS, “<VM_POOL></VM_POOL>”
Tue Oct 18 23:21:06 2016 [Z0][ReM][D]: Req:3168 UID:0 VirtualMachinePoolInfo invoked , -2, -1, -1, -1
Tue Oct 18 23:21:06 2016 [Z0][ReM][D]: Req:3168 UID:0 VirtualMachinePoolInfo result SUCCESS, “<VM_POOL></VM_POOL>”
Tue Oct 18 23:21:13 2016 [Z0][InM][D]: Monitoring datastore default (1)
Tue Oct 18 23:21:13 2016 [Z0][InM][D]: Monitoring datastore files (2)
Tue Oct 18 23:21:13 2016 [Z0][InM][D]: Monitoring datastore imagens (103)
Tue Oct 18 23:21:14 2016 [Z0][ImM][D]: Datastore files (2) successfully monitored.
Tue Oct 18 23:21:14 2016 [Z0][ImM][D]: Datastore imagens (103) successfully monitored.
Tue Oct 18 23:21:14 2016 [Z0][ImM][D]: Datastore default (1) successfully monitored.
Tue Oct 18 23:21:36 2016 [Z0][ReM][D]: Req:8960 UID:0 VirtualMachinePoolInfo invoked , -2, -1, -1, -1
Tue Oct 18 23:21:36 2016 [Z0][ReM][D]: Req:8960 UID:0 VirtualMachinePoolInfo result SUCCESS, “<VM_POOL></VM_POOL>”
Tue Oct 18 23:21:36 2016 [Z0][ReM][D]: Req:1136 UID:0 VirtualMachinePoolInfo invoked , -2, -1, -1, -1
Tue Oct 18 23:21:36 2016 [Z0][ReM][D]: Req:1136 UID:0 VirtualMachinePoolInfo result SUCCESS, “<VM_POOL></VM_POOL>”

Solved. The problem was with the outdated ESXi 5.0- Build 474610.
Somehow every time I log in with the oneadmin user via SSH the ESXI
don’t grant me the root privilege, even with the user oneadmin being in
the root group and with administrator role activated. Thus the df
command could not be executed and the monitor.py script couldn’t
determine the datastore space.

After the update to ESXi 5.0 build 3982828 released in 2016-06-14 this problem was solved. But 'til that a lot of headache.