Bootstrapping OpenNebula 5.4 from source

Community Support
1

Hi all,

I’m in the process of trying to get OpenNebula to run on a newly built cluster. The full gory details of my set-up is at https://hackaday.io/project/10529 but the basics are:

  • 3-node Ceph cluster (combined OSD/MON nodes)
  • 2-node OpenNebula cluster for running VMs
  • Front-end will be a HA-VM running as a VM with back-end via RBD on Ceph

At present, all nodes have Intel Atom C2750 CPUs (8-core 2.4GHz) and have 8GB ECC RAM, a 120GB SSD (Samsung 850-EVO)… the storage nodes have a 1TB HDD as well (HGST HTS541010A9).

For now I’ve just configured a VM in KVM/libvirt which I plan to set up in heartbeat/corosync to allow it to migrate between the two compute nodes. That’s a future project. I know OpenNebula now supports a HA front-end, but once you have a HA VM storage back-end, it’s easy to just have a VM configured on two hosts and start it on one.

The VM itself is running Gentoo Linux with the musl C library (to reduce the memory footprint): it has 512MB RAM, one CPU core and 8GB of disk allocated. Gentoo was chosen because the sources are right there handy, which when developing software is a godsend (no need to figure out which -dev or -devel package is needed)!

I compiled OpenNebula easy enough. I’ve since discovered that I need to call install.sh multiple times with different flags to install the various components. I initially ran ./install.sh -u opennebula -g opennebula -d /opt/opennebula and it installed nearly everything… Sunstone would tell me that “Sinatra doesn’t know that ditty”.

I discovered I had to separately install Sunstone with ./install.sh -u opennebula -g opennebula -d /opt/opennebula -s -p.

I have configured /opt/opennebula/etc/oned.conf with my MySQL database credentials and manually created /opt/opennebula/.one/one_auth (weak password for now: will change it once oned is up). Starting oned -f (which must be done as the opennebula user with ONE_LOCATION set to /opt/opennebula), I get the following in /opt/opennebula/var/oned.log:

Sat Jul 29 00:56:54 2017 [Z0][ONE][I]: Starting OpenNebula 5.4.0
----------------------------------------
     OpenNebula Configuration File      
----------------------------------------
AUTH_MAD=AUTHN=ssh,x509,ldap,server_cipher,server_x509,EXECUTABLE=one_auth_mad
AUTH_MAD_CONF=DRIVER_MANAGED_GROUPS=NO,MAX_TOKEN_TIME=-1,NAME=core,PASSWORD_CHANGE=YES
AUTH_MAD_CONF=DRIVER_MANAGED_GROUPS=NO,MAX_TOKEN_TIME=-1,NAME=public,PASSWORD_CHANGE=NO
AUTH_MAD_CONF=DRIVER_MANAGED_GROUPS=NO,MAX_TOKEN_TIME=-1,NAME=ssh,PASSWORD_CHANGE=YES
AUTH_MAD_CONF=DRIVER_MANAGED_GROUPS=NO,MAX_TOKEN_TIME=-1,NAME=x509,PASSWORD_CHANGE=NO
AUTH_MAD_CONF=DRIVER_MANAGED_GROUPS=YES,MAX_TOKEN_TIME=86400,NAME=ldap,PASSWORD_CHANGE=YES
AUTH_MAD_CONF=DRIVER_MANAGED_GROUPS=NO,MAX_TOKEN_TIME=-1,NAME=server_cipher,PASSWORD_CHANGE=NO
AUTH_MAD_CONF=DRIVER_MANAGED_GROUPS=NO,MAX_TOKEN_TIME=-1,NAME=server_x509,PASSWORD_CHANGE=NO
DATASTORE_CAPACITY_CHECK=yes
DATASTORE_LOCATION=/opt/opennebula/var//datastores
DATASTORE_MAD=ARGUMENTS=-t 15 -d dummy,fs,lvm,ceph,dev,iscsi_libvirt,vcenter -s shared,ssh,ceph,fs_lvm,qcow2,vcenter,EXECUTABLE=one_datastore
DB=BACKEND=mysql,DB_NAME=opennebula,PASSWD=fnsN9i3U4EGA1DpKV9aAw4qRtPk9RjqxpSuhpzOKGvxrPrFrmCybbFslPrDV7+xf,PORT=0,SERVER=localhost,USER=opennebula
DEFAULT_AUTH=default
DEFAULT_CDROM_DEVICE_PREFIX=hd
DEFAULT_COST=CPU_COST=0,DISK_COST=0,MEMORY_COST=0
DEFAULT_DEVICE_PREFIX=hd
DEFAULT_IMAGE_PERSISTENT=
DEFAULT_IMAGE_PERSISTENT_NEW=
DEFAULT_IMAGE_TYPE=OS
DEFAULT_UMASK=177
DS_MAD_CONF=MARKETPLACE_ACTIONS=export,NAME=ceph,PERSISTENT_ONLY=NO,REQUIRED_ATTRS=DISK_TYPE,BRIDGE_LIST
DS_MAD_CONF=NAME=dev,PERSISTENT_ONLY=YES,REQUIRED_ATTRS=DISK_TYPE
DS_MAD_CONF=NAME=iscsi_libvirt,PERSISTENT_ONLY=YES,REQUIRED_ATTRS=DISK_TYPE,ISCSI_HOST
DS_MAD_CONF=NAME=dummy,PERSISTENT_ONLY=NO,REQUIRED_ATTRS=
DS_MAD_CONF=MARKETPLACE_ACTIONS=export,NAME=fs,PERSISTENT_ONLY=NO,REQUIRED_ATTRS=
DS_MAD_CONF=NAME=lvm,PERSISTENT_ONLY=NO,REQUIRED_ATTRS=DISK_TYPE,BRIDGE_LIST
DS_MAD_CONF=MARKETPLACE_ACTIONS=export,NAME=vcenter,PERSISTENT_ONLY=NO,REQUIRED_ATTRS=VCENTER_INSTANCE_ID, VCENTER_DS_REF, VCENTER_DC_REF
DS_MAD_CONF=NAME=shared,PERSISTENT_ONLY=NO,REQUIRED_ATTRS=
DS_MAD_CONF=NAME=ssh,PERSISTENT_ONLY=NO,REQUIRED_ATTRS=
DS_MAD_CONF=NAME=vmfs,PERSISTENT_ONLY=NO,REQUIRED_ATTRS=BRIDGE_LIST
ENABLE_OTHER_PERMISSIONS=YES
FEDERATION=MASTER_ONED=,MODE=STANDALONE,SERVER_ID=-1,ZONE_ID=0
HM_MAD=EXECUTABLE=one_hm
HOST_MONITORING_EXPIRATION_TIME=43200
HOST_PER_INTERVAL=15
IMAGE_RESTRICTED_ATTR=SOURCE
IM_MAD=ARGUMENTS=-p 4124 -f 5 -t 50 -i 20,EXECUTABLE=collectd,NAME=collectd
IM_MAD=ARGUMENTS=-r 3 -t 15 kvm,EXECUTABLE=one_im_ssh,NAME=kvm,SUNSTONE_NAME=KVM
IM_MAD=ARGUMENTS=-c -t 15 -r 0 vcenter,EXECUTABLE=one_im_sh,NAME=vcenter,SUNSTONE_NAME=VMWare vCenter
INHERIT_DATASTORE_ATTR=CEPH_HOST
INHERIT_DATASTORE_ATTR=CEPH_SECRET
INHERIT_DATASTORE_ATTR=CEPH_KEY
INHERIT_DATASTORE_ATTR=CEPH_USER
INHERIT_DATASTORE_ATTR=CEPH_CONF
INHERIT_DATASTORE_ATTR=POOL_NAME
INHERIT_DATASTORE_ATTR=ISCSI_USER
INHERIT_DATASTORE_ATTR=ISCSI_USAGE
INHERIT_DATASTORE_ATTR=ISCSI_HOST
INHERIT_DATASTORE_ATTR=GLUSTER_HOST
INHERIT_DATASTORE_ATTR=GLUSTER_VOLUME
INHERIT_DATASTORE_ATTR=DISK_TYPE
INHERIT_DATASTORE_ATTR=ALLOW_ORPHANS
INHERIT_DATASTORE_ATTR=VCENTER_ADAPTER_TYPE
INHERIT_DATASTORE_ATTR=VCENTER_DISK_TYPE
INHERIT_DATASTORE_ATTR=VCENTER_DS_REF
INHERIT_DATASTORE_ATTR=VCENTER_DS_IMAGE_DIR
INHERIT_DATASTORE_ATTR=VCENTER_DS_VOLATILE_DIR
INHERIT_DATASTORE_ATTR=VCENTER_INSTANCE_ID
INHERIT_IMAGE_ATTR=ISCSI_USER
INHERIT_IMAGE_ATTR=ISCSI_USAGE
INHERIT_IMAGE_ATTR=ISCSI_HOST
INHERIT_IMAGE_ATTR=ISCSI_IQN
INHERIT_IMAGE_ATTR=DISK_TYPE
INHERIT_IMAGE_ATTR=VCENTER_ADAPTER_TYPE
INHERIT_IMAGE_ATTR=VCENTER_DISK_TYPE
INHERIT_VNET_ATTR=VLAN_TAGGED_ID
INHERIT_VNET_ATTR=FILTER_IP_SPOOFING
INHERIT_VNET_ATTR=FILTER_MAC_SPOOFING
INHERIT_VNET_ATTR=MTU
INHERIT_VNET_ATTR=INBOUND_AVG_BW
INHERIT_VNET_ATTR=INBOUND_PEAK_BW
INHERIT_VNET_ATTR=INBOUND_PEAK_KB
INHERIT_VNET_ATTR=OUTBOUND_AVG_BW
INHERIT_VNET_ATTR=OUTBOUND_PEAK_BW
INHERIT_VNET_ATTR=OUTBOUND_PEAK_KB
INHERIT_VNET_ATTR=CONF
INHERIT_VNET_ATTR=BRIDGE_CONF
INHERIT_VNET_ATTR=IP_LINK_CONF
INHERIT_VNET_ATTR=VCENTER_NET_REF
INHERIT_VNET_ATTR=VCENTER_SWITCH_NAME
INHERIT_VNET_ATTR=VCENTER_SWITCH_NPORTS
INHERIT_VNET_ATTR=VCENTER_PORTGROUP_TYPE
INHERIT_VNET_ATTR=VCENTER_CCR_REF
INHERIT_VNET_ATTR=VCENTER_INSTANCE_ID
IPAM_MAD=ARGUMENTS=-t 1 -i dummy,EXECUTABLE=one_ipam
KEEPALIVE_MAX_CONN=30
KEEPALIVE_TIMEOUT=15
LISTEN_ADDRESS=0.0.0.0
LOG=DEBUG_LEVEL=3,SYSTEM=file
LOG_CALL_FORMAT=Req:%i UID:%u %m invoked %l
MAC_PREFIX=02:00
MANAGER_TIMER=15
MARKET_MAD=ARGUMENTS=-t 15 -m http,s3,one,EXECUTABLE=one_market
MARKET_MAD_CONF=APP_ACTIONS=monitor,NAME=one,PUBLIC=yes,REQUIRED_ATTRS=,SUNSTONE_NAME=OpenNebula.org Marketplace
MARKET_MAD_CONF=APP_ACTIONS=create, delete, monitor,NAME=http,REQUIRED_ATTRS=BASE_URL,PUBLIC_DIR,SUNSTONE_NAME=HTTP server
MARKET_MAD_CONF=APP_ACTIONS=create, delete, monitor,NAME=s3,REQUIRED_ATTRS=ACCESS_KEY_ID,SECRET_ACCESS_KEY,REGION,BUCKET,SUNSTONE_NAME=Amazon S3
MAX_CONN=15
MAX_CONN_BACKLOG=15
MESSAGE_SIZE=1073741824
MONITORING_INTERVAL=60
MONITORING_THREADS=50
NETWORK_SIZE=254
PCI_PASSTHROUGH_BUS=0x01
PORT=2633
RAFT=BROADCAST_TIMEOUT_MS=500,ELECTION_TIMEOUT_MS=2500,LOG_PURGE_TIMEOUT=600,LOG_RETENTION=500000,XMLRPC_TIMEOUT_MS=2000
RPC_LOG=NO
SCRIPTS_REMOTE_DIR=/var/tmp/one
SESSION_EXPIRATION_TIME=900
TIMEOUT=15
TM_MAD=ARGUMENTS=-t 15 -d dummy,lvm,shared,fs_lvm,qcow2,ssh,ceph,dev,vcenter,iscsi_libvirt,EXECUTABLE=one_tm
TM_MAD_CONF=CLONE_TARGET=SYSTEM,DS_MIGRATE=YES,LN_TARGET=NONE,NAME=dummy,SHARED=YES
TM_MAD_CONF=CLONE_TARGET=SELF,LN_TARGET=NONE,NAME=lvm,SHARED=YES
TM_MAD_CONF=CLONE_TARGET=SYSTEM,DS_MIGRATE=YES,LN_TARGET=NONE,NAME=shared,SHARED=YES
TM_MAD_CONF=CLONE_TARGET=SYSTEM,DRIVER=raw,LN_TARGET=SYSTEM,NAME=fs_lvm,SHARED=YES
TM_MAD_CONF=CLONE_TARGET=SYSTEM,DRIVER=qcow2,LN_TARGET=NONE,NAME=qcow2,SHARED=YES
TM_MAD_CONF=CLONE_TARGET=SYSTEM,DS_MIGRATE=YES,LN_TARGET=SYSTEM,NAME=ssh,SHARED=NO
TM_MAD_CONF=ALLOW_ORPHANS=yes,CLONE_TARGET=SELF,DRIVER=raw,DS_MIGRATE=NO,LN_TARGET=NONE,NAME=ceph,SHARED=YES
TM_MAD_CONF=CLONE_TARGET=SELF,DS_MIGRATE=NO,LN_TARGET=NONE,NAME=iscsi_libvirt,SHARED=YES
TM_MAD_CONF=CLONE_TARGET=NONE,LN_TARGET=NONE,NAME=dev,SHARED=YES
TM_MAD_CONF=CLONE_TARGET=SYSTEM,LN_TARGET=NONE,NAME=vcenter,SHARED=YES
TM_MAD_CONF=CLONE_TARGET=SYSTEM,DRIVER=,DS_MIGRATE=NO,LN_TARGET=NONE,NAME=vmfs,SHARED=YES
VLAN_IDS=RESERVED=0, 1, 4095,START=2
VM_ADMIN_OPERATIONS=migrate, delete, recover, retry, deploy, resched
VM_INDIVIDUAL_MONITORING=no
VM_MAD=ARGUMENTS=-t 15 -r 0 kvm,DEFAULT=vmm_exec/vmm_exec_kvm.conf,EXECUTABLE=one_vmm_exec,IMPORTED_VMS_ACTIONS=terminate, terminate-hard, hold, release, suspend,
        resume, delete, reboot, reboot-hard, resched, unresched, disk-attach,
        disk-detach, nic-attach, nic-detach, snap-create, snap-delete,KEEP_SNAPSHOTS=no,NAME=kvm,SUNSTONE_NAME=KVM,TYPE=kvm
VM_MAD=ARGUMENTS=-p -t 15 -r 0 vcenter -s sh,DEFAULT=vmm_exec/vmm_exec_vcenter.conf,EXECUTABLE=one_vmm_sh,IMPORTED_VMS_ACTIONS=terminate, terminate-hard, hold, release, suspend,
        resume, delete, reboot, reboot-hard, resched, unresched, poweroff,
        poweroff-hard, disk-attach, disk-detach, nic-attach, nic-detach,
        snap-create, snap-delete,KEEP_SNAPSHOTS=yes,NAME=vcenter,SUNSTONE_NAME=VMWare vCenter,TYPE=xml
VM_MANAGE_OPERATIONS=undeploy, hold, release, stop, suspend, resume, reboot,
    poweroff, disk-attach, nic-attach, disk-snapshot, terminate, disk-resize,
    snapshot, updateconf, rename, resize, update, disk-saveas
VM_MONITORING_EXPIRATION_TIME=14400
VM_PER_INTERVAL=5
VM_RESTRICTED_ATTR=CONTEXT/FILES
VM_RESTRICTED_ATTR=NIC/MAC
VM_RESTRICTED_ATTR=NIC/VLAN_ID
VM_RESTRICTED_ATTR=NIC/BRIDGE
VM_RESTRICTED_ATTR=NIC_DEFAULT/MAC
VM_RESTRICTED_ATTR=NIC_DEFAULT/VLAN_ID
VM_RESTRICTED_ATTR=NIC_DEFAULT/BRIDGE
VM_RESTRICTED_ATTR=CPU_COST
VM_RESTRICTED_ATTR=MEMORY_COST
VM_RESTRICTED_ATTR=DISK_COST
VM_RESTRICTED_ATTR=PCI
VM_RESTRICTED_ATTR=EMULATOR
VM_RESTRICTED_ATTR=RAW
VM_RESTRICTED_ATTR=USER_PRIORITY
VM_SUBMIT_ON_HOLD=NO
VM_USE_OPERATIONS=
VNC_PORTS=START=5900
VNET_RESTRICTED_ATTR=VN_MAD
VNET_RESTRICTED_ATTR=PHYDEV
VNET_RESTRICTED_ATTR=VLAN_ID
VNET_RESTRICTED_ATTR=BRIDGE
VNET_RESTRICTED_ATTR=CONF
VNET_RESTRICTED_ATTR=BRIDGE_CONF
VNET_RESTRICTED_ATTR=IP_LINK_CONF
VNET_RESTRICTED_ATTR=AR/VN_MAD
VNET_RESTRICTED_ATTR=AR/PHYDEV
VNET_RESTRICTED_ATTR=AR/VLAN_ID
VNET_RESTRICTED_ATTR=AR/BRIDGE
VXLAN_IDS=START=2
----------------------------------------
Sat Jul 29 00:56:54 2017 [Z0][ONE][I]: Log level:3 [0=ERROR,1=WARNING,2=INFO,3=DEBUG]
Sat Jul 29 00:56:54 2017 [Z0][ONE][I]: Support for xmlrpc-c > 1.31: no. MAX_CONN and MAX_CONN_BACKLOG configuration will not be used
Sat Jul 29 00:56:54 2017 [Z0][ONE][I]: Checking database version.
Sat Jul 29 00:56:54 2017 [Z0][ONE][I]: oned is using version 5.4.0 for local_db_versioning
Sat Jul 29 00:56:54 2017 [Z0][ONE][I]: oned is using version 5.4.0 for db_versioning
Sat Jul 29 00:56:54 2017 [Z0][ACL][I]: Starting ACL Manager...
Sat Jul 29 00:56:54 2017 [Z0][ACL][I]: ACL Manager started.
Sat Jul 29 00:56:54 2017 [Z0][HKM][I]: Starting Hook Manager...
Sat Jul 29 00:56:54 2017 [Z0][HKM][I]: Loading Hook Manager driver.
Sat Jul 29 00:56:54 2017 [Z0][HKM][I]: Hook Manager started.
Sat Jul 29 00:56:54 2017 [Z0][HKM][I]: 	Hook Manager loaded
Sat Jul 29 00:56:54 2017 [Z0][ONE][I]: oned started in solo mode.
Sat Jul 29 00:56:54 2017 [Z0][RCM][I]: Starting Raft Consensus Manager...
Sat Jul 29 00:56:54 2017 [Z0][FRM][I]: Starting Federation Replica Manager...
Sat Jul 29 00:56:54 2017 [Z0][VMM][I]: Starting Virtual Machine Manager...
Sat Jul 29 00:56:54 2017 [Z0][LCM][I]: Starting Life-cycle Manager...
Sat Jul 29 00:56:54 2017 [Z0][InM][I]: Starting Information Manager...
Sat Jul 29 00:56:54 2017 [Z0][TrM][I]: Starting Transfer Manager...
Sat Jul 29 00:56:54 2017 [Z0][DiM][I]: Starting Dispatch Manager...
Sat Jul 29 00:56:54 2017 [Z0][AuM][I]: Starting Auth Manager...
Sat Jul 29 00:56:54 2017 [Z0][ImM][I]: Starting Image Manager...
Sat Jul 29 00:56:54 2017 [Z0][ImM][I]: Starting Marketplace Manager...
Sat Jul 29 00:56:54 2017 [Z0][IPM][I]: Starting IPAM Manager...
Sat Jul 29 00:56:54 2017 [Z0][IPM][I]: IPAM Manager started.
Sat Jul 29 00:56:54 2017 [Z0][MKP][I]: Marketplace Manager started.
Sat Jul 29 00:56:54 2017 [Z0][ImM][I]: Image Manager started.
Sat Jul 29 00:56:54 2017 [Z0][AuM][I]: Authorization Manager started.
Sat Jul 29 00:56:54 2017 [Z0][DiM][I]: Dispatch Manager started.
Sat Jul 29 00:56:54 2017 [Z0][TrM][I]: Transfer Manager started.
Sat Jul 29 00:56:54 2017 [Z0][LCM][I]: Life-cycle Manager started.
Sat Jul 29 00:56:54 2017 [Z0][InM][I]: Information Manager started.
Sat Jul 29 00:56:54 2017 [Z0][VMM][I]: Virtual Machine Manager started.
Sat Jul 29 00:56:54 2017 [Z0][FRM][I]: Federation Replica Manger started.
Sat Jul 29 00:56:54 2017 [Z0][RCM][I]: Raft Consensus Manager started.
Sat Jul 29 00:56:56 2017 [Z0][VMM][I]: Loading Virtual Machine Manager drivers.
Sat Jul 29 00:56:56 2017 [Z0][VMM][I]: 	Loading driver: kvm (KVM)
Sat Jul 29 00:56:56 2017 [Z0][VMM][E]: Wrong action: snap-create
Sat Jul 29 00:56:56 2017 [Z0][VMM][E]: Wrong action: snap-delete
Sat Jul 29 00:56:57 2017 [Z0][VMM][I]: 	Driver kvm loaded.
Sat Jul 29 00:56:57 2017 [Z0][VMM][I]: 	Loading driver: vcenter (XML)
Sat Jul 29 00:56:57 2017 [Z0][VMM][E]: Wrong action: snap-create
Sat Jul 29 00:56:57 2017 [Z0][VMM][E]: Wrong action: snap-delete
Sat Jul 29 00:56:58 2017 [Z0][VMM][I]: 	Driver vcenter loaded.
Sat Jul 29 00:56:58 2017 [Z0][InM][I]: Loading Information Manager drivers.
Sat Jul 29 00:56:58 2017 [Z0][InM][I]: 	Loading driver: collectd
Sat Jul 29 00:56:58 2017 [Z0][MAD][E]: MAD did not answer INIT command
Sat Jul 29 00:56:58 2017 [Z0][ONE][E]: Could not load driver

Of particular concern are these lines:

Sat Jul 29 00:56:56 2017 [Z0][VMM][I]: 	Loading driver: kvm (KVM)
Sat Jul 29 00:56:56 2017 [Z0][VMM][E]: Wrong action: snap-create
Sat Jul 29 00:56:56 2017 [Z0][VMM][E]: Wrong action: snap-delete
Sat Jul 29 00:56:58 2017 [Z0][MAD][E]: MAD did not answer INIT command
Sat Jul 29 00:56:58 2017 [Z0][ONE][E]: Could not load driver

Any idea where these errors are creeping in?

2

Okay, so not sure what’s going on… but it would appear that the out-of-box oned.conf needs snap-create and snap-delete removed from the list for kvm:

VM_MAD = [
    NAME           = "kvm",
    SUNSTONE_NAME  = "KVM",
    EXECUTABLE     = "one_vmm_exec",
    ARGUMENTS      = "-t 15 -r 0 kvm",
    DEFAULT        = "vmm_exec/vmm_exec_kvm.conf",
    TYPE           = "kvm",
    KEEP_SNAPSHOTS = "no",
    IMPORTED_VMS_ACTIONS = "terminate, terminate-hard, hold, release, suspend,
        resume, delete, reboot, reboot-hard, resched, unresched, disk-attach,
        disk-detach, nic-attach, nic-detach"
        #, snap-create, snap-delete"
]

and the collectd section commented out:

#IM_MAD = [
#      NAME       = "collectd",
#      EXECUTABLE = "collectd",
#      ARGUMENTS  = "-p 4124 -f 5 -t 50 -i 20" ]

Otherwise, it refuses to start.

3

… even then we don’t seem to be responding to requests.

I’ve cleaned up my install scripts so it installs in the standard locations now. The LSB init scripts have been converted to OpenRC scripts, I’m able to launch the server as normal.

It starts off well enough:

opennebula /var/log/one # tail -n +0 -F *.log
==> oned.log <==
Sat Jul 29 05:34:04 2017 [Z0][ONE][I]: Starting OpenNebula 5.4.0
----------------------------------------
     OpenNebula Configuration File
----------------------------------------
AUTH_MAD=AUTHN=ssh,x509,ldap,server_cipher,server_x509,EXECUTABLE=one_auth_mad
AUTH_MAD_CONF=DRIVER_MANAGED_GROUPS=NO,MAX_TOKEN_TIME=-1,NAME=core,PASSWORD_CHANGE=YES
AUTH_MAD_CONF=DRIVER_MANAGED_GROUPS=NO,MAX_TOKEN_TIME=-1,NAME=public,PASSWORD_CHANGE=NO
AUTH_MAD_CONF=DRIVER_MANAGED_GROUPS=NO,MAX_TOKEN_TIME=-1,NAME=ssh,PASSWORD_CHANGE=YES
AUTH_MAD_CONF=DRIVER_MANAGED_GROUPS=NO,MAX_TOKEN_TIME=-1,NAME=x509,PASSWORD_CHANGE=NO
AUTH_MAD_CONF=DRIVER_MANAGED_GROUPS=YES,MAX_TOKEN_TIME=86400,NAME=ldap,PASSWORD_CHANGE=YES
AUTH_MAD_CONF=DRIVER_MANAGED_GROUPS=NO,MAX_TOKEN_TIME=-1,NAME=server_cipher,PASSWORD_CHANGE=NO
AUTH_MAD_CONF=DRIVER_MANAGED_GROUPS=NO,MAX_TOKEN_TIME=-1,NAME=server_x509,PASSWORD_CHANGE=NO
DATASTORE_CAPACITY_CHECK=yes
DATASTORE_LOCATION=/var/lib/one//datastores
DATASTORE_MAD=ARGUMENTS=-t 15 -d dummy,fs,lvm,ceph,dev,iscsi_libvirt,vcenter -s shared,ssh,ceph,fs_lvm,qcow2,vcenter,EXECUTABLE=one_datastore
DB=BACKEND=sqlite
DEFAULT_AUTH=default
DEFAULT_CDROM_DEVICE_PREFIX=hd
DEFAULT_COST=CPU_COST=0,DISK_COST=0,MEMORY_COST=0
DEFAULT_DEVICE_PREFIX=hd
DEFAULT_IMAGE_PERSISTENT=
DEFAULT_IMAGE_PERSISTENT_NEW=
DEFAULT_IMAGE_TYPE=OS
DEFAULT_UMASK=177
DS_MAD_CONF=MARKETPLACE_ACTIONS=export,NAME=ceph,PERSISTENT_ONLY=NO,REQUIRED_ATTRS=DISK_TYPE,BRIDGE_LIST
DS_MAD_CONF=NAME=dev,PERSISTENT_ONLY=YES,REQUIRED_ATTRS=DISK_TYPE
DS_MAD_CONF=NAME=iscsi_libvirt,PERSISTENT_ONLY=YES,REQUIRED_ATTRS=DISK_TYPE,ISCSI_HOST
DS_MAD_CONF=NAME=dummy,PERSISTENT_ONLY=NO,REQUIRED_ATTRS=
DS_MAD_CONF=MARKETPLACE_ACTIONS=export,NAME=fs,PERSISTENT_ONLY=NO,REQUIRED_ATTRS=
DS_MAD_CONF=NAME=lvm,PERSISTENT_ONLY=NO,REQUIRED_ATTRS=DISK_TYPE,BRIDGE_LIST
DS_MAD_CONF=MARKETPLACE_ACTIONS=export,NAME=vcenter,PERSISTENT_ONLY=NO,REQUIRED_ATTRS=VCENTER_INSTANCE_ID, VCENTER_DS_REF, VCENTER_DC_REF
DS_MAD_CONF=NAME=shared,PERSISTENT_ONLY=NO,REQUIRED_ATTRS=
DS_MAD_CONF=NAME=ssh,PERSISTENT_ONLY=NO,REQUIRED_ATTRS=
DS_MAD_CONF=NAME=vmfs,PERSISTENT_ONLY=NO,REQUIRED_ATTRS=BRIDGE_LIST
ENABLE_OTHER_PERMISSIONS=YES
FEDERATION=MASTER_ONED=,MODE=STANDALONE,SERVER_ID=-1,ZONE_ID=0
HM_MAD=EXECUTABLE=one_hm
HOST_MONITORING_EXPIRATION_TIME=43200
HOST_PER_INTERVAL=15
IMAGE_RESTRICTED_ATTR=SOURCE
IM_MAD=ARGUMENTS=-r 3 -t 15 kvm,EXECUTABLE=one_im_ssh,NAME=kvm,SUNSTONE_NAME=KVM
IM_MAD=ARGUMENTS=-c -t 15 -r 0 vcenter,EXECUTABLE=one_im_sh,NAME=vcenter,SUNSTONE_NAME=VMWare vCenter
INHERIT_DATASTORE_ATTR=CEPH_HOST
INHERIT_DATASTORE_ATTR=CEPH_SECRET
INHERIT_DATASTORE_ATTR=CEPH_KEY
INHERIT_DATASTORE_ATTR=CEPH_USER
INHERIT_DATASTORE_ATTR=CEPH_CONF
INHERIT_DATASTORE_ATTR=POOL_NAME
INHERIT_DATASTORE_ATTR=ISCSI_USER
INHERIT_DATASTORE_ATTR=ISCSI_USAGE
INHERIT_DATASTORE_ATTR=ISCSI_HOST
INHERIT_DATASTORE_ATTR=GLUSTER_HOST
INHERIT_DATASTORE_ATTR=GLUSTER_VOLUME
INHERIT_DATASTORE_ATTR=DISK_TYPE
INHERIT_DATASTORE_ATTR=ALLOW_ORPHANS
INHERIT_DATASTORE_ATTR=VCENTER_ADAPTER_TYPE
INHERIT_DATASTORE_ATTR=VCENTER_DISK_TYPE
INHERIT_DATASTORE_ATTR=VCENTER_DS_REF
INHERIT_DATASTORE_ATTR=VCENTER_DS_IMAGE_DIR
INHERIT_DATASTORE_ATTR=VCENTER_DS_VOLATILE_DIR
INHERIT_DATASTORE_ATTR=VCENTER_INSTANCE_ID
INHERIT_IMAGE_ATTR=ISCSI_USER
INHERIT_IMAGE_ATTR=ISCSI_USAGE
INHERIT_IMAGE_ATTR=ISCSI_HOST
INHERIT_IMAGE_ATTR=ISCSI_IQN
INHERIT_IMAGE_ATTR=DISK_TYPE
INHERIT_IMAGE_ATTR=VCENTER_ADAPTER_TYPE
INHERIT_IMAGE_ATTR=VCENTER_DISK_TYPE
INHERIT_VNET_ATTR=VLAN_TAGGED_ID
INHERIT_VNET_ATTR=FILTER_IP_SPOOFING
INHERIT_VNET_ATTR=FILTER_MAC_SPOOFING
INHERIT_VNET_ATTR=MTU
INHERIT_VNET_ATTR=INBOUND_AVG_BW
INHERIT_VNET_ATTR=INBOUND_PEAK_BW
INHERIT_VNET_ATTR=INBOUND_PEAK_KB
INHERIT_VNET_ATTR=OUTBOUND_AVG_BW
INHERIT_VNET_ATTR=OUTBOUND_PEAK_BW
INHERIT_VNET_ATTR=OUTBOUND_PEAK_KB
INHERIT_VNET_ATTR=CONF
INHERIT_VNET_ATTR=BRIDGE_CONF
INHERIT_VNET_ATTR=IP_LINK_CONF
INHERIT_VNET_ATTR=VCENTER_NET_REF
INHERIT_VNET_ATTR=VCENTER_SWITCH_NAME
INHERIT_VNET_ATTR=VCENTER_SWITCH_NPORTS
INHERIT_VNET_ATTR=VCENTER_PORTGROUP_TYPE
INHERIT_VNET_ATTR=VCENTER_CCR_REF
INHERIT_VNET_ATTR=VCENTER_INSTANCE_ID
IPAM_MAD=ARGUMENTS=-t 1 -i dummy,EXECUTABLE=one_ipam
KEEPALIVE_MAX_CONN=30
KEEPALIVE_TIMEOUT=15
LISTEN_ADDRESS=0.0.0.0
LOG=DEBUG_LEVEL=3,SYSTEM=file
LOG_CALL_FORMAT=Req:%i UID:%u %m invoked %l
MAC_PREFIX=02:00
MANAGER_TIMER=15
MARKET_MAD=ARGUMENTS=-t 15 -m http,s3,one,EXECUTABLE=one_market
MARKET_MAD_CONF=APP_ACTIONS=monitor,NAME=one,PUBLIC=yes,REQUIRED_ATTRS=,SUNSTONE_NAME=OpenNebula.org Marketplace
MARKET_MAD_CONF=APP_ACTIONS=create, delete, monitor,NAME=http,REQUIRED_ATTRS=BASE_URL,PUBLIC_DIR,SUNSTONE_NAME=HTTP server
MARKET_MAD_CONF=APP_ACTIONS=create, delete, monitor,NAME=s3,REQUIRED_ATTRS=ACCESS_KEY_ID,SECRET_ACCESS_KEY,REGION,BUCKET,SUNSTONE_NAME=Amazon S3
MAX_CONN=15
MAX_CONN_BACKLOG=15
MESSAGE_SIZE=1073741824
MONITORING_INTERVAL=60
MONITORING_THREADS=50
NETWORK_SIZE=254
PCI_PASSTHROUGH_BUS=0x01
PORT=2633
RAFT=BROADCAST_TIMEOUT_MS=500,ELECTION_TIMEOUT_MS=2500,LOG_PURGE_TIMEOUT=600,LOG_RETENTION=500000,XMLRPC_TIMEOUT_MS=2000
RPC_LOG=NO
SCRIPTS_REMOTE_DIR=/var/tmp/one
SESSION_EXPIRATION_TIME=900
TIMEOUT=15
TM_MAD=ARGUMENTS=-t 15 -d dummy,lvm,shared,fs_lvm,qcow2,ssh,ceph,dev,vcenter,iscsi_libvirt,EXECUTABLE=one_tm
TM_MAD_CONF=CLONE_TARGET=SYSTEM,DS_MIGRATE=YES,LN_TARGET=NONE,NAME=dummy,SHARED=YES
TM_MAD_CONF=CLONE_TARGET=SELF,LN_TARGET=NONE,NAME=lvm,SHARED=YES
TM_MAD_CONF=CLONE_TARGET=SYSTEM,DS_MIGRATE=YES,LN_TARGET=NONE,NAME=shared,SHARED=YES
TM_MAD_CONF=CLONE_TARGET=SYSTEM,DRIVER=raw,LN_TARGET=SYSTEM,NAME=fs_lvm,SHARED=YES
TM_MAD_CONF=CLONE_TARGET=SYSTEM,DRIVER=qcow2,LN_TARGET=NONE,NAME=qcow2,SHARED=YES
TM_MAD_CONF=CLONE_TARGET=SYSTEM,DS_MIGRATE=YES,LN_TARGET=SYSTEM,NAME=ssh,SHARED=NO
TM_MAD_CONF=ALLOW_ORPHANS=yes,CLONE_TARGET=SELF,DRIVER=raw,DS_MIGRATE=NO,LN_TARGET=NONE,NAME=ceph,SHARED=YES
TM_MAD_CONF=CLONE_TARGET=SELF,DS_MIGRATE=NO,LN_TARGET=NONE,NAME=iscsi_libvirt,SHARED=YES
TM_MAD_CONF=CLONE_TARGET=NONE,LN_TARGET=NONE,NAME=dev,SHARED=YES
TM_MAD_CONF=CLONE_TARGET=SYSTEM,LN_TARGET=NONE,NAME=vcenter,SHARED=YES
TM_MAD_CONF=CLONE_TARGET=SYSTEM,DRIVER=,DS_MIGRATE=NO,LN_TARGET=NONE,NAME=vmfs,SHARED=YES
VLAN_IDS=RESERVED=0, 1, 4095,START=2
VM_ADMIN_OPERATIONS=migrate, delete, recover, retry, deploy, resched
VM_INDIVIDUAL_MONITORING=no
VM_MAD=ARGUMENTS=-t 15 -r 0 kvm,DEFAULT=vmm_exec/vmm_exec_kvm.conf,EXECUTABLE=one_vmm_exec,IMPORTED_VMS_ACTIONS=terminate, terminate-hard, hold, release, suspend,
        resume, delete, reboot, reboot-hard, resched, unresched, disk-attach,
        disk-detach, nic-attach, nic-detach,KEEP_SNAPSHOTS=no,NAME=kvm,SUNSTONE_NAME=KVM,TYPE=kvm
VM_MANAGE_OPERATIONS=undeploy, hold, release, stop, suspend, resume, reboot,
    poweroff, disk-attach, nic-attach, disk-snapshot, terminate, disk-resize,
    snapshot, updateconf, rename, resize, update, disk-saveas
VM_MONITORING_EXPIRATION_TIME=14400
VM_PER_INTERVAL=5
VM_RESTRICTED_ATTR=CONTEXT/FILES
VM_RESTRICTED_ATTR=NIC/MAC
VM_RESTRICTED_ATTR=NIC/VLAN_ID
VM_RESTRICTED_ATTR=NIC/BRIDGE
VM_RESTRICTED_ATTR=NIC_DEFAULT/MAC
VM_RESTRICTED_ATTR=NIC_DEFAULT/VLAN_ID
VM_RESTRICTED_ATTR=NIC_DEFAULT/BRIDGE
VM_RESTRICTED_ATTR=CPU_COST
VM_RESTRICTED_ATTR=MEMORY_COST
VM_RESTRICTED_ATTR=DISK_COST
VM_RESTRICTED_ATTR=PCI
VM_RESTRICTED_ATTR=EMULATOR
VM_RESTRICTED_ATTR=RAW
VM_RESTRICTED_ATTR=USER_PRIORITY
VM_SUBMIT_ON_HOLD=NO
VM_USE_OPERATIONS=
VNC_PORTS=START=5900
VNET_RESTRICTED_ATTR=VN_MAD
VNET_RESTRICTED_ATTR=PHYDEV
VNET_RESTRICTED_ATTR=VLAN_ID
VNET_RESTRICTED_ATTR=BRIDGE
VNET_RESTRICTED_ATTR=CONF
VNET_RESTRICTED_ATTR=BRIDGE_CONF
VNET_RESTRICTED_ATTR=IP_LINK_CONF
VNET_RESTRICTED_ATTR=AR/VN_MAD
VNET_RESTRICTED_ATTR=AR/PHYDEV
VNET_RESTRICTED_ATTR=AR/VLAN_ID
VNET_RESTRICTED_ATTR=AR/BRIDGE
VXLAN_IDS=START=2
----------------------------------------
Sat Jul 29 05:34:04 2017 [Z0][ONE][I]: Log level:3 [0=ERROR,1=WARNING,2=INFO,3=DEBUG]
Sat Jul 29 05:34:04 2017 [Z0][ONE][I]: Support for xmlrpc-c > 1.31: no. MAX_CONN and MAX_CONN_BACKLOG configuration will not be used
Sat Jul 29 05:34:04 2017 [Z0][ONE][I]: Checking database version.
Sat Jul 29 05:34:04 2017 [Z0][ONE][I]: oned is using version 5.4.0 for local_db_versioning
Sat Jul 29 05:34:04 2017 [Z0][ONE][I]: oned is using version 5.4.0 for db_versioning
Sat Jul 29 05:34:04 2017 [Z0][ACL][I]: Starting ACL Manager...
Sat Jul 29 05:34:04 2017 [Z0][ACL][I]: ACL Manager started.
Sat Jul 29 05:34:04 2017 [Z0][HKM][I]: Starting Hook Manager...
Sat Jul 29 05:34:04 2017 [Z0][HKM][I]: Loading Hook Manager driver.
Sat Jul 29 05:34:04 2017 [Z0][HKM][I]: Hook Manager started.
Sat Jul 29 05:34:04 2017 [Z0][HKM][I]:  Hook Manager loaded
Sat Jul 29 05:34:04 2017 [Z0][ONE][I]: oned started in solo mode.
Sat Jul 29 05:34:04 2017 [Z0][RCM][I]: Starting Raft Consensus Manager...
Sat Jul 29 05:34:04 2017 [Z0][FRM][I]: Starting Federation Replica Manager...
Sat Jul 29 05:34:04 2017 [Z0][FRM][I]: Federation Replica Manger started.
Sat Jul 29 05:34:04 2017 [Z0][VMM][I]: Starting Virtual Machine Manager...
Sat Jul 29 05:34:04 2017 [Z0][LCM][I]: Starting Life-cycle Manager...
Sat Jul 29 05:34:04 2017 [Z0][VMM][I]: Virtual Machine Manager started.
Sat Jul 29 05:34:04 2017 [Z0][LCM][I]: Life-cycle Manager started.
Sat Jul 29 05:34:04 2017 [Z0][InM][I]: Starting Information Manager...
Sat Jul 29 05:34:04 2017 [Z0][InM][I]: Information Manager started.
Sat Jul 29 05:34:04 2017 [Z0][TrM][I]: Starting Transfer Manager...
Sat Jul 29 05:34:04 2017 [Z0][DiM][I]: Starting Dispatch Manager...
Sat Jul 29 05:34:04 2017 [Z0][TrM][I]: Transfer Manager started.
Sat Jul 29 05:34:04 2017 [Z0][RCM][I]: Raft Consensus Manager started.
Sat Jul 29 05:34:04 2017 [Z0][AuM][I]: Starting Auth Manager...
Sat Jul 29 05:34:04 2017 [Z0][ImM][I]: Starting Image Manager...
Sat Jul 29 05:34:04 2017 [Z0][AuM][I]: Authorization Manager started.
Sat Jul 29 05:34:04 2017 [Z0][ImM][I]: Image Manager started.
Sat Jul 29 05:34:04 2017 [Z0][DiM][I]: Dispatch Manager started.
Sat Jul 29 05:34:04 2017 [Z0][ImM][I]: Starting Marketplace Manager...
Sat Jul 29 05:34:04 2017 [Z0][IPM][I]: Starting IPAM Manager...
Sat Jul 29 05:34:04 2017 [Z0][MKP][I]: Marketplace Manager started.
Sat Jul 29 05:34:04 2017 [Z0][IPM][I]: IPAM Manager started.
Sat Jul 29 05:34:07 2017 [Z0][VMM][I]: Loading Virtual Machine Manager drivers.
Sat Jul 29 05:34:07 2017 [Z0][VMM][I]:  Loading driver: kvm (KVM)
Sat Jul 29 05:34:07 2017 [Z0][VMM][I]:  Driver kvm loaded.
Sat Jul 29 05:34:07 2017 [Z0][InM][I]: Loading Information Manager drivers.
Sat Jul 29 05:34:07 2017 [Z0][InM][I]:  Loading driver: kvm
Sat Jul 29 05:34:08 2017 [Z0][InM][I]:  Driver kvm loaded
Sat Jul 29 05:34:08 2017 [Z0][InM][I]:  Loading driver: vcenter
Sat Jul 29 05:34:08 2017 [Z0][InM][I]:  Driver vcenter loaded
Sat Jul 29 05:34:08 2017 [Z0][TM][I]: Loading Transfer Manager driver.
Sat Jul 29 05:34:08 2017 [Z0][TM][I]:   Transfer manager driver loaded
Sat Jul 29 05:34:08 2017 [Z0][ImM][I]: Loading Image Manager driver.
Sat Jul 29 05:34:08 2017 [Z0][ImM][I]:  Image Manager loaded
Sat Jul 29 05:34:08 2017 [Z0][MKP][I]: Loading Marketplace Manager driver.
Sat Jul 29 05:34:09 2017 [Z0][MKP][I]:  Marketplace Manager loaded
Sat Jul 29 05:34:09 2017 [Z0][IPM][I]: Loading IPAM Manager driver.
Sat Jul 29 05:34:09 2017 [Z0][IPM][I]:  IPAM Manager loaded
Sat Jul 29 05:34:09 2017 [Z0][AuM][I]: Loading Auth. Manager driver.
Sat Jul 29 05:34:10 2017 [Z0][AuM][I]:  Auth Manager loaded
Sat Jul 29 05:34:10 2017 [Z0][ReM][I]: Starting Request Manager...
Sat Jul 29 05:34:10 2017 [Z0][ReM][I]: Starting XML-RPC server, port 2633 ...
Sat Jul 29 05:34:10 2017 [Z0][ReM][I]: Request Manager started.
Sat Jul 29 05:34:10 2017 [Z0][ReM][D]: Req:0 UID:0 one.system.config invoked
Sat Jul 29 05:34:10 2017 [Z0][ReM][D]: Req:0 UID:0 one.system.config result SUCCESS, "<TEMPLATE><AUTH_MAD>..."
Sat Jul 29 05:34:19 2017 [Z0][MKP][D]: Monitoring marketplace OpenNebula Public (0)
Sat Jul 29 05:34:19 2017 [Z0][InM][D]: Monitoring datastore default (1)
Sat Jul 29 05:34:19 2017 [Z0][InM][D]: Monitoring datastore files (2)
Sat Jul 29 05:34:20 2017 [Z0][ImM][D]: Datastore files (2) successfully monitored.
Sat Jul 29 05:34:20 2017 [Z0][ImM][D]: Datastore default (1) successfully monitored.

==> sched.log <==
Sat Jul 29 05:34:04 2017 [Z0][SCHED][I]: Init Scheduler Log system
Sat Jul 29 05:34:04 2017 [Z0][SCHED][I]: Starting Scheduler Daemon
----------------------------------------
     Scheduler Configuration File
----------------------------------------
DEFAULT_DS_SCHED=POLICY=1
DEFAULT_SCHED=POLICY=1
LIVE_RESCHEDS=0
LOG=DEBUG_LEVEL=3,SYSTEM=file
MAX_DISPATCH=30
MAX_HOST=1
MAX_VM=5000
MESSAGE_SIZE=1073741824
ONE_XMLRPC=http://localhost:2633/RPC2
SCHED_INTERVAL=30
TIMEOUT=60
----------------------------------------
Sat Jul 29 05:34:04 2017 [Z0][SCHED][I]: XML-RPC client using 1073741824 bytes for response buffer.

Sat Jul 29 05:34:04 2017 [Z0][SCHED][E]: Cannot contact oned, will retry... Error: libcurl failed to execute the HTTP POST transaction, explaining:  Failed to connect to localhost port 2633: Connection refused
Sat Jul 29 05:34:06 2017 [Z0][SCHED][E]: Cannot contact oned, will retry... Error: libcurl failed to execute the HTTP POST transaction, explaining:  Failed to connect to localhost port 2633: Connection refused
Sat Jul 29 05:34:08 2017 [Z0][SCHED][E]: Cannot contact oned, will retry... Error: libcurl failed to execute the HTTP POST transaction, explaining:  Failed to connect to localhost port 2633: Connection refused
Sat Jul 29 05:34:10 2017 [Z0][SCHED][I]: oned successfully contacted.
Sat Jul 29 05:34:10 2017 [Z0][SCHED][I]: Configuring scheduler for Zone ID: 0
Sat Jul 29 05:34:10 2017 [Z0][SCHED][I]: Starting scheduler loop...
Sat Jul 29 05:34:10 2017 [Z0][SCHED][I]: Scheduler loop started.

==> oned.log <==
Sat Jul 29 05:34:20 2017 [Z0][MKP][D]: Marketplace OpenNebula Public (0) successfully monitored.
Sat Jul 29 05:34:40 2017 [Z0][ReM][D]: Req:0 UID:0 one.zone.raftstatus invoked
Sat Jul 29 05:34:40 2017 [Z0][ReM][D]: Req:0 UID:0 one.zone.raftstatus result SUCCESS, "<RAFT><SERVER_ID>-1<..."
Sat Jul 29 05:34:40 2017 [Z0][ReM][D]: Req:0 UID:0 one.vmpool.info invoked , -2, -1, -1, -1
Sat Jul 29 05:34:40 2017 [Z0][ReM][D]: Req:0 UID:0 one.vmpool.info result SUCCESS, "<VM_POOL></VM_POOL>"
Sat Jul 29 05:34:40 2017 [Z0][ReM][D]: Req:0 UID:0 one.vmpool.info invoked , -2, -1, -1, -1
Sat Jul 29 05:34:40 2017 [Z0][ReM][D]: Req:0 UID:0 one.vmpool.info result SUCCESS, "<VM_POOL></VM_POOL>"
Sat Jul 29 05:35:10 2017 [Z0][ReM][D]: Req:0 UID:0 one.zone.raftstatus invoked
Sat Jul 29 05:35:10 2017 [Z0][ReM][D]: Req:0 UID:0 one.zone.raftstatus result SUCCESS, "<RAFT><SERVER_ID>-1<..."
Sat Jul 29 05:35:10 2017 [Z0][ReM][D]: Req:0 UID:0 one.vmpool.info invoked , -2, -1, -1, -1
Sat Jul 29 05:35:10 2017 [Z0][ReM][D]: Req:0 UID:0 one.vmpool.info result SUCCESS, "<VM_POOL></VM_POOL>"
Sat Jul 29 05:35:10 2017 [Z0][ReM][D]: Req:0 UID:0 one.vmpool.info invoked , -2, -1, -1, -1
Sat Jul 29 05:35:10 2017 [Z0][ReM][D]: Req:0 UID:0 one.vmpool.info result SUCCESS, "<VM_POOL></VM_POOL>"
Sat Jul 29 05:35:19 2017 [Z0][MKP][D]: Monitoring marketplace OpenNebula Public (0)
Sat Jul 29 05:35:19 2017 [Z0][InM][D]: Monitoring datastore default (1)
Sat Jul 29 05:35:19 2017 [Z0][InM][D]: Monitoring datastore files (2)
Sat Jul 29 05:35:20 2017 [Z0][ImM][D]: Datastore files (2) successfully monitored.
Sat Jul 29 05:35:20 2017 [Z0][ImM][D]: Datastore default (1) successfully monitored.
Sat Jul 29 05:35:21 2017 [Z0][MKP][D]: Marketplace OpenNebula Public (0) successfully monitored.
Sat Jul 29 05:35:40 2017 [Z0][ReM][D]: Req:0 UID:0 one.zone.raftstatus invoked
Sat Jul 29 05:35:40 2017 [Z0][ReM][D]: Req:0 UID:0 one.zone.raftstatus result SUCCESS, "<RAFT><SERVER_ID>-1<..."
Sat Jul 29 05:35:40 2017 [Z0][ReM][D]: Req:0 UID:0 one.vmpool.info invoked , -2, -1, -1, -1
Sat Jul 29 05:35:40 2017 [Z0][ReM][D]: Req:0 UID:0 one.vmpool.info result SUCCESS, "<VM_POOL></VM_POOL>"
Sat Jul 29 05:35:40 2017 [Z0][ReM][D]: Req:0 UID:0 one.vmpool.info invoked , -2, -1, -1, -1
Sat Jul 29 05:35:40 2017 [Z0][ReM][D]: Req:0 UID:0 one.vmpool.info result SUCCESS, "<VM_POOL></VM_POOL>"
Sat Jul 29 05:36:10 2017 [Z0][ReM][D]: Req:0 UID:0 one.zone.raftstatus invoked
Sat Jul 29 05:36:10 2017 [Z0][ReM][D]: Req:0 UID:0 one.zone.raftstatus result SUCCESS, "<RAFT><SERVER_ID>-1<..."
Sat Jul 29 05:36:10 2017 [Z0][ReM][D]: Req:0 UID:0 one.vmpool.info invoked , -2, -1, -1, -1
Sat Jul 29 05:36:10 2017 [Z0][ReM][D]: Req:0 UID:0 one.vmpool.info result SUCCESS, "<VM_POOL></VM_POOL>"
Sat Jul 29 05:36:10 2017 [Z0][ReM][D]: Req:0 UID:0 one.vmpool.info invoked , -2, -1, -1, -1
Sat Jul 29 05:36:10 2017 [Z0][ReM][D]: Req:0 UID:0 one.vmpool.info result SUCCESS, "<VM_POOL></VM_POOL>"
Sat Jul 29 05:36:19 2017 [Z0][MKP][D]: Monitoring marketplace OpenNebula Public (0)
Sat Jul 29 05:36:19 2017 [Z0][InM][D]: Monitoring datastore default (1)
Sat Jul 29 05:36:19 2017 [Z0][InM][D]: Monitoring datastore files (2)
Sat Jul 29 05:36:20 2017 [Z0][ImM][D]: Datastore default (1) successfully monitored.
Sat Jul 29 05:36:20 2017 [Z0][ImM][D]: Datastore files (2) successfully monitored.
Sat Jul 29 05:36:20 2017 [Z0][MKP][D]: Marketplace OpenNebula Public (0) successfully monitored.
Sat Jul 29 05:36:40 2017 [Z0][ReM][D]: Req:0 UID:0 one.zone.raftstatus invoked
Sat Jul 29 05:36:40 2017 [Z0][ReM][D]: Req:0 UID:0 one.zone.raftstatus result SUCCESS, "<RAFT><SERVER_ID>-1<..."
Sat Jul 29 05:36:40 2017 [Z0][ReM][D]: Req:0 UID:0 one.vmpool.info invoked , -2, -1, -1, -1
Sat Jul 29 05:36:40 2017 [Z0][ReM][D]: Req:0 UID:0 one.vmpool.info result SUCCESS, "<VM_POOL></VM_POOL>"
Sat Jul 29 05:37:19 2017 [Z0][MKP][D]: Monitoring marketplace OpenNebula Public (0)
Sat Jul 29 05:37:19 2017 [Z0][InM][D]: Monitoring datastore default (1)
Sat Jul 29 05:37:19 2017 [Z0][InM][D]: Monitoring datastore files (2)
Sat Jul 29 05:37:20 2017 [Z0][ImM][D]: Datastore default (1) successfully monitored.
Sat Jul 29 05:37:20 2017 [Z0][ImM][D]: Datastore files (2) successfully monitored.
Sat Jul 29 05:37:20 2017 [Z0][MKP][D]: Marketplace OpenNebula Public (0) successfully monitored.

but then things go pear shaped:

==> sched.log <==
Sat Jul 29 05:37:40 2017 [Z0][VM][E]: Exception raised: libcurl failed to execute the HTTP POST transaction, explaining:  Operation timed out after 60020 milliseconds with 0 bytes received
Sat Jul 29 05:37:40 2017 [Z0][POOL][E]: Could not retrieve pool info from ONE

==> oned.log <==
Sat Jul 29 05:38:19 2017 [Z0][MKP][D]: Monitoring marketplace OpenNebula Public (0)
Sat Jul 29 05:38:19 2017 [Z0][InM][D]: Monitoring datastore default (1)
Sat Jul 29 05:38:19 2017 [Z0][InM][D]: Monitoring datastore files (2)
Sat Jul 29 05:38:20 2017 [Z0][ImM][D]: Datastore files (2) successfully monitored.
Sat Jul 29 05:38:20 2017 [Z0][ImM][D]: Datastore default (1) successfully monitored.
Sat Jul 29 05:38:20 2017 [Z0][MKP][D]: Marketplace OpenNebula Public (0) successfully monitored.

==> sched.log <==
Sat Jul 29 05:39:10 2017 [Z0][SCHED][E]: Cannot contact oned: libcurl failed to execute the HTTP POST transaction, explaining:  Operation timed out after 60015 milliseconds with 0 bytes received

==> oned.log <==
Sat Jul 29 05:39:19 2017 [Z0][MKP][D]: Monitoring marketplace OpenNebula Public (0)
Sat Jul 29 05:39:19 2017 [Z0][InM][D]: Monitoring datastore default (1)
Sat Jul 29 05:39:19 2017 [Z0][InM][D]: Monitoring datastore files (2)
Sat Jul 29 05:39:20 2017 [Z0][ImM][D]: Datastore default (1) successfully monitored.
Sat Jul 29 05:39:20 2017 [Z0][ImM][D]: Datastore files (2) successfully monitored.
Sat Jul 29 05:39:20 2017 [Z0][MKP][D]: Marketplace OpenNebula Public (0) successfully monitored.
Sat Jul 29 05:40:19 2017 [Z0][MKP][D]: Monitoring marketplace OpenNebula Public (0)
Sat Jul 29 05:40:19 2017 [Z0][InM][D]: Monitoring datastore default (1)
Sat Jul 29 05:40:19 2017 [Z0][InM][D]: Monitoring datastore files (2)
Sat Jul 29 05:40:20 2017 [Z0][ImM][D]: Datastore files (2) successfully monitored.
Sat Jul 29 05:40:20 2017 [Z0][ImM][D]: Datastore default (1) successfully monitored.

oned seems to not be replying to queries from other components.

4

Further experimentation, I tried on my laptop (which uses the standard glibc, rather than musl) and cannot reproduce the problem. So, I’m re-building the VM using glibc instead of musl, see if that fixes it.

musl is more strict on its POSIX complaince than glibc, particularly in regards to threading, which may be what’s tripping up oned.

If this is in fact the cause, then it’s likely people trying to install OpenNebula on Alpine Linux will hit the same problem.

5

Thanks Stuart for the testing and feedback. I’ve just filled an issue to take care of this in an upcoming release

https://dev.opennebula.org/issues/5295