Hi,
I created a datastore and it selected /var/lib/one//datastores/118 as the datastore off the Sunstone VM. This I then mounted to a gluster volume off a remote host.
However the datastore doesn’t pick up the change. Anyway to force it to pick up the change in the datastore size without restarting sunstone?
Thx,
TK
Hello @TomK
You need to wait until the datastore is monitored, the time you have to wait is specified in the value of MONITORING_INTERVAL_DATASTORE in /etc/one/oned.conf.
Thanks Alejandro,
Mon Nov 11 03:23:45 2019 : Cannot dispatch VM to any Host. Possible reasons: Not enough capacity in Host or System DS, dispatch limit reached, or limit of free leases reached.
Getting the above. My GlusterFS DS shows a size of -/-. Guessing this is normal as my other system DS’s that are sshfs mounted also show -/-? It’s been a few hours and there’s no change.
However, my GlusterFS lists a size of 0M free. How would this happen when I have 4TB allocated and can write the file to the storage without issue:
[oneadmin@one01 ~]$ onedatastore list
ID NAME SIZE AVAIL CLUSTERS IMAGES TYPE DS TM STAT
120 mdskvm-p02.ni - - 100 0 sys - ssh on
119 mdskvm-p01.ni - - 100 0 sys - ssh on
118 mdskvmgv-c01 0M - 100 0 sys - shared on
117 mdskvm-p02.ni 4T 100% 100 2 img fs ssh on
115 mdskvm-p01.ni 4T 99% 100 2 img fs ssh on
2 files 40G 94% 0 0 fil fs ssh on
1 default 40G 94% 0 0 img fs ssh on
0 system - - 0 0 sys - ssh on
[oneadmin@one01 ~]$ onedatastore show 118
DATASTORE 118 INFORMATION
ID : 118
NAME : mdskvmgv-c01
USER : oneadmin
GROUP : oneadmin
CLUSTERS : 100
TYPE : SYSTEM
DS_MAD : -
TM_MAD : shared
BASE PATH : /var/lib/one//datastores/118
DISK_TYPE : FILE
STATE : READY
DATASTORE CAPACITY
TOTAL: : 0M
FREE: : 0M
USED: : 0M
LIMIT: : -
PERMISSIONS
OWNER : um-
GROUP : u--
OTHER : ---
DATASTORE TEMPLATE
ALLOW_ORPHANS="NO"
BRIDGE_LIST="onebr01"
CLUSTER="100"
DATASTORE_CAPACITY_CHECK="YES"
DISK_TYPE="FILE"
DS_MIGRATE="YES"
NO_DECOMPRESS="YES"
RESTRICTED_DIRS="/"
SAFE_DIRS="/var/lib/one/datastores/118"
SHARED="YES"
TM_MAD="shared"
TYPE="SYSTEM_DS"
IMAGES
[oneadmin@one01 ~]$ cd datastores/
[oneadmin@one01 datastores]$ df -h 118
Filesystem Size Used Avail Use% Mounted on
mdskvm-pc01.nix.mds.xyz:/mdskvmgv-c01 4.0T 41G 4.0T 2% /var/lib/one/datastores/118
[oneadmin@one01 datastores]$Getting the above. My GlusterFS DS shows a size of -/-. Guessing this is normal as my other system DS’s that are sshfs mounted also show -/-? It’s been a few hours and there’s no change.
Yes, it’s completely normal.
As you are using gluster, it’s preferable to have qcow2 as TM_MAD, try to update the TM_MAD and try again.
Also, verify that you don’t have any monitoring errors in your /var/log/one/oned.log.
Same message:
Mon Nov 11 10:35:41 2019 : Cannot dispatch VM to any Host. Possible reasons: Not enough capacity in Host or System DS, dispatch limit reached, or limit of free leases reached.
And yes, you’re completely right. There is an error in oned.log and it has to do with that datastore:
Mon Nov 11 10:34:44 2019 [Z0][ReM][D]: Req:4176 UID:0 one.hostpool.info result SUCCESS, "<HOST_POOL><HOST><ID..."
Mon Nov 11 10:34:45 2019 [Z0][ImM][I]: Command execution failed (exit code: 255): /var/lib/one/remotes/tm/shared/monitor PERTX0RSSVZFUl9BQ1RJT05fREFUQT48REFUQVNUT1JFPjxJRD4xMTg8L0lEPjxVSUQ+MDwvVUlEPjxHSUQ+MDwvR0lEPjxVTkFNRT5vbmVhZG1pbjwvVU5BTUU+PEdOQU1FPm9uZWFkbWluPC9HTkFNRT48TkFNRT5tZHNrdm1ndi1jMDE8L05BTUU+PFBFUk1JU1NJT05TPjxPV05FUl9VPjE8L09XTkVSX1U+PE9XTkVSX00+MTwvT1dORVJfTT48T1dORVJfQT4wPC9PV05FUl9BPjxHUk9VUF9VPjE8L0dST1VQX1U+PEdST1VQX00+MDwvR1JPVVBfTT48R1JPVVBfQT4wPC9HUk9VUF9BPjxPVEhFUl9VPjA8L09USEVSX1U+PE9USEVSX00+MDwvT1RIRVJfTT48T1RIRVJfQT4wPC9PVEhFUl9BPjwvUEVSTUlTU0lPTlM+PERTX01BRD48IVtDREFUQVstXV0+PC9EU19NQUQ+PFRNX01BRD48IVtDREFUQVtzaGFyZWRdXT48L1RNX01BRD48QkFTRV9QQVRIPjwhW0NEQVRBWy92YXIvbGliL29uZS8vZGF0YXN0b3Jlcy8xMThdXT48L0JBU0VfUEFUSD48VFlQRT4xPC9UWVBFPjxESVNLX1RZUEU+MDwvRElTS19UWVBFPjxTVEFURT4wPC9TVEFURT48Q0xVU1RFUlM+PElEPjEwMDwvSUQ+PC9DTFVTVEVSUz48VE9UQUxfTUI+MDwvVE9UQUxfTUI+PEZSRUVfTUI+MDwvRlJFRV9NQj48VVNFRF9NQj4wPC9VU0VEX01CPjxJTUFHRVM+PC9JTUFHRVM+PFRFTVBMQVRFPjxBTExPV19PUlBIQU5TPjwhW0NEQVRBW05PXV0+PC9BTExPV19PUlBIQU5TPjxCUklER0VfTElTVD48IVtDREFUQVtvbmVicjAxXV0+PC9CUklER0VfTElTVD48Q0xVU1RFUj48IVtDREFUQVsxMDBdXT48L0NMVVNURVI+PERBVEFTVE9SRV9DQVBBQ0lUWV9DSEVDSz48IVtDREFUQVtZRVNdXT48L0RBVEFTVE9SRV9DQVBBQ0lUWV9DSEVDSz48RElTS19UWVBFPjwhW0NEQVRBW0ZJTEVdXT48L0RJU0tfVFlQRT48RFNfTUlHUkFURT48IVtDREFUQVtZRVNdXT48L0RTX01JR1JBVEU+PE5PX0RFQ09NUFJFU1M+PCFbQ0RBVEFbWUVTXV0+PC9OT19ERUNPTVBSRVNTPjxSRVNUUklDVEVEX0RJUlM+PCFbQ0RBVEFbL11dPjwvUkVTVFJJQ1RFRF9ESVJTPjxTQUZFX0RJUlM+PCFbQ0RBVEFbL3Zhci9saWIvb25lL2RhdGFzdG9yZXMvMTE4XV0+PC9TQUZFX0RJUlM+PFNIQVJFRD48IVtDREFUQVtZRVNdXT48L1NIQVJFRD48VE1fTUFEPjwhW0NEQVRBW3NoYXJlZF1dPjwvVE1fTUFEPjxUWVBFPjwhW0NEQVRBW1NZU1RFTV9EU11dPjwvVFlQRT48L1RFTVBMQVRFPjwvREFUQVNUT1JFPjxEQVRBU1RPUkVfTE9DQVRJT04+L3Zhci9saWIvb25lLy9kYXRhc3RvcmVzPC9EQVRBU1RPUkVfTE9DQVRJT04+PC9EU19EUklWRVJfQUNUSU9OX0RBVEE+ 118
Mon Nov 11 10:34:45 2019 [Z0][ImM][E]: Error monitoring datastore 118: LQ==. Decoded info: -
Ran above through bash -x. It’s trying to ssh to the BRIDGE_LIST entry?
done
echo "\"]"
done' 'Remote monitor script'
+ MONITOR_DATA='+++ ssh onebr01 bash -s
++ SSH_EXEC_OUT=
++ SSH_EXEC_RC=255
++ '\''['\'' 255 -ne 0 '\'']'\''
++ '\''['\'' -n '\''Remote monitor script'\'' '\'']'\''
[oneadmin@one01 ~]$ ssh onebr01 bash -s
ssh_exchange_identification: Connection closed by remote host
[oneadmin@one01 ~]$
[oneadmin@one01 ~]$ ssh onebr01 bash -s
ssh_exchange_identification: Connection closed by remote host
[oneadmin@one01 ~]$ echo $?
255
[oneadmin@one01 ~]$
I’ll have to dig into it further but please suggest an action if this is familiar to you.
I wanted to avoid it because it’s very painful to extend a qcow2 images and it’s slower. ( If I have a 2TB qcow2 image, I need to make a copy before I can use it as input to extend the source qcow2 disk.
Thx,
TK
Fixed!
Pointed the BRIDGE_LIST to the VIP of the GlusterFS volume:
BRIDGE_LIST mdskvm-pc01.nix.mds.xyz.
The idea is that the monitor will always point to one of the online GlusterFS hosts irrespective if one of them goes down. (untested).