Slow operations in Sunstone


(Roman) #1

Every actions with VMs (attach disk, poweroff, resume, migrate & etc) takes from 30 to 60 seconds.
Host is not loaded.
Sometimes Sunstone stopes and process migrate to another host without errors.

Logs are clean. Maybe it is the problem with libvirt? But there are no problems with VMs and libvirtd log same clean.

How i can find problem?

Versions of the related components and OS (frontend, hypervisors, VMs):
CentOS 7, KVM, One 5.4, 33VMs, HA_Cluster


(Kristian Feldsam) #2

Hello, are you use mysql, right? Are you tune mysql setings? Like innodb buffer pool size etc…?


(Roman) #3

Hi, yes, I use galera cluster.
All as default. There are no another options.

innodb_autoinc_lock_mode=2
innodb_locks_unsafe_for_binlog=1
query_cache_size=0
query_cache_type=0
default_storage_engine=InnoDB
innodb_log_file_size=100M
innodb_file_per_table
innodb_flush_log_at_trx_commit=2

(Kristian Feldsam) #4

I experience simillar problem - sunstone is slow and problem was with small default innodb buffer, whic is just 128MB and our opennebula DB have about 1GB. You can use tool https://github.com/major/MySQLTuner-perl


(Roman) #5

Yes, I know about this utility. Thank you.
I shall try to tune mysql.


(Kristian Feldsam) #6

Hello, let us know if it helps. If it help, them it should be added to official documentation. Thank you


(Roman) #7

Now I set

[galera]
wsrep_on=ON
binlog_format=ROW
innodb_autoinc_lock_mode=2
innodb_locks_unsafe_for_binlog=1
query_cache_size=0
query_cache_type=0
default_storage_engine=InnoDB
innodb_log_file_size=16M
innodb_file_per_table
innodb_flush_log_at_trx_commit=0
wsrep_provider=/usr/lib64/galera/libgalera_smm.so
#wsrep_cluster_address="gcomm://"
wsrep_cluster_address="gcomm://10.0.10.1,10.0.10.2,10.0.10.3"
wsrep_cluster_name='galera_cluster'
wsrep_node_address='10.0.10.1'
wsrep_node_name='Server-1'
wsrep_sst_method=rsync
wsrep_sst_auth=USER:PASSWORD

tmp_table_size=32M
max_heap_table_size=64MB
thread_cache_size=4
performance_schema = ON
innodb_buffer_pool_instances=1
wsrep_slave_threads=32

Much better, but still not the same.
I will try to improve, and also I will tune NGINX.


(Roman) #8

The problem persists.
After entering sunstone menu loading 30-60s.
All actions with VMs too.

From the cli the same problem with error Net::ReadTimeout

my.cnf

innodb_log_file_size=1G
tmp_table_size=128M
max_heap_table_size=256MB
thread_cache_size=8
performance_schema = ON
innodb_buffer_pool_instances=4
innodb_buffer_pool_size = 4096M
wsrep_slave_threads=64


#9

Hello @GabrielDias,

We know this problem but not with those measured times or that amount of VMs (22).

We’ve tested load times in Sunstone with the old dashboard (yours).
In environments with more than 2000 VMs, the dashboard takes about 15 seconds to load. We have optimized this with a new dashboard in the next release (5.6.0).

it’s strange to me that with 22 machines, it takes almost a minute to load. I think this shouldn’t be for OpenNebula. The problem must reside in your environment probably.

Anyway, in the next release Sunstone’s charging time decreases substantially.

Regards,
Abel.


(Roman) #10

Hi,
Thank you for an answer.

I tried yo update all packages, update kernel, gems.
Try to improved performance of mariadb, but nothing helps me =/

I will be waiting for the new release.


(Kristian Feldsam) #11

Hi, new release is there. Do you analyse mysql process during dashboard loading?

Would you try to run this mysql cmd during slow loading? SHOW FULL PROCESSLIST You should execute it more times to see what happens. Also you should check CPU usage to see what process loads most.


(Roman) #12

Yes, I checked MySQL.

MariaDB> SHOW FULL PROCESSLIST;
+----+-------------+----------------+------+---------+--------+--------------------+-----------------------+----------+
| Id | User        | Host           | db   | Command | Time   | State              | Info                  | Progress |
+----+-------------+----------------+------+---------+--------+--------------------+-----------------------+----------+
| 1  | system user |                | NULL | Sleep   | 457081 | wsrep aborter idle | NULL                  | 0.000    |
| 2  | system user |                | NULL | Sleep   | 25     | committed 52391149 | NULL                  | 0.000    |
| 3  | system user |                | NULL | Sleep   | 6      | committed 52391189 | NULL                  | 0.000    |
| 4  | system user |                | NULL | Sleep   | 13     | committed 52391167 | NULL                  | 0.000    |
| 5  | system user |                | NULL | Sleep   | 25     | committed 52391138 | NULL                  | 0.000    |
| 6  | system user |                | NULL | Sleep   | 13     | committed 52391169 | NULL                  | 0.000    |
| 7  | system user |                | NULL | Sleep   | 25     | committed 52391162 | NULL                  | 0.000    |
| 8  | system user |                | NULL | Sleep   | 13     | committed 52391168 | NULL                  | 0.000    |
| 9  | system user |                | NULL | Sleep   | 13     | committed 52391171 | NULL                  | 0.000    |
| 10 | system user |                | NULL | Sleep   | 21     | committed 52391165 | NULL                  | 0.000    |
| 11 | system user |                | NULL | Sleep   | 25     | committed 52391142 | NULL                  | 0.000    |
| 12 | system user |                | NULL | Sleep   | 13     | committed 52391172 | NULL                  | 0.000    |
| 13 | system user |                | NULL | Sleep   | 25     | committed 52391152 | NULL                  | 0.000    |
| 14 | system user |                | NULL | Sleep   | 13     | committed 52391175 | NULL                  | 0.000    |
| 15 | system user |                | NULL | Sleep   | 12     | committed 52391180 | NULL                  | 0.000    |
| 16 | system user |                | NULL | Sleep   | 25     | committed 52391151 | NULL                  | 0.000    |
| 17 | system user |                | NULL | Sleep   | 13     | committed 52391173 | NULL                  | 0.000    |
| 18 | system user |                | NULL | Sleep   | 0      | committed 52391193 | NULL                  | 0.000    |
| 19 | system user |                | NULL | Sleep   | 25     | committed 52391156 | NULL                  | 0.000    |
| 20 | system user |                | NULL | Sleep   | 25     | committed 52391163 | NULL                  | 0.000    |
| 21 | system user |                | NULL | Sleep   | 6      | committed 52391187 | NULL                  | 0.000    |
| 22 | system user |                | NULL | Sleep   | 25     | committed 52391141 | NULL                  | 0.000    |
| 23 | system user |                | NULL | Sleep   | 25     | committed 52391160 | NULL                  | 0.000    |
| 24 | system user |                | NULL | Sleep   | 25     | committed 52391143 | NULL                  | 0.000    |
| 25 | system user |                | NULL | Sleep   | 25     | committed 52391134 | NULL                  | 0.000    |
| 26 | system user |                | NULL | Sleep   | 13     | committed 52391166 | NULL                  | 0.000    |
| 27 | system user |                | NULL | Sleep   | 25     | committed 52391136 | NULL                  | 0.000    |
| 28 | system user |                | NULL | Sleep   | 6      | committed 52391190 | NULL                  | 0.000    |
| 29 | system user |                | NULL | Sleep   | 25     | committed 52391133 | NULL                  | 0.000    |
| 30 | system user |                | NULL | Sleep   | 25     | committed 52391135 | NULL                  | 0.000    |
| 31 | system user |                | NULL | Sleep   | 25     | committed 52391154 | NULL                  | 0.000    |
| 32 | system user |                | NULL | Sleep   | 25     | committed 52391159 | NULL                  | 0.000    |
| 33 | system user |                | NULL | Sleep   | 11     | committed 52391184 | NULL                  | 0.000    |
| 34 | system user |                | NULL | Sleep   | 12     | committed 52391178 | NULL                  | 0.000    |
| 35 | system user |                | NULL | Sleep   | 25     | committed 52391145 | NULL                  | 0.000    |
| 36 | system user |                | NULL | Sleep   | 25     | committed 52391155 | NULL                  | 0.000    |
| 37 | system user |                | NULL | Sleep   | 13     | committed 52391174 | NULL                  | 0.000    |
| 38 | system user |                | NULL | Sleep   | 12     | committed 52391181 | NULL                  | 0.000    |
| 39 | system user |                | NULL | Sleep   | 25     | committed 52391139 | NULL                  | 0.000    |
| 40 | system user |                | NULL | Sleep   | 10     | committed 52391185 | NULL                  | 0.000    |
| 41 | system user |                | NULL | Sleep   | 25     | committed 52391146 | NULL                  | 0.000    |
| 42 | system user |                | NULL | Sleep   | 45     | committed 52391073 | NULL                  | 0.000    |
| 43 | system user |                | NULL | Sleep   | 0      | committed 52391194 | NULL                  | 0.000    |
| 44 | system user |                | NULL | Sleep   | 12     | committed 52391183 | NULL                  | 0.000    |
| 45 | system user |                | NULL | Sleep   | 12     | committed 52391182 | NULL                  | 0.000    |
| 46 | system user |                | NULL | Sleep   | 13     | committed 52391177 | NULL                  | 0.000    |
| 47 | system user |                | NULL | Sleep   | 6      | committed 52391188 | NULL                  | 0.000    |
| 48 | system user |                | NULL | Sleep   | 25     | committed 52391137 | NULL                  | 0.000    |
| 49 | system user |                | NULL | Sleep   | 25     | committed 52391148 | NULL                  | 0.000    |
| 50 | system user |                | NULL | Sleep   | 25     | committed 52391140 | NULL                  | 0.000    |
| 51 | system user |                | NULL | Sleep   | 25     | committed 52391157 | NULL                  | 0.000    |
| 52 | system user |                | NULL | Sleep   | 10     | committed 52391186 | NULL                  | 0.000    |
| 53 | system user |                | NULL | Sleep   | 21     | committed 52391164 | NULL                  | 0.000    |
| 54 | system user |                | NULL | Sleep   | 25     | committed 52391158 | NULL                  | 0.000    |
| 55 | system user |                | NULL | Sleep   | 25     | committed 52391150 | NULL                  | 0.000    |
| 56 | system user |                | NULL | Sleep   | 12     | committed 52391179 | NULL                  | 0.000    |
| 57 | system user |                | NULL | Sleep   | 25     | committed 52391147 | NULL                  | 0.000    |
| 58 | system user |                | NULL | Sleep   | 13     | committed 52391176 | NULL                  | 0.000    |
| 59 | system user |                | NULL | Sleep   | 25     | committed 52391144 | NULL                  | 0.000    |
| 60 | system user |                | NULL | Sleep   | 6      | committed 52391191 | NULL                  | 0.000    |
| 61 | system user |                | NULL | Sleep   | 6      | committed 52391192 | NULL                  | 0.000    |
| 62 | system user |                | NULL | Sleep   | 45     | committed 52391085 | NULL                  | 0.000    |
| 63 | system user |                | NULL | Sleep   | 13     | committed 52391170 | NULL                  | 0.000    |
| 64 | system user |                | NULL | Sleep   | 25     | committed 52391161 | NULL                  | 0.000    |
| 65 | system user |                | NULL | Sleep   | 25     | committed 52391153 | NULL                  | 0.000    |
| 72 | root        | localhost:9457 | NULL | Sleep   | 53     |                    | NULL                  | 0.000    |
| 73 | root        | localhost:9459 | NULL | Query   | 0      | init               | SHOW FULL PROCESSLIST | 0.000    |
+----+-------------+----------------+------+---------+--------+--------------------+-----------------------+----------+
67 rows in set (0.09 sec)

When VM or Dash loading - there are no any processes with high loading.

How it look like…
Dash
-12-2018%2020-43-07

VMs
-12-2018%2020-46-31

UPD: