OpenNebula Sunstone: Novnc server does not run if started by Sunstone

Dear Forum People,

I have recently moved our ONE management with its database to a new server.
ONED running, all good on that side.

However, Sunstone is struggling to step into play…

The short version of the problem that I am facing (or what I have gathered I am facing) is that I cannot start Sunstone service as it complains about its dependencies (more specifically NoVNC).

Long version and my findings:

tail /var/log/one/novnc.log

I, [2017-05-10T16:21:13.630096 #9582] INFO – : Starting VNC proxy: python2 /usr/share/one/websockify/websocketproxy.py --target-config=/var/lib/one/sunstone_vnc_tokens --cert /etc/ssl/certs/ng.xx.yz.crt --key /etc/ssl/certs/ng.xx.yz.key.pem 80
WebSocket server settings:

  • Listen on :80
  • Flash security policy server
  • SSL/TLS support
    Traceback (most recent call last):
    File “/usr/share/one/websockify/websocketproxy.py”, line 471, in
    websockify_init()
    File “/usr/share/one/websockify/websocketproxy.py”, line 419, in websockify_init
    server.start_server()
    File “/usr/share/one/websockify/websocket.py”, line 905, in start_server
    tcp_keepintvl=self.tcp_keepintvl)
    File “/usr/share/one/websockify/websocket.py”, line 694, in socket
    sock.bind(addrs[0][4])
    File “/usr/lib64/python2.7/socket.py”, line 224, in meth
    return getattr(self._sock,name)(*args)
    socket.error: [Errno 13] Permission denied
    I, [2017-05-10T16:21:14.638699 #9582] INFO – : Deleting stale lock file
    E, [2017-05-10T16:21:14.638859 #9582] ERROR – : Error starting VNC proxy

Now, “novnc-server start” starts the service if entered on its own however if I try to start Sunstone afterwards, it complains that the socket is already in use - not so surprising.

The permissions error is however is a very curious thing for me, I am running all services as superuser.

To complicate things we have our ONED+SUNSTONE running behind a NGINX server.
I believe this should not be causing the issue as configuration has not changed on NGINX and networking/naming of the new server was done to take the place of our previous OpenNebula deployment.

Sunstone configuration:

/etc/one/sunstone-server.conf
**
:vnc_proxy_port: 80
:vnc_proxy_support_wss: yes
:vnc_proxy_cert: /etc/ssl/certs/ng.xx.yz.crt
:vnc_proxy_key: /etc/ssl/certs/ng.xx.yz.key.pem
:vnc_proxy_ipv6: false
:vnc_request_password: false
**

To show how things are linked up, here is a diagram (hopefully it is clear enough :sweat_smile:):

Thanks in advance!
Erik

###—[Edit1]—
I think my problems might be stemming from installation of a gem called “rack”. When I run the install_gems script, I receive the following:

gem install --no-ri --no-rdoc rack --version “< 2.0.0”
ERROR: Could not find a valid gem ‘rack’ (< 2.0.0), here is why:
Unable to download data from https://rubygems.org/ - Errno::ECONNRESET: Connection reset by peer - SSL_connect (https://rubygems.org/specs.4.8.gz)
Error executing gem install --no-ri --no-rdoc rack --version “< 2.0.0”

###—[Edit2]—

Currently investigating whether our corporate firewall would be causing the issues with the gem installation.

###—[Edit3]—

Successfully edited the “install_gems” script to utilise proxy and managed to install the gems.

Just some thoughts: selinux and apparmor could trigger permission denied error. I believe nginx is running on different server? Otherwise there is same port for websocket and proxy…

Not sure is it helping…

Best,
Anton Todorov

Yes, NGINX is on a different server.
I have selinux disabled and firewalld disabled now.
No change, same error.

Dear Forum People,

I have recently moved our ONE management with its database to a new
server.
ONED running, all good on that side.

However, Sunstone is struggling to step into play…

The short version of the problem that I am facing (or what I have
gathered I am facing) is that I cannot start Sunstone service as it
complains about its dependencies (more specifically NoVNC).

Long version and my findings:

tail /var/log/one/novnc.log

I, [2017-05-10T16:21:13.630096 #9582] INFO – : Starting VNC proxy:
python2 /usr/share/one/websockify/websocketproxy.py
–target-config=/var/lib/one/sunstone_vnc_tokens --cert
/etc/ssl/certs/ng.xx.yz.crt --key /etc/ssl/certs/ng.xx.yz.key.pem 80
WebSocket server settings:

  • Listen on :80
  • Flash security policy server
  • SSL/TLS support
    Traceback (most recent call last):
    File “/usr/share/one/websockify/websocketproxy.py”, line 471, in

    websockify_init()
    File “/usr/share/one/websockify/websocketproxy.py”, line 419, in
    websockify_init
    server.start_server()
    File “/usr/share/one/websockify/websocket.py”, line 905, in
    start_server
    tcp_keepintvl=self.tcp_keepintvl)
    File “/usr/share/one/websockify/websocket.py”, line 694, in socket
    sock.bind(addrs[0][4])
    File “/usr/lib64/python2.7/socket.py”, line 224, in meth
    return getattr(self._sock,name)(*args)
    socket.error: [Errno 13] Permission denied
    I, [2017-05-10T16:21:14.638699 #9582] INFO – : Deleting stale lock
    file
    E, [2017-05-10T16:21:14.638859 #9582] ERROR – : Error starting VNC
    proxy

Now, “novnc-server start” starts the service if entered on its own
however if I try to start Sunstone afterwards, it complains that the
socket is already in use - not so surprising.

Presumably you do that as root.

The permissions error is however is a very curious thing for me, I am
running all services as superuser.

That would be unusual and imprudent. I also suspect it’s not actually
true.

You may be starting them from a root shell, but OpenNebula is packaged
to start all daemons as the user ‘oneadmin’ if you use normal startup
mechanisms (i.e. systemd or init.) Obviously you can change the init
scripts or systemd service definitions to run everything as root, but it
would not surprise me if that caused problems you’d need to fix yourself
in the Ruby and shell parts of OpenNebula.

The error logged is what you get when a non-root user attempts to bind a
TCP listener socket to a port <1024.

To complicate things we have our ONED+SUNSTONE running behind a NGINX
server.
I believe this should not be causing the issue as configuration has
not changed on NGINX and networking/naming of the new server was done
to take the place of our previous OpenNebula deployment.

Sunstone configuration:

/etc/one/sunstone-server.conf
**
:vnc_proxy_port: 80

Like that.

Change vnc_proxy_port to some unassigned port above 1024 (the default
29876 should be fine.) Obviously you’ll also need to change your nginx
proxy config as well to talk to the new port number.

I can confirm that putting Sunstone behind a nginx proxy on a different
machine CAN work, since it is how we set up all of our installations.

1 Like

Good point. I’ve missed that possibility.

Best Regards,
Anton Todorov