Troubleshooting a VNC Remote Session Failure

SUSE Linux Enterprise Server 11
SUSE Linux Enterprise Server 11 SP1
SUSE Linux Enterprise Desktop 11
SUSE Linux Enterprise Desktop 11 SP1

Attempting to remotely access the system through either a VNC client, or a web browser (http) results in a failed connection, gray screen, or black screen.

Troubleshooting steps for VNC connection when "Remote Administration (VNC)" has been enabled in YaST.

These steps are not intended for troubleshooting a VNC server that is being started outside of the YaST configuration.

The instructions below will require being logged in as the root user at a command line terminal.

Disable the Firewall
By default the firewall is enabled and will block the VNC communication unless the box was checked in the YaST configuration for Remote Administration to open the port.

Even if it was checked it's always a good idea to disable the firewall to rule it out.  Very quick and easy to do.

To verify the status use this command:

rcSuSEfirewall2 status

This will return the status of the firewall as either "unused" or "running" (the status shows up on the very right hand side of the terminal).  If it is "running" then shut down the firewall with this command:

rcSuSEfirewall2 stop

Run the status command again to verify it is now "unused".

Test the VNC connection again to see if there is any change.  If not, leave the firewall disabled until troubleshooting is complete.


Listening Port
Use the following command to verify that port 5901 is in a listening state and is owned by xinetd:

netstat -lpdn | grep 5901

The following should be displayed (other than a different PID which is fine):

tcp  0  0 0.0.0.0:5901  0.0.0.0:*  LISTEN   3351/xinetd

If this information is not displayed then it would indicate that the "Remote Administration (VNC)" selection in YaST has not been configured or the the xinetd service has been disabled (which can also be checked in YaST).  Both selections can be found in the"Network Services" section of YaST.


Displaymanager
Using your favorite editor open the following file:

/etc/sysconfig/displaymanager

In this file look for the following entries:

DISPLAYMANAGER="gdm"
DISPLAYMANAGER_REMOTE_ACCESS="yes"
DISPLAYMANAGER_ROOT_LOGIN_REMOTE="yes"

The "DISPLAYMANAGER=" setting will normally be set to either gdm (for GNOME) or kdm4 (for KDE4).  GNOME/gdm is the default unless KDE4 was specifically selected as the desktop during installation.  This is important to know so that the appropriate instructions can be followed in the section below for either gdm or kdm4.

DISPLAYMANAGER_REMOTE_ACCESS must be enabled to allow the connection.

DISPLAYMANAGER_ROOT_LOGIN_REMOTE must be enabled to allow the root user to login remotely.

If any changes are made to the file then run "SuSEconfig" after.


gdm as the Displaymanager
There are two configuration files that must checked for gdm, /etc/gdm/custom.conf and /etc/gdm/gdm.schemas.

/etc/gdm/custom.conf
This is a very small file and only these two settings need to be verified and set to true:

# SuSEconfig: displaymanager:DISPLAYMANAGER_REMOTE_ACCESS
Enable=true

# SuSEconfig: displaymanager:DISPLAYMANAGER_ROOT_LOGIN_REMOTE,DISPLAYMANAGER_SHUTDOWN, security:PERMISSION_SECURITY
AllowRemoteRoot=true

The second setting only needs to be true if the root user will be allowed to login remotely.


/etc/gdm/gdm.schemas
This is a much larger file but there are only two settins that need to be verified in this one as well.  Search for the following entries and set to true:

<schema>
    <key>xdmcp/Enable</key>
    <signature>b</signature>
    <default>true</default>
</schema>

<schema>
    <key>security/AllowRoot</key>
    <signature>b</signature>
    <default>true</default>
</schema>

Again, the second entry is only needed if root will be allowed to login remotely.


Restarting the Graphics
If any changes are made to either of these files then the graphics system must be restarted to take effect.  A reboot will do the job but may be overkill if you are on a production server.  Restarting the graphics manually will kill anyone currently logged into the graphic desktop (local or remote) so use caution.  Services being provided by the server are generally not disrupted by restarting the graphics with the following commands:

rcxdm stop

Then verify there are no X server processes running with this command (note: The X is a capital letter):

ps -e | grep X

If any processes are displayed they will need to be killed by running this command (again use caution as this will pull the rug out from under anyone that may be logged into the graphic desktop on the system):

killall X

Run the command "ps -e | grep X" again to verify all X processes are dead.  If there are any left try killing them with this command:

killall -9 X

Use the command "ps -e | grep X" again.  If there are still X processes running then a system reboot will be required.


kdm4 as the Displaymanager
kdm4 uses one configuration file, "/usr/share/kde4/config/kdm/kdmrc".

In the configuration file look for the following entries:

Under the [Xdmcp] section:
Enable=true

Under the [X-*-Core] section (way down the list):
AllowRootLogin=true

The default settings look like this:
#Enable=false
#AllowRootLogin=false
If they are still default, remove the comment (#) and change to true.

Of course, the second entry only needs to be true if root will be allowed to login remotely.

To have any changes recognized follow the instructions given in the gdm section for restarting the graphics.


xinetd
Enabling the "Remote Administration (VNC)" in YaST turns on the xinetd service and enables the entry for VNC and for the http VNC access.  The configuration for starting the VNC service is kept in the file "/etc/xinetd.d/vnc".  Edit this file and look for the following entries:

service vnc1
{
   socket_type     = stream
   protocol        = tcp
   wait            = no
   user            = nobody
   server          = /usr/bin/Xvnc
   server_args     = -noreset -inetd -once -query localhost -geometry 1280x800 -depth 16
   type            = UNLISTED
   port            = 5901

The "server_args" is where there may be a problem.  If there is an entry for "-from localhost" remove it.  Remove only the "-from localhost" and leave the other parameters as they are.

service vnchttpd1
{
   socket_type     = stream
   protocol        = tcp
   wait            = no
   user            = nobody
   server          = /usr/bin/vnc_inetd_httpd
   server_args     = 1024 768 5901
   type            = UNLISTED
   port            = 5801

Again, looking at the "server_args", if there is an entry for "-noreset" remove it.  Remove only the "-noreset" and leave the other parameters as they are.

If any changes have been made to this file run the following command to restart xinetd and have the changes recognized:

rcxinetd restart


Conclusion
After making these changes try the connection through VNC again.  Hopefully you now have better results.

If you normally have the firewall running you can start it back up with this command:

rcSuSEfirewall2 start

After starting the firewall verify that your VNC connection still works.  If it fails then you will need to modify the allowed ports in your firewall exceptions (TCP 5901 for a connection from a VNC client and TCP 5801 for access from a web browser through http).