VNC connection fails with "Invalid Protocol"

This document (7000039) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Linux Enterprise Server 11 (SLES 11)
SUSE Linux Enterprise Desktop 11 (SLED 11)
Remote Administration (VNC) enabled.
VNC client or browser being used to establish the VNC connection.

Situation

After connecting through VNC 3 times successfully, and logging out each time, the fourth attempt to connect fails with an "Invalid Protocol" error when using Ultra VNC. The failure will happen with any VNC client but may report a slightly different error. For example, when using vncviewer from another SLED/SLES system the message indicates that the system being connected to is not a VNC server.

The error may go away after some time (15-30 minutes) or a reboot may clear the error.

Resolution

While there are no users logged into the desktop or connected remotely through VNC, stop the X server, remove the files indicated below, then start the X server back up again. Use a SSH session or one of the VTs (Ctrl-Alt-F1 through F6). Log in as the root user and run these commands:

rcxdm stop

rm /tmp/.X*-lock

rm /tmp/.X11-unix/X*

rcxdm start

Test the VNC connection to see if the problem is resolved.

Cause

The /tmp/.X*.lock are the lock files created by the X server when it starts up for the specific display number (the number after the X is the displaynumber).

The /tmp/.X11-unix/X* are the socket files created by the X server when it starts up for the specific display number.

When an instance of the X server is terminated properly it will remove these files, indicating that the display is no longer in use. If for some reason the X server is not terminated properly then these files can be left behind and eventually cause the X server to report that there are no displays available.

The message reported by the VNC client does not give any indication of the problem. The only place to see the display error is in a packet trace.

Additional Information

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.