I am using Oracle VM Manager 3.2.2 and VNC console too. It generally works, but it does not work after several weeks, and then reboot Oracle VM Manager server, it works again, but after several weeks, it does not work again. The same phenomenon as up floor.
If your guest OS does not have a graphic interface, to access your VM console without VNC:
1. find out VM uuid from VM wizard of VM Manager (select the VM, click "edit". It's the field "ID")
2. login to VM server on which the VM is running, and run command:
xm list | grep vm_uuid
remember the domain id (the 2nd column of output)
3. run command
xm console domain_id
and click "enter" two times
4. you can exist by click "ctrl+]"