You can add users to the VM Manager but they will all have administrative access. There is no such thing as role based access with the VM Manager. You maybe able to hook your VM Manager to Enterprise Manager and create some type of custom access. Never tried it but might be possible.
If you're doing some type of Oracle Linux installation, then you could do a VNC installation and give this person access to the vnc session to run the install.... or you could configure your VNC server with a custom username and set that user to use a custom VNC port for local access. I wouldn't necessarily call it a "console session" but its pretty close.