Automatically provision and access Oracle Cloud Infrastructure VM serial consoles

Jan-Oracle

We have created a Proof of Concept (POC) how to use the Python SDK to manage access to VM serial consoles in Oracle Cloud Infrastructure (OCI). This POC currently is implemented by creating an Oracle Secure Global Desktop (SGD) application that launches a python script on the SGD server.

Problem Statement

There is a feature for VM shapes provisioned in OCI that allows to create a connection to the serial console of the VM to troubleshoot issues that might have occurred when booting the VM. In order to create this connection access to the OCI Web Console or the API is required. Finally access via ssh can be established to the console. The ssh command to accomplish this is a little bit involved and requires some familiarity with the ssh command options, and even more tweaking when using it from Windows with putty, for example.

$ ssh -o ProxyCommand='ssh -W %h:%p -p 443 ocid1.instanceconsoleconnection.oc1.<a href="mailto:p[email protected]instance-console.us-phoenix-1.oraclecloud.com">p[email protected]instance-console.us-phoenix-1.oraclecloud.com</a>' ocid1.instance.oc1.phx.abyhqljrks2ie4ph25d266gpua7q52j3oaft2uoytc7wr2nojxsum3zezbpq

Use Cases

• Simplify access to Serial Consoles in OCI
• Give users who don't have API credentials access to OCI resources
• Granular Access Control to Serial Consoles for users who do NOT have API access

Proof of Concept

This POC script does the following

• relies on a properly configured OCI SDK

• supports profiles

• lists compartments

• lists VMs in a compartment

• checks if a VM has a Serial Console Connection configured (SC)

• checks if the  SC has been configured through the script and deletes it if it has been created outside the POC

• creates a SC with a generic ssh key known to the POC

• drops the SGD user into the console

Serial Console Access without SGD

In order for a user to access the Serial Console of a VM, valid credentials for the OCI API are required. Either the user goes to the Web UI and creates the SC and then uses the provided ssh command to connect, or does the same via oci-cli

Both ways will require the user to either specify the used ssh key as the default key in ${HOME}/.ssh/id_rsa or modify the command to include the required key, twice.

Serial Console Access with SGD

All a user needs is access to an SGD server configured with this POC. After authentication to the SGD server, the user is offered the typical workspace in the web browser with SGD applications the user is authorized to access

After launching the OCI Console SGD application (name is arbitrary) the user will be presented with a menu to choose the VM to connect to. Either a Serial Console Connection is being created on the fly, or an existing SC is being used and the appropriate ssh command is being launched

All the user needs to know are the credentials for SGD, no OCI API setup needs to be performed on the users system.

Setup Instructions

To setup an SGD server in OCI, please follow my instructions in SGD Linux Install Cookbook.

The easiest way to setup console access with the Python SDK is to install the OCI CLI by installing the rpm and some other packages to be able to more easily upgrade the SDK

# yum install python-oci-cli gcc libffi-devel python-devel openssl-devel

# wget https://bootstrap.pypa.io/get-pip.py

# python get-pip.py

# pip install oci-cli --upgrade

and then run the interactive setup

# oci setup config

Enter a location for your config [/root/.oci/config]: /opt/tarantella/oci/.oci/config

In the POC I opted to store the configuration in /opt/tarantella/oci and changed the ownership that directory to ttasys:ttaserv after successful installation. Once the configuration file has been created in /opt/tarantella/oci/.oci we can test access the API with

[[email protected] ~]$ oci --config-file /opt/tarantella/oci/.oci/config compute instance list

{

  "data": [

    {

      "availability-domain": "BGmc:PHX-AD-1",

      "compartment-id": "ocid1.compartment.oc1..aaaaaaaaisfvfwpvveaswlzhfodpilhadffjbsgudtsxgrc32f2chpizsgta",

      "display-name": "SGD53p2",

      "extended-metadata": {},

      "id": "ocid1.instance.oc1.phx.abyhqljstgq2s2est5fuaar3q3vzaowwyr7wqghpgro2lv6qaa7akzvngqfq",

      "image-id": "ocid1.image.oc1.phx.aaaaaaaari2tnfzvll54rcdzc7g3wvq2bvnftked2c6rcivxt5am4tzbkq4q",

      "ipxe-script": null,

      "lifecycle-state": "RUNNING",

      "metadata": {

        "ssh_authorized_keys": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnYTcEMODqhFLPUETz47bS1nrpwfFK4TmQVRxBXCk5WeP5zDjEBRz+GsxKXJVfJ4n0U/VFyHX5h37vKRGQnl8p/Rgxy9souwYh0ttRGnf+3I9oUXtPsyEJkLWX5oe/+8SawNlOYq/I6C1Pnes1YeHIfc4/LWSvBlxUgLfNoALELLVc3LYN5vB3Jy5Vw+5Z5G6N17KjOGZP2Wxi9E60xQx4OfgTcl9klDPZFXoJ+X9QvIq8meT1CYDgs4lqV2Fp7eIk/vs+cIhUog7+f7TGWA5kfs3k43o+hWNJ2GDiID+Db1e9fkYLFepu0BwEDyyVNjl74pa+gUZxNqGiyXIRSj93 <a class="jive-link-email-small" href="mailto:[email protected]dhcp.oraclecorp.com">[email protected]dhcp.oraclecorp.com</a>\n"

      },

      "region": "phx",

      "shape": "VM.Standard1.1",

      "time-created": "2017-10-11T18:30:23.046000+00:00"

    },

.... snip ....

]

}

[[email protected] ~]$

Since I have configured the /opt/tarantella/oci/.oci/oci_cli_rc it did not ask me for a compartment id.

Now we need to make sure our python script console-access.py always runs as the SGD user ttasys. When a user authenticates to SGD, the application launched will run with the privileges of that user. We want to make sure our python script always runs as the same user (ttasys) so we won't have problems with file permissions. The python script itself will detect that it needs to run as ttasys and will re-launch itself via sudo. For this to work we need to add some configuration to /etc/sudoers. This allows any user authenticated to SGD and authorized to launch the application to do sudo -u ttasys /opt/tarantella/oci/console-connection.py without password, while also preserving environment variables. That is important, because thus we can feed parameters into the python script from the SGD Application definition.

We need to add the following to /etc/sudoers

/etc/sudoers
Cmnd_Alias OCI_CONSOLE=/opt/tarantella/oci/console-connection.py ALL ALL=(ttasys) NOPASSWD:SETENV: OCI_CONSOLE

Now we need to create our generic RSA key to be used for console connections and a directory where our python script stores some information. The generic RSA key is being used to create console connections in OCI. Rather than have the user upload his own public key, we have the script use the same key for any console connection, since we can assume that all connections to consoles will be managed via SGD and this script

[[email protected] ~]$ sudo -u ttasys ssh-keygen -N "" -q -f /opt/tarantella/oci/.oci/oci_console_key [[email protected] ~]$ sudo -u ttasys mkdir /opt/tarantella/oci/info

Once we copy the attached python script to /opt/tarantella/oci we can test it from the command line

[opc@sgd53p2 ~]$ /opt/tarantella/oci/console-connection.py -h usage: console-connection.py [-h] [--verbose] [--nottasys]                              [--compartment_id COMPARTMENT_ID]                              [--instance_id INSTANCE_ID] [--profile PROFILE]                              [--list_compartments] Create Console Connection for OCI VM instance. Commandline options can be specified via environment variables where indicated optional arguments:   -h, --help            show this help message and exit   --verbose, -v         add verbosity to the output, env VERBOSE   --nottasys, -n        do not check for ttasys [FALSE]   --compartment_id COMPARTMENT_ID, -c COMPARTMENT_ID                         specify compartment ID, env COMPARTMENT_ID   --instance_id INSTANCE_ID, -i INSTANCE_ID                         specify instance ID, env INSTANCE_ID   --profile PROFILE, -p PROFILE                         specify config profile to use, env OCI_PROFILE   --list_compartments, -l                         list all compartments and exit [opc@sgd53p2 ~]$

Now we create the SGD application and assign it to users. We can specify any overrides or defaults in the environment variables for the SGD application. In this example we are assuming that the SGD server itself is called "o=appservers/cn=Tarantella server". We call the app "OCI Console JHM Compartment" because we specify the COMPARTMENT_ID of the user jhm. We then add the application to the gamely user profile object

[opc@sgd53p2 ~]$ sudo /opt/tarantella/bin/tarantella object script <<EOF

new_xapp --name "o=applications/cn=OCI Console JHM Compartment" \

  --app /usr/bin/gnome-terminal \

  --args "-e /opt/tarantella/oci/console-connection.py --full-screen --hide-menubar" \

  --appserv "o=appservers/cn=Tarantella server" \

  --displayusing multiplewindows \

  --env "COMPARTMENT_ID='ocid1.compartment.oc1..aaaaaaaaisfvfwpvveaswlzhfodpilhadffjbsgudtsxgrc32f2chpizsgta'" \

  --execution automatic \

  --variablerootsize 1 \

  --icon vt420.gif \

  --keepopen 1 \

  --login unix.exp \

  --method telnet \

  --maxinstances 1 \

  --resumable forever \

  --share 1 \

  --unixaudiopreload 0 \

  --variablerootsize 0 \

<code>  --windowclose notifyclient \

<code><span style="font-family: 'Helvetica Neue', Helvetica, Arial, 'Lucida Grande', sans-serif;">EOF</span>

[opc@sgd53p2 ~]$ sudo /opt/tarantella/bin/tarantella object add_link \

    --name dc=com/dc=oraclevcn/dc=sgdvcn/dc=sub09151850171/cn=gmelo \

    --link "o=applications/cn=OCI Console JHM Compartment"

[opc@sgd53p2 ~]$

Now, when the user gmelo authenticates to SGD, the workspace will present itself like in the screenshot above

The console-connection.py script

#!/bin/env python

#

# https://oracle-cloud-infrastructure-python-sdk.readthedocs.io/en/latest/api/index.html

#

import oci

import sys

import os

import getpass

import argparse

from time import sleep

import subprocess

PK = None

SLEEP_TIME = 2.0

arguments = None

compute = None

identity = None

OCI_CONFIG_ROOT = "/opt/tarantella/oci"

OCI_CONFIG_FILE = "%s/.oci/config" % OCI_CONFIG_ROOT

OCI_CONSOLE_KEY = "%s/.oci/oci_console_key" % OCI_CONFIG_ROOT

#

# read in our public key generated with ssh-keygen -N "" -q -f oci_console_key

#

def readPK():

  pub = open("%s.pub" % OCI_CONSOLE_KEY,"r")

  PK = pub.read()

  if not PK:

    print "no public key in %s" % OCI_CONSOLE_KEY

    enter = raw_input("Hit Enter to continue")

    sys.exit(-1)

  else:

    return PK

def paginate(operation, *args, **kwargs):

    while True:

      try:

        response = operation(*args, **kwargs)

      except oci.exceptions.ServiceError as err:

        print err.message

        break

      else:

        for value in response.data:

            yield value

        kwargs["page"] = response.next_page

        if not response.has_next_page:

          break

        # else:

        #     print "next page"

def list_compartments(config):

  #

  # create identity object

  #

  identity = oci.identity.IdentityClient(config)

  #

  # list compartments in root

  #

  i=1

  compartment = []

  for c in paginate(identity.list_compartments, compartment_id=config["tenancy"],limit=4):

    if arguments.show_ocid:

      print("%-3d %-8s %s [%s]" % (i, c.lifecycle_state,c.name, c.id))

    else:

      print("%-3d %-8s %s" % (i, c.lifecycle_state,c.name))

    compartment.append(c)

    i+=1

  if arguments.list_compartments:

    sys.exit(0)

  else:

    while True:

      i = input("\nWhich compartment (pick number, 0 to exit): ")

      if i <= len(compartment):

        break

    if i>0:

      return compartment[i-1].id

    else:

      return False

def get_image_os(image_id):

  result = compute.get_image(image_id)

  os = "None"

  if result.status and result.data and result.data.operating_system:

    os = result.data.operating_system

  return os

def get_instances():

  instances=[]

  for ins in paginate(compute.list_instances,compartment_id=arguments.compartment_id):

    os = get_image_os(ins.image_id)

    # we need to filter out Windows images

    if ins.shape.startswith('VM.') and os != "Windows":

      instances.append(ins)

  if len(instances)>0:

    print "\nPick one of the instances:\n"

    i=1

    for ins in instances:

        if arguments.show_ocid:

          print "\t%2i) %s (%s) [%s]" % (i,ins.display_name,ins.availability_domain,ins.id)

        else:

          print "\t%2i) %s (%s)" % (i,ins.display_name,ins.availability_domain)

        i+=1

    while True:

      i = input("\nWhich instance (pick number, 0 to exit): ")

      if i <= len(instances):

        break

    if i>0:

      return instances[i-1].id

    else:

      return False

  else:

    return False

if __name__ == '__main__':

  parser = argparse.ArgumentParser(description='Create Console Connection for OCI VM instance. Commandline options can be specified via environment variables where indicated')

  parser.add_argument("--verbose","-v",action="store_true",help="add verbosity to the output, env VERBOSE ")

  parser.add_argument("--nottasys","-n",action="store_true",default=False,help="do not check for ttasys [FALSE]")

  parser.add_argument("--compartment_id","-c",help="specify compartment ID, env COMPARTMENT_ID")

  parser.add_argument("--instance_id","-i",help="specify instance ID, env INSTANCE_ID")

  parser.add_argument("--profile","-p",default="DEFAULT",help="specify config profile to use, env OCI_PROFILE")

  parser.add_argument("--show_ocid","-s",action="store_true",help="show the OCID IDs, env SHOW_OCID")

  parser.add_argument("--list_compartments","-l",action="store_true",help="list all compartments and exit")

  arguments = parser.parse_args()

  #

  # for the next to work the following needs to be added to /etc/sudoers

  #

  # Cmnd_Alias OCI_CONSOLE=/opt/tarantella/oci/console-connection.py

  # ALL ALL=(ttasys) NOPASSWD:SETENV: OCI_CONSOLE

  #

  if not arguments.nottasys and not getpass.getuser() == "ttasys":

    # build arguments for execv

    args = ["sudo","-Eu","ttasys" ]

    args.extend(sys.argv)

    os.execv("/bin/sudo",args)

  #

  # check for environment variables

  #

  if "COMPARTMENT_ID" in os.environ: arguments.compartment_id = os.environ["COMPARTMENT_ID"]

  if "INSTANCE_ID" in os.environ: arguments.instance_id = os.environ["INSTANCE_ID"]

  if "VERBOSE" in os.environ: arguments.verbose = os.environ["VERBOSE"]

  if "SHOW_OCID" in os.environ: arguments.show_ocid = True

  if "OCI_PROFILE" in os.environ: arguments.profile = os.environ["OCI_PROFILE"]

  if "OCI_CONFIG_ROOT" in os.environ:

    OCI_CONFIG_ROOT = os.environ["OCI_CONFIG_ROOT"]

    OCI_CONFIG_FILE = "%s/.oci/config" % OCI_CONFIG_ROOT

    OCI_CONSOLE_KEY = "%s/.oci/oci_console_key" % OCI_CONFIG_ROOT

  if arguments.verbose: print "Running as %s\n" % getpass.getuser()

  #

  # get the required profile configuration

  #

  try:

    config = oci.config.from_file(OCI_CONFIG_FILE, arguments.profile)

  except oci.exceptions.ProfileNotFound as err:

    print err

    enter = raw_input("Hit Enter to continue")

    sys.exit(-1)

  if arguments.list_compartments: list_compartments(config)

  PK = readPK()

  if not arguments.compartment_id:

    arguments.compartment_id = list_compartments(config)

    if not arguments.compartment_id:

      print "\nERROR: you must supply a compartment ID"

      enter = raw_input("Hit Enter to continue")

      sys.exit(-1)

  #

  # get compute object used in subsequent API calls

  #

  compute = oci.core.compute_client.ComputeClient(config)

  if not arguments.instance_id: arguments.instance_id = get_instances()

  if not arguments.instance_id:

    print "\nERROR: you must supply an instance ID"

    enter = raw_input("Hit Enter to continue")

    sys.exit(-1)

  if arguments.verbose: print "get_instance(%s)" % arguments.instance_id

  instance = compute.get_instance(arguments.instance_id)

  if(instance.status):

    instance = instance.data

  else:

    instance = None

    print "The instance with ID %s does not exist" % arguments.instance_id

    enter = raw_input("Hit Enter to continue")

    sys.exit(-1)

  if not instance.shape.startswith("VM."):

    print "Only VM shapes can have consoles, this instance is of shape %s" % instance.shape

    enter = raw_input("Hit Enter to continue")

    sys.exit(-1)

  console = None

  #

  # find active console

  #

  if arguments.verbose: print "list_instance_console_connections"

  for c in paginate(compute.list_instance_console_connections,compartment_id=arguments.compartment_id,instance_id=arguments.instance_id):

    if c.lifecycle_state == "ACTIVE" or c.lifecycle_state == "CREATING":

      if arguments.verbose: print "%-10s %s" % (c.lifecycle_state, c.id)

      console = c

  #

  # now check if the console we found already has a connect script. If it does continue

  # otherwise we need to remove the console and create a new one, cause it most likely

  # was one created manually and doesn't have our Public Key

  #

  if console and console.lifecycle_state == "ACTIVE":

    if not os.path.exists("%s/info/connect.%s.sh" % (OCI_CONFIG_ROOT, console.id)):

      #

      # delete this console

      #

      if arguments.verbose: print "delete_instance_console_connection(%s)" % console.id

      repsonse = compute.delete_instance_console_connection(console.id)

      #

      # now wait for this console to be deleted

      #

      while True:

        response = compute.get_instance_console_connection(console.id)

        if response and response.data and response.data.lifecycle_state == "DELETED":

          break

        print " ... waiting for console to be DELETED (%s)" % response.data.lifecycle_state

        sleep(SLEEP_TIME)

      console = None

  if not console:

    #

    # we have no console, so create one

    #

    if arguments.verbose: print "no console for %s" % arguments.instance_id

    cc = oci.core.models.CreateInstanceConsoleConnectionDetails()

    cc.instance_id = arguments.instance_id

    cc.public_key = PK

    response = compute.create_instance_console_connection(cc)

    if response.status:

      console = response.data

  #

  # wait for console to become active

  #

  while console.lifecycle_state != "ACTIVE":

    response = compute.get_instance_console_connection(console.id)

    print " ... waiting for console connection to become active (%s)" % console.lifecycle_state

    if response.status:

      console = response.data

    else:

      #print dir(response)

      break

    sleep(SLEEP_TIME)

  ssh_fn = "%s/info/connect.%s.sh" % (OCI_CONFIG_ROOT, console.id)

  ssh_err = "%s/info/connect.%s.log" % (OCI_CONFIG_ROOT, console.id)

  ProxyCommand="ProxyCommand='/usr/bin/ssh -e @ -i %s -o LogLevel=QUIET -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -W %%h:%%p -p 443 %[email protected]'" % (OCI_CONSOLE_KEY,console.id)

  if not os.path.exists(ssh_fn):

    try:

      ssh = open(ssh_fn, "w")

      ssh.write("/usr/bin/ssh -e @ -i %s -o LogLevel=QUIET -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o %s %s 2>%s || cat %s\n" % (OCI_CONSOLE_KEY, ProxyCommand, arguments.instance_id, ssh_err, ssh_err))

      ssh.write("echo '';echo 'Press Enter';read ans\n")

      ssh.close()

      #os.chmod(ssh_fn, 0666)

    except IOError:

      print "could not write %s" % ssh_fn

      enter = raw_input("Hit Enter to continue")

      sys.exit(-1)

  print "\n---------------------------------------------------"

  print "| launching ssh connection. terminate by typing @. |"

  print "| hit return to see console login prompt.          |"

  print "----------------------------------------------------"

  try:

    os.execv("/bin/sh",["sh",ssh_fn])

  except:

    print "Failure to launch %s" % ssh_fn

  enter = raw_input("Hit Enter to continue")

  sys.exit(-1)