Automatically provision and access Oracle Cloud Infrastructure VM serial consoles

Version 4

    We have created a Proof of Concept (POC) how to use the Python SDK to manage access to VM serial consoles in Oracle Cloud Infrastructure (OCI). This POC currently is implemented by creating an Oracle Secure Global Desktop (SGD) application that launches a python script on the SGD server.

     

    Problem Statement

     

    There is a feature for VM shapes provisioned in OCI that allows to create a connection to the serial console of the VM to troubleshoot issues that might have occurred when booting the VM. In order to create this connection access to the OCI Web Console or the API is required. Finally access via ssh can be established to the console. The ssh command to accomplish this is a little bit involved and requires some familiarity with the ssh command options, and even more tweaking when using it from Windows with putty, for example.

     

    $ ssh -o ProxyCommand='ssh -W %h:%p -p 443 ocid1.instanceconsoleconnection.oc1.phx.abyhqljrgt3wvfxt765457ew3hpk7mwibnhnyo2rltyv3icfbfahwjmstava@instance-console.us-phoenix-1.oraclecloud.com' ocid1.instance.oc1.phx.abyhqljrks2ie4ph25d266gpua7q52j3oaft2uoytc7wr2nojxsum3zezbpq

     

    Use Cases

     

    • Simplify access to Serial Consoles in OCI
    • Give users who don't have API credentials access to OCI resources
    • Granular Access Control to Serial Consoles for users who do NOT have API access

    Proof of Concept

     

    This POC script does the following

    • relies on a properly configured OCI SDK

    • supports profiles

    • lists compartments

    • lists VMs in a compartment

    • checks if a VM has a Serial Console Connection configured (SC)

    • checks if the  SC has been configured through the script and deletes it if it has been created outside the POC

    • creates a SC with a generic ssh key known to the POC

    • drops the SGD user into the console

    OCI_access_diagram.png

    Serial Console Access without SGD

     

    In order for a user to access the Serial Console of a VM, valid credentials for the OCI API are required. Either the user goes to the Web UI and creates the SC and then uses the provided ssh command to connect, or does the same via oci-cli

    OCI_SC.png

    Both ways will require the user to either specify the used ssh key as the default key in ${HOME}/.ssh/id_rsa or modify the command to include the required key, twice.

     

    Serial Console Access with SGD

     

    All a user needs is access to an SGD server configured with this POC. After authentication to the SGD server, the user is offered the typical workspace in the web browser with SGD applications the user is authorized to access

    OCI_SGD_workspace.png

    After launching the OCI Console SGD application (name is arbitrary) the user will be presented with a menu to choose the VM to connect to. Either a Serial Console Connection is being created on the fly, or an existing SC is being used and the appropriate ssh command is being launched

    OCI_SGD_console.png

     

    All the user needs to know are the credentials for SGD, no OCI API setup needs to be performed on the users system.

     

    Setup Instructions

     

    To setup an SGD server in OCI, please follow my instructions in SGD Linux Install Cookbook.

     

    The easiest way to setup console access with the Python SDK is to install the OCI CLI by installing the rpm and some other packages to be able to more easily upgrade the SDK

     

    # yum install python-oci-cli gcc libffi-devel python-devel openssl-devel

    # wget https://bootstrap.pypa.io/get-pip.py

    # python get-pip.py

    # pip install oci-cli --upgrade

     

    and then run the interactive setup

     

    # oci setup config

    Enter a location for your config [/root/.oci/config]: /opt/tarantella/oci/.oci/config

     

    In the POC I opted to store the configuration in /opt/tarantella/oci and changed the ownership that directory to ttasys:ttaserv after successful installation. Once the configuration file has been created in /opt/tarantella/oci/.oci we can test access the API with

     

    [opc@sgd53p2 ~]$ oci --config-file /opt/tarantella/oci/.oci/config compute instance list

    {

      "data": [

        {

          "availability-domain": "BGmc:PHX-AD-1",

          "compartment-id": "ocid1.compartment.oc1..aaaaaaaaisfvfwpvveaswlzhfodpilhadffjbsgudtsxgrc32f2chpizsgta",

          "display-name": "SGD53p2",

          "extended-metadata": {},

          "id": "ocid1.instance.oc1.phx.abyhqljstgq2s2est5fuaar3q3vzaowwyr7wqghpgro2lv6qaa7akzvngqfq",

          "image-id": "ocid1.image.oc1.phx.aaaaaaaari2tnfzvll54rcdzc7g3wvq2bvnftked2c6rcivxt5am4tzbkq4q",

          "ipxe-script": null,

          "lifecycle-state": "RUNNING",

          "metadata": {

            "ssh_authorized_keys": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnYTcEMODqhFLPUETz47bS1nrpwfFK4TmQVRxBXCk5WeP5zDjEBRz+GsxKXJVfJ4n0U/VFyHX5h37vKRGQnl8p/Rgxy9souwYh0ttRGnf+3I9oUXtPsyEJkLWX5oe/+8SawNlOYq/I6C1Pnes1YeHIfc4/LWSvBlxUgLfNoALELLVc3LYN5vB3Jy5Vw+5Z5G6N17KjOGZP2Wxi9E60xQx4OfgTcl9klDPZFXoJ+X9QvIq8meT1CYDgs4lqV2Fp7eIk/vs+cIhUog7+f7TGWA5kfs3k43o+hWNJ2GDiID+Db1e9fkYLFepu0BwEDyyVNjl74pa+gUZxNqGiyXIRSj93 jmangold@dhcp-whq-5op-9th-and-10th-floor-gen-off-10-211-55-13.usdhcp.oraclecorp.com\n"

          },

          "region": "phx",

          "shape": "VM.Standard1.1",

          "time-created": "2017-10-11T18:30:23.046000+00:00"

        },

    .... snip ....

    ]

    }

    [opc@sgd53p2 ~]$


    Since I have configured the /opt/tarantella/oci/.oci/oci_cli_rc it did not ask me for a compartment id.


    Now we need to make sure our python script console-access.py always runs as the SGD user ttasys. When a user authenticates to SGD, the application launched will run with the privileges of that user. We want to make sure our python script always runs as the same user (ttasys) so we won't have problems with file permissions. The python script itself will detect that it needs to run as ttasys and will re-launch itself via sudo. For this to work we need to add some configuration to /etc/sudoers. This allows any user authenticated to SGD and authorized to launch the application to do sudo -u ttasys /opt/tarantella/oci/console-connection.py without password, while also preserving environment variables. That is important, because thus we can feed parameters into the python script from the SGD Application definition.

     

    We need to add the following to /etc/sudoers

    /etc/sudoers

    Cmnd_Alias OCI_CONSOLE=/opt/tarantella/oci/console-connection.py

    ALL ALL=(ttasys) NOPASSWD:SETENV: OCI_CONSOLE

     

    Now we need to create our generic RSA key to be used for console connections and a directory where our python script stores some information. The generic RSA key is being used to create console connections in OCI. Rather than have the user upload his own public key, we have the script use the same key for any console connection, since we can assume that all connections to consoles will be managed via SGD and this script

     

    [opc@sgd53p2 ~]$ sudo -u ttasys ssh-keygen -N "" -q -f /opt/tarantella/oci/.oci/oci_console_key

    [opc@sgd53p2 ~]$ sudo -u ttasys mkdir /opt/tarantella/oci/info

     

    Once we copy the attached python script to /opt/tarantella/oci we can test it from the command line

     

    [opc@sgd53p2 ~]$ /opt/tarantella/oci/console-connection.py -h

     

    usage: console-connection.py [-h] [--verbose] [--nottasys]

                                 [--compartment_id COMPARTMENT_ID]

                                 [--instance_id INSTANCE_ID] [--profile PROFILE]

                                 [--list_compartments]

     

    Create Console Connection for OCI VM instance. Commandline options can be

    specified via environment variables where indicated

     

    optional arguments:

      -h, --help            show this help message and exit

      --verbose, -v         add verbosity to the output, env VERBOSE

      --nottasys, -n        do not check for ttasys [FALSE]

      --compartment_id COMPARTMENT_ID, -c COMPARTMENT_ID

                            specify compartment ID, env COMPARTMENT_ID

      --instance_id INSTANCE_ID, -i INSTANCE_ID

                            specify instance ID, env INSTANCE_ID

      --profile PROFILE, -p PROFILE

                            specify config profile to use, env OCI_PROFILE

      --list_compartments, -l

                            list all compartments and exit

    [opc@sgd53p2 ~]$

    Now we create the SGD application and assign it to users. We can specify any overrides or defaults in the environment variables for the SGD application. In this example we are assuming that the SGD server itself is called "o=appservers/cn=Tarantella server". We call the app "OCI Console JHM Compartment" because we specify the COMPARTMENT_ID of the user jhm. We then add the application to the gamely user profile object

    [opc@sgd53p2 ~]$ sudo /opt/tarantella/bin/tarantella object script <<EOF
    new_xapp --name "o=applications/cn=OCI Console JHM Compartment" \
      --app /usr/bin/gnome-terminal \
      --args "-e /opt/tarantella/oci/console-connection.py --full-screen --hide-menubar" \
      --appserv "o=appservers/cn=Tarantella server" \
      --displayusing multiplewindows \
      --env "COMPARTMENT_ID='ocid1.compartment.oc1..aaaaaaaaisfvfwpvveaswlzhfodpilhadffjbsgudtsxgrc32f2chpizsgta'" \
      --execution automatic \

      --variablerootsize 1 \

      --icon vt420.gif \

      --keepopen 1 \
      --login unix.exp \
      --method telnet \
      --maxinstances 1 \
      --resumable forever \
      --share 1 \
      --unixaudiopreload 0 \
      --variablerootsize 0 \
      --windowclose notifyclient \
    EOF
    [opc@sgd53p2 ~]$ sudo /opt/tarantella/bin/tarantella object add_link \
        --name dc=com/dc=oraclevcn/dc=sgdvcn/dc=sub09151850171/cn=gmelo \
        --link "o=applications/cn=OCI Console JHM Compartment"
    [opc@sgd53p2 ~]$

     

    Now, when the user gmelo authenticates to SGD, the workspace will present itself like in the screenshot above

     

    The console-connection.py script

     

    #!/bin/env python

    #

    # https://oracle-cloud-infrastructure-python-sdk.readthedocs.io/en/latest/api/index.html

    #

    import oci

    import sys

    import os

    import getpass

    import argparse

    from time import sleep

    import subprocess

     

     

    PK = None

    SLEEP_TIME = 2.0

    arguments = None

    compute = None

    identity = None

    OCI_CONFIG_ROOT = "/opt/tarantella/oci"

    OCI_CONFIG_FILE = "%s/.oci/config" % OCI_CONFIG_ROOT

    OCI_CONSOLE_KEY = "%s/.oci/oci_console_key" % OCI_CONFIG_ROOT

    #

    # read in our public key generated with ssh-keygen -N "" -q -f oci_console_key

    #

    def readPK():

      pub = open("%s.pub" % OCI_CONSOLE_KEY,"r")

      PK = pub.read()

     

     

      if not PK:

        print "no public key in %s" % OCI_CONSOLE_KEY

        enter = raw_input("Hit Enter to continue")

        sys.exit(-1)

      else:

        return PK

     

     

    def paginate(operation, *args, **kwargs):

        while True:

          try:

            response = operation(*args, **kwargs)

          except oci.exceptions.ServiceError as err:

            print err.message

            break

          else:

            for value in response.data:

                yield value

            kwargs["page"] = response.next_page

            if not response.has_next_page:

              break

            # else:

            #     print "next page"

     

     

    def list_compartments(config):

      #

      # create identity object

      #

      identity = oci.identity.IdentityClient(config)

      #

      # list compartments in root

      #

      i=1

      compartment = []

      for c in paginate(identity.list_compartments, compartment_id=config["tenancy"],limit=4):

        if arguments.show_ocid:

          print("%-3d %-8s %s [%s]" % (i, c.lifecycle_state,c.name, c.id))

        else:

          print("%-3d %-8s %s" % (i, c.lifecycle_state,c.name))

        compartment.append(c)

        i+=1

      if arguments.list_compartments:

        sys.exit(0)

      else:

        while True:

          i = input("\nWhich compartment (pick number, 0 to exit): ")

          if i <= len(compartment):

            break

        if i>0:

          return compartment[i-1].id

        else:

          return False

     

     

    def get_image_os(image_id):

      result = compute.get_image(image_id)

      os = "None"

      if result.status and result.data and result.data.operating_system:

        os = result.data.operating_system

      return os

     

     

    def get_instances():

      instances=[]

      for ins in paginate(compute.list_instances,compartment_id=arguments.compartment_id):

        os = get_image_os(ins.image_id)

        # we need to filter out Windows images

        if ins.shape.startswith('VM.') and os != "Windows":

          instances.append(ins)

     

     

      if len(instances)>0:

        print "\nPick one of the instances:\n"

        i=1

        for ins in instances:

            if arguments.show_ocid:

              print "\t%2i) %s (%s) [%s]" % (i,ins.display_name,ins.availability_domain,ins.id)

            else:

              print "\t%2i) %s (%s)" % (i,ins.display_name,ins.availability_domain)

            i+=1

        while True:

          i = input("\nWhich instance (pick number, 0 to exit): ")

          if i <= len(instances):

            break

        if i>0:

          return instances[i-1].id

        else:

          return False

      else:

        return False

     

     

    if __name__ == '__main__':

      parser = argparse.ArgumentParser(description='Create Console Connection for OCI VM instance. Commandline options can be specified via environment variables where indicated')

      parser.add_argument("--verbose","-v",action="store_true",help="add verbosity to the output, env VERBOSE ")

      parser.add_argument("--nottasys","-n",action="store_true",default=False,help="do not check for ttasys [FALSE]")

      parser.add_argument("--compartment_id","-c",help="specify compartment ID, env COMPARTMENT_ID")

      parser.add_argument("--instance_id","-i",help="specify instance ID, env INSTANCE_ID")

      parser.add_argument("--profile","-p",default="DEFAULT",help="specify config profile to use, env OCI_PROFILE")

      parser.add_argument("--show_ocid","-s",action="store_true",help="show the OCID IDs, env SHOW_OCID")

      parser.add_argument("--list_compartments","-l",action="store_true",help="list all compartments and exit")

      arguments = parser.parse_args()

      #

      # for the next to work the following needs to be added to /etc/sudoers

      #

      # Cmnd_Alias OCI_CONSOLE=/opt/tarantella/oci/console-connection.py

      # ALL ALL=(ttasys) NOPASSWD:SETENV: OCI_CONSOLE

      #

      if not arguments.nottasys and not getpass.getuser() == "ttasys":

        # build arguments for execv

        args = ["sudo","-Eu","ttasys" ]

        args.extend(sys.argv)

        os.execv("/bin/sudo",args)

      #

      # check for environment variables

      #

      if "COMPARTMENT_ID" in os.environ: arguments.compartment_id = os.environ["COMPARTMENT_ID"]

      if "INSTANCE_ID" in os.environ: arguments.instance_id = os.environ["INSTANCE_ID"]

      if "VERBOSE" in os.environ: arguments.verbose = os.environ["VERBOSE"]

      if "SHOW_OCID" in os.environ: arguments.show_ocid = True

      if "OCI_PROFILE" in os.environ: arguments.profile = os.environ["OCI_PROFILE"]

      if "OCI_CONFIG_ROOT" in os.environ:

        OCI_CONFIG_ROOT = os.environ["OCI_CONFIG_ROOT"]

        OCI_CONFIG_FILE = "%s/.oci/config" % OCI_CONFIG_ROOT

        OCI_CONSOLE_KEY = "%s/.oci/oci_console_key" % OCI_CONFIG_ROOT

     

     

      if arguments.verbose: print "Running as %s\n" % getpass.getuser()

      #

      # get the required profile configuration

      #

      try:

        config = oci.config.from_file(OCI_CONFIG_FILE, arguments.profile)

      except oci.exceptions.ProfileNotFound as err:

        print err

        enter = raw_input("Hit Enter to continue")

        sys.exit(-1)

     

     

      if arguments.list_compartments: list_compartments(config)

     

     

      PK = readPK()

     

     

      if not arguments.compartment_id:

        arguments.compartment_id = list_compartments(config)

        if not arguments.compartment_id:

          print "\nERROR: you must supply a compartment ID"

          enter = raw_input("Hit Enter to continue")

          sys.exit(-1)

      #

      # get compute object used in subsequent API calls

      #

      compute = oci.core.compute_client.ComputeClient(config)

     

     

      if not arguments.instance_id: arguments.instance_id = get_instances()

      if not arguments.instance_id:

        print "\nERROR: you must supply an instance ID"

        enter = raw_input("Hit Enter to continue")

        sys.exit(-1)

     

     

      if arguments.verbose: print "get_instance(%s)" % arguments.instance_id

      instance = compute.get_instance(arguments.instance_id)

      if(instance.status):

        instance = instance.data

      else:

        instance = None

        print "The instance with ID %s does not exist" % arguments.instance_id

        enter = raw_input("Hit Enter to continue")

        sys.exit(-1)

      if not instance.shape.startswith("VM."):

        print "Only VM shapes can have consoles, this instance is of shape %s" % instance.shape

        enter = raw_input("Hit Enter to continue")

        sys.exit(-1)

      console = None

      #

      # find active console

      #

      if arguments.verbose: print "list_instance_console_connections"

      for c in paginate(compute.list_instance_console_connections,compartment_id=arguments.compartment_id,instance_id=arguments.instance_id):

        if c.lifecycle_state == "ACTIVE" or c.lifecycle_state == "CREATING":

          if arguments.verbose: print "%-10s %s" % (c.lifecycle_state, c.id)

          console = c

      #

      # now check if the console we found already has a connect script. If it does continue

      # otherwise we need to remove the console and create a new one, cause it most likely

      # was one created manually and doesn't have our Public Key

      #

      if console and console.lifecycle_state == "ACTIVE":

        if not os.path.exists("%s/info/connect.%s.sh" % (OCI_CONFIG_ROOT, console.id)):

          #

          # delete this console

          #

          if arguments.verbose: print "delete_instance_console_connection(%s)" % console.id

          repsonse = compute.delete_instance_console_connection(console.id)

          #

          # now wait for this console to be deleted

          #

          while True:

            response = compute.get_instance_console_connection(console.id)

            if response and response.data and response.data.lifecycle_state == "DELETED":

              break

            print " ... waiting for console to be DELETED (%s)" % response.data.lifecycle_state

            sleep(SLEEP_TIME)

          console = None

     

     

      if not console:

        #

        # we have no console, so create one

        #

        if arguments.verbose: print "no console for %s" % arguments.instance_id

        cc = oci.core.models.CreateInstanceConsoleConnectionDetails()

        cc.instance_id = arguments.instance_id

        cc.public_key = PK

        response = compute.create_instance_console_connection(cc)

        if response.status:

          console = response.data

      #

      # wait for console to become active

      #

      while console.lifecycle_state != "ACTIVE":

        response = compute.get_instance_console_connection(console.id)

        print " ... waiting for console connection to become active (%s)" % console.lifecycle_state

        if response.status:

          console = response.data

        else:

          #print dir(response)

          break

        sleep(SLEEP_TIME)

     

     

      ssh_fn = "%s/info/connect.%s.sh" % (OCI_CONFIG_ROOT, console.id)

      ssh_err = "%s/info/connect.%s.log" % (OCI_CONFIG_ROOT, console.id)

      ProxyCommand="ProxyCommand='/usr/bin/ssh -e @ -i %s -o LogLevel=QUIET -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -W %%h:%%p -p 443 %s@instance-console.us-phoenix-1.oraclecloud.com'" % (OCI_CONSOLE_KEY,console.id)

      if not os.path.exists(ssh_fn):

        try:

          ssh = open(ssh_fn, "w")

          ssh.write("/usr/bin/ssh -e @ -i %s -o LogLevel=QUIET -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o %s %s 2>%s || cat %s\n" % (OCI_CONSOLE_KEY, ProxyCommand, arguments.instance_id, ssh_err, ssh_err))

          ssh.write("echo '';echo 'Press Enter';read ans\n")

          ssh.close()

          #os.chmod(ssh_fn, 0666)

        except IOError:

          print "could not write %s" % ssh_fn

          enter = raw_input("Hit Enter to continue")

          sys.exit(-1)

     

     

      print "\n---------------------------------------------------"

      print "| launching ssh connection. terminate by typing @. |"

      print "| hit return to see console login prompt.          |"

      print "----------------------------------------------------"

     

     

      try:

        os.execv("/bin/sh",["sh",ssh_fn])

      except:

        print "Failure to launch %s" % ssh_fn

      enter = raw_input("Hit Enter to continue")

      sys.exit(-1)