6 Replies Latest reply: Apr 9, 2014 10:09 PM by Avi Miller-Oracle RSS

    Make uln-yum-mirror more flexible

    micheloe

      Hi,

       

      Looking at the uln-yum-mirror script, can we please do something to make it more future-proof? Every now and then a new channel gets added and then the code that interprets the channel names into directories fails to recognise this. I understand you cannot always anticipate what happens in the future, but the way things are being done now looks like a kludge to me, to be honest. For example, with the release of UEKR3 the path translation goes wrong and we end up with the UEKR3 in REP_BASE/REP_OL/x86_64/latest/UEKR3 instead of REP_BASE/REP_OL/UEKR3/latest/x86_64. I'm talking about this particular part from the script:

       

      # convert the channel label into a path suitable for our yum conventions

      DIR=`echo $LABEL | sed -e 's/x86_64/x86-64/g' \

            -e 's/oracle_addons/oracle-addons/g' -e 's/UEK_/UEK-/g' \

            -e 's/Dtrace_/Dtrace-/g' -e 's/gdm_/gdm-/g' | \

            sed 's/_u\([0-9][0-9]*\)_/_\1_/' | awk -F _ '{if($1!="exadata") \

            {printf("%s/", toupper($1))} else {printf("%s/", $1)};count=1;\

            while (++count<NF){if(count==NF-1) {print $(count+1)"/"$(count)} \

            else{printf("%s/", $(count))}}}' | sed -e 's/x86-64/x86_64/g' \

            -e 's/oracle-addons/oracle_addons/g' -e 's/UEK-/UEK\//g' \

            -e 's/Dtrace-/Dtrace/g' -e 's/gdm-/gdm_/g'`

       

      As I understand it, we like the following things to be done:

       

      • extract the architecture and move it to the last part of the directory
      • Translate most (but not all) prefixes (el5, ol6, et.c) to uppercase
      • Keep some of the underscores in the path names (instead of making it another sub directory)

       

      If we make all of these things configurable through /etc/sysconfig/uln-yum-mirror, we get the added benefit that when should things change again in the future, we now at least have some control over it. I rewrote this part with the use of bash extensions (see diff below), but it could have easily been done without if really necessary. With this rewrite, the variables ARCHS, MAIN_KEEP and REP_KEEP could be moved to /etc/sysconfig/uln-yum-mirror to make it more flexible (so you don't have to alter the script every single time something changes). It is just an idea, but it would be a whole lot less of a PITA in case UEKR4 gets released... ;-) I'm happy to create something that does not rely on bash features if you like. You can get the diff here: http://filebin.ca/1CrloxnQpHQp/uln-yum-mirror.diff

       

      Regards,

      Michel

       

      --- uln-yum-mirror.orig 2014-01-21 23:26:26.000000000 +0100

      +++ uln-yum-mirror      2014-02-18 12:41:20.172595280 +0100

      @@ -142,17 +142,37 @@

       

       

         echo "## BEGIN PROCESSING $LABEL ##"

         # convert the channel label into a path suitable for our yum conventions

      -  DIR=`echo $LABEL | sed -e 's/x86_64/x86-64/g' \

      -        -e 's/oracle_addons/oracle-addons/g' -e 's/UEK_/UEK-/g' \

      -        -e 's/Dtrace_/Dtrace-/g' -e 's/gdm_/gdm-/g' | \

      -        sed 's/_u\([0-9][0-9]*\)_/_\1_/' | awk -F _ '{if($1!="exadata") \

      -        {printf("%s/", toupper($1))} else {printf("%s/", $1)};count=1;\

      -        while (++count<NF){if(count==NF-1) {print $(count+1)"/"$(count)} \

      -        else{printf("%s/", $(count))}}}' | sed -e 's/x86-64/x86_64/g' \

      -        -e 's/oracle-addons/oracle_addons/g' -e 's/UEK-/UEK\//g' \

      -        -e 's/Dtrace-/Dtrace/g' -e 's/gdm-/gdm_/g'`

       

       

         # determine the correct yum repository directory

      +  ARCHS="i386 x86_64 ia64"

      +  MAIN_KEEP="exadata"

      +  REP_KEEP="oracle_addons Dtrace_ gdm_"

      +  # Find arch and remove it from label

      +  for i in $ARCHS; do

      +    if [[ $LABEL =~ ${i} ]]; then

      +      REP_ARCH=${i}

      +      LABEL_NOARCH=${LABEL/${i}_/}

      +    fi

      +  done

      +  echo "arch: $REP_ARCH ; label: $LABEL_NOARCH"

      +  REP_MAIN=${LABEL_NOARCH%%_*}

      +  # Make uppercase if not present in MAIN_KEEP

      +  if ! { for i in $MAIN_KEEP; do [[ $REP_MAIN = "${i}" ]] && break; done }; then

      +    REP_MAIN=${REP_MAIN^^}

      +  fi

      +  # Get the remainder of the label without the main name

      +  REP_REMAIN=${LABEL_NOARCH#*_}

      +  # Escape sublabels containing _ with - to prevent substitution

      +  for l in $REP_KEEP; do

      +    REP_REMAIN=${REP_REMAIN/${l}/${l/_/-}}

      +  done

      +  REP_REMAIN=${REP_REMAIN//_/\/}

      +  # Unescape sublabels

      +  for l in $REP_KEEP; do

      +    REP_REMAIN=${REP_REMAIN/${l/_/-}/${l}}

      +  done

      +  DIR=$REP_MAIN/$REP_REMAIN/$REP_ARCH

      +

         # set REP_DIR for further use in the repo file creation

         REP_DIR=$REP_UNK/$DIR

         echo $LABEL | grep '^exadata*_' >/dev/null 2>&1 && \

        • 1. Re: Make uln-yum-mirror more flexible
          Avi Miller-Oracle

          Hey, thanks for this. I'll send this to the developer for review.

          • 2. Re: Make uln-yum-mirror more flexible
            micheloe

            Hi Avi, thanks. After posting the message I encountered an incompatibility with Oracle Enterprise Linux 5. I've created a new patch that can be obtained from here: http://filebin.ca/1ET8Y3APnuJH/uln-yum-mirror.diff. It turns out some of the bash replacement functions I used don't work with bash 3.2, neither does it support case modification. See below for the full patch.

             

            Regards,

            Michel

             

            --- uln-yum-mirror.orig2014-02-20 15:13:58.000000000 +0100
            +++ uln-yum-mirror2014-02-20 15:40:49.000000000 +0100

            @@ -143,15 +143,36 @@

             

             

               echo "## BEGIN PROCESSING $LABEL ##"

               # convert the channel label into a path suitable for our yum conventions

            -  DIR=`echo $LABEL | sed -e 's/x86_64/x86-64/g' \

            -        -e 's/oracle_addons/oracle-addons/g' -e 's/UEK_/UEK-/g' \

            -        -e 's/Dtrace_/Dtrace-/g' -e 's/gdm_/gdm-/g' | \

            -        sed 's/_u\([0-9][0-9]*\)_/_\1_/' | awk -F _ '{if($1!="exadata") \

            -        {printf("%s/", toupper($1))} else {printf("%s/", $1)};count=1;\

            -        while (++count<NF){if(count==NF-1) {print $(count+1)"/"$(count)} \

            -        else{printf("%s/", $(count))}}}' | sed -e 's/x86-64/x86_64/g' \

            -        -e 's/oracle-addons/oracle_addons/g' -e 's/UEK-/UEK\//g' \

            -        -e 's/Dtrace-/Dtrace/g' -e 's/gdm-/gdm_/g'`

            +  ARCHS="i386 x86_64 ia64"

            +  MAIN_KEEP="exadata"

            +  REP_KEEP="oracle_addons Dtrace_ gdm_"

            +  # Find arch and remove it from label

            +  for i in $ARCHS; do

            +    if [[ $LABEL =~ ${i} ]]; then

            +      REP_ARCH=${i}

            +      LABEL_NOARCH=${LABEL/${i}_/}

            +    fi

            +  done

            +  REP_MAIN=${LABEL_NOARCH%%_*}

            +  # Make uppercase if not present in MAIN_KEEP

            +  if ! { for i in $MAIN_KEEP; do [[ $REP_MAIN = "${i}" ]] && break; done }; then

            +    REP_MAIN=$(echo ${REP_MAIN} | tr '[:lower:]' '[:upper:]')

            +  fi

            +  # Get the remainder of the label without the main name

            +  REP_REMAIN=${LABEL_NOARCH#*_}

            +  # Escape sublabels containing _ with - to prevent substitution

            +  for l in $REP_KEEP; do

            +    SUB=${l/_/-}

            +    REP_REMAIN=${REP_REMAIN/${l}/${SUB}}

            +  done

            +  REP_REMAIN=${REP_REMAIN//_/\/}

            +  # Unescape sublabels

            +  for l in $REP_KEEP; do

            +    SUB=${l/_/-}

            +    REP_REMAIN=${REP_REMAIN/${SUB}/${l}}

            +  done

            +  DIR=$REP_MAIN/$REP_REMAIN/$REP_ARCH

            +

             

             

               # determine the correct yum repository directory

               # set REP_DIR for further use in the repo file creation

            • 3. Re: Make uln-yum-mirror more flexible
              Avi Miller-Oracle

              I'm hoping to test this code out next week. I've also asked another developer to review it. BTW, is the diff created against uln-yum-mirror-0.2.0-1, i.e. the latest released version?

              • 4. Re: Make uln-yum-mirror more flexible
                micheloe

                Hi Avi. Sorry, but that patch was against an older version. I've upgraded locally and created a new diff compatible with 0.2.0-1: http://filebin.ca/1EYPFCCNh2wl/uln-yum-mirror.diff

                I've also fixed indenting to match the original code. Please let me know your findings once you're done testing. Any issues or comments I'd be happy to address.

                 

                --- uln-yum-mirror.orig 2014-03-06 08:56:17.422678613 +0100

                +++ uln-yum-mirror      2014-03-06 09:05:35.648098278 +0100

                @@ -143,15 +143,35 @@

                 

                 

                   echo "## BEGIN PROCESSING $LABEL ##"

                   # convert the channel label into a path suitable for our yum conventions

                -  DIR=`echo $LABEL | sed -e 's/x86_64/x86-64/g' \

                -        -e 's/oracle_addons/oracle-addons/g' -e 's/UEK_/UEK-/g' \

                -        -e 's/Dtrace_/Dtrace-/g' -e 's/gdm_/gdm-/g' | \

                -        sed 's/_u\([0-9][0-9]*\)_/_\1_/' | awk -F _ '{if($1!="exadata") \

                -        {printf("%s/", toupper($1))} else {printf("%s/", $1)};count=1;\

                -        while (++count<NF){if(count==NF-1) {print $(count+1)"/"$(count)} \

                -        else{printf("%s/", $(count))}}}' | sed -e 's/x86-64/x86_64/g' \

                -        -e 's/oracle-addons/oracle_addons/g' -e 's/UEK-/UEK\//g' \

                -        -e 's/Dtrace-/Dtrace/g' -e 's/gdm-/gdm_/g'`

                +  ARCHS="i386 x86_64 ia64"

                +  MAIN_KEEP="exadata"

                +  REP_KEEP="oracle_addons Dtrace_ gdm_"

                +  # Find arch and remove it from label

                +  for i in $ARCHS; do

                +      if [[ $LABEL =~ ${i} ]]; then

                +          REP_ARCH=${i}

                +          LABEL_NOARCH=${LABEL/${i}_/}

                +      fi

                +  done

                +  REP_MAIN=${LABEL_NOARCH%%_*}

                +  # Make uppercase if not present in MAIN_KEEP

                +  if ! { for i in $MAIN_KEEP; do [[ $REP_MAIN = "${i}" ]] && break; done }; then

                +      REP_MAIN=$(echo ${REP_MAIN} | tr '[:lower:]' '[:upper:]')

                +  fi

                +  # Get the remainder of the label without the main name

                +  REP_REMAIN=${LABEL_NOARCH#*_}

                +  # Escape sublabels containing _ with - to prevent substitution

                +  for l in $REP_KEEP; do

                +      SUB=${l/_/-}

                +      REP_REMAIN=${REP_REMAIN/${l}/${SUB}}

                +  done

                +  REP_REMAIN=${REP_REMAIN//_/\/}

                +  # Unescape sublabels

                +  for l in $REP_KEEP; do

                +      SUB=${l/_/-}

                +      REP_REMAIN=${REP_REMAIN/${SUB}/${l}}

                +  done

                +  DIR=$REP_MAIN/$REP_REMAIN/$REP_ARCH

                 

                 

                   # determine the correct yum repository directory

                   # set REP_DIR for further use in the repo file creation

                • 5. Re: Make uln-yum-mirror more flexible
                  Avi Miller-Oracle

                  Groovy, thanks. Have this on my TODO list for next Tuesday in Australia (Monday is a public holiday!).

                  • 6. Re: Make uln-yum-mirror more flexible
                    Avi Miller-Oracle

                    Sorry, this dropped off my radar for a bit. I'm hoping to take a look at it soon.