1 2 Previous Next 26 Replies Latest reply on Jun 3, 2016 5:15 AM by Jason Brincat Go to original post
      • 15. Re: adop phase=cutover / ORA-20010: ERROR: Unable to acquire lock on ad_adop_sessions table
        Jason Brincat

        From session 4 which is the latest cutover that failed:

         

        [UNEXPECTED]Error occurred running "perl /u03/ebt1/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPValidations.pl  -contextfile=/u03/ebt1/fs1/inst/apps/EBT1_erp1/appl/admin/EBT1_erp1.xml -patchctxfile=/u03/ebt1/fs2/inst/apps/EBT1_erp1/appl/admin/EBT1_erp1.xml -phase=cutover -logloc=/u03/ebt1/fs_ne/EBSapps/log/adop/4/cutover_20160505_234153/EBT1_erp1 -promptmsg=hide"

            [UNEXPECTED]Error 1 occurred while Executing txkADOPValidation script on erp1

         

        [ERROR]: Patch file system WebLogic domain edit lock is enabled.

        ValidatePortSync:

         

        ACTION PLAN

        ----------------------

        Login to the wls admin url

        Release the lock via the left hand side button

        Note:the lock is held on the patch file system admin server so you will need to shutdown the admin service and other services on the run file system.

        Start the admin server on the patch file using using the forcepatchfs command

        See the following note for more details:

        Adop Prepare Phase Failed On TxkADOPValidation ERROR: Domain is not in edit mode. Validating /config/edit.lok file in patch file system (Doc ID 1920804.1)

        Once complete re-run cutover

         

        regards

        Jason

        • 16. Re: adop phase=cutover / ORA-20010: ERROR: Unable to acquire lock on ad_adop_sessions table
          Jason Brincat

          Any luck with my suggestions?

          if it worked please do mark this as resolved

           

          best regards

          Jason

          • 18. Re: adop phase=cutover / ORA-20010: ERROR: Unable to acquire lock on ad_adop_sessions table
            user11952526

            Hi Jason,

             

            I am getting the below error...but when I check file system got changed...

             

            [applebt1@erp1][/home/applebt1]
            $ . ./EBSapps.env run

              E-Business Suite Environment Information
              ----------------------------------------
              RUN File System           : /u03/ebt1/fs2/EBSapps/appl
              PATCH File System         : /u03/ebt1/fs1/EBSapps/appl
              Non-Editioned File System : /u03/ebt1/fs_ne


              DB Host: erp1.ebs.com  Service/SID: EBT1


              Sourcing the RUN File System ...

            [applebt1@erp1][/home/applebt1]

             

            Error :

             

             

            ===========================================================================

            ADOP (C.Delta.7)

            Session ID: 4

            Node: erp1

            Phase: cutover

            Log: /u03/ebt1/fs_ne/EBSapps/log/adop/4/adop_20160512_231105.log

            ===========================================================================

             

            Validating configuration on node: [erp1].

                Log: /u03/ebt1/fs_ne/EBSapps/log/adop/4/cutover_20160512_231105/EBT1_erp1

             

            Checking if finalize required.

             

            Verifying data dictionary.

             

            Running finalize phase database actions.

                Log: @ADZDSHOWLOG.sql "2016/05/12 23:11:56"

                Compiling invalid objects.

             

            Submitting request for Internal Concurrent Manager shutdown.

             

            You are running adcmctl.sh version 120.19.12020000.7

             

            Shutting down concurrent managers for EBT1 ...

            ORACLE Password:

            Submitted request 460358 for CONCURRENT FND SHUTDOWN

             

             

            1. adcmctl.sh: exiting with status 0

             

             

            1. adcmctl.sh: check the logfile /u03/ebt1/fs1/inst/apps/EBT1_erp1/logs/appl/admin/log/adcmctl.txt for more information ...

             

                Cancelling ADZDPATCH concurrent request

             

            Waiting for Internal Concurrent Manager to go down.

             

            Stopping patch edition Admin Server.

             

            You are running adadminsrvctl.sh version 120.10.12020000.10

             

             

            The AdminServer is already shutdown

             

            AdminServer logs are located at /u03/ebt1/fs2/FMW_Home/user_projects/domains/EBS_domain_EBT1/servers/AdminServer/logs

             

            1. adadminsrvctl.sh: exiting with status 2

             

            1. adadminsrvctl.sh: check the logfile /u03/ebt1/fs2/inst/apps/EBT1_erp1/logs/appl/admin/log/adadminsrvctl.txt for more information ...

             

             

            Stopping patch edition Node Manager.

             

            You are running adnodemgrctl.sh version 120.11.12020000.12

             

             

            The Node Manager is already shutdown

             

            NodeManager log is located at /u03/ebt1/fs2/FMW_Home/wlserver_10.3/common/nodemanager/nmHome1

             

            1. adnodemgrctl.sh: exiting with status 2

             

            1. adnodemgrctl.sh: check the logfile /u03/ebt1/fs2/inst/apps/EBT1_erp1/logs/appl/admin/log/adnodemgrctl.txt for more information ...

             

             

            Stopping the application tier services on current node.

                Log: /u03/ebt1/fs_ne/EBSapps/log/adop/4/cutover_20160512_231105/EBT1_erp1

             

            Stopping run edition Admin Server.

             

            You are running adadminsrvctl.sh version 120.10.12020000.10

             

            #

            # A fatal error has been detected by the Java Runtime Environment:

            #

            #  SIGSEGV (0xb) at pc=0x00007f71abc63fcf, pid=9929, tid=1083554112

            #

            # JRE version: 7.0_25-b15

            # Java VM: Java HotSpot(TM) 64-Bit Server VM (23.25-b01 mixed mode linux-amd64 compressed oops)

            # Problematic frame:

            # V  [libjvm.so+0x7a0fcf]  oopDesc* PSPromotionManager::copy_to_survivor_space<false>(oopDesc*)+0x12f

            #

            # Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again

            #

            # An error report file with more information is saved as:

            # /home/applebt1/hs_err_pid9929.log

            #

            # If you would like to submit a bug report, please visit:

            #   http://bugreport.sun.com/bugreport/crash.jsp

            #

            Stopping WLS Admin Server...

            Refer /u03/ebt1/fs1/inst/apps/EBT1_erp1/logs/appl/admin/log/adadminsrvctl.txt for details

             

            AdminServer logs are located at /u03/ebt1/fs1/FMW_Home/user_projects/domains/EBS_domain_EBT1/servers/AdminServer/logs

             

            1. adadminsrvctl.sh: exiting with status 0

             

            1. adadminsrvctl.sh: check the logfile /u03/ebt1/fs1/inst/apps/EBT1_erp1/logs/appl/admin/log/adadminsrvctl.txt for more information ...

             

             

            Stopping run edition Node Manager.

             

            You are running adnodemgrctl.sh version 120.11.12020000.12

             

             

            NodeManager log is located at /u03/ebt1/fs1/FMW_Home/wlserver_10.3/common/nodemanager/nmHome1

             

            1. adnodemgrctl.sh: exiting with status 0

             

            1. adnodemgrctl.sh: check the logfile /u03/ebt1/fs1/inst/apps/EBT1_erp1/logs/appl/admin/log/adnodemgrctl.txt for more information ...

             

             

            Running stored cutover DDLs.

                Log: /u03/ebt1/fs_ne/EBSapps/log/adop/4/cutover_20160512_231105/EBT1_erp1/log/cutover.log

             

            Loading JAR files into database.

                No JAR files found to load

             

            Performing database cutover.

                Log: @ADZDSHOWLOG.sql "2016/05/12 23:32:07"

             

            Flipping snapshots in run and patch editions.

             

            Performing file system cutover.

             

            Checking if file system cutover completed.

             

            Broadcast message from applebt1 (Thu May 12 23:33:03 2016):

             

            EBT1 environment has changed.

            All users must re-source the environment.

             

            Starting run edition Admin Server.

             

            Starting application tier services.

            Executing service control script:

            /u03/ebt1/fs2/inst/apps/EBT1_erp1/admin/scripts/adapcctl.sh start

            Timeout specified in context file: 100 second(s)

             

            script returned:

            ****************************************************

             

            You are running adapcctl.sh version 120.0.12020000.6

             

            Starting OPMN managed Oracle HTTP Server (OHS) instance ...

             

            1. adapcctl.sh: exiting with status 204

             

            1. adapcctl.sh: check the logfile /u03/ebt1/fs2/inst/apps/EBT1_erp1/logs/appl/admin/log/adapcctl.txt for more information ...

             

             

            .end std out.

             

            .end err out.

             

            ****************************************************

             

             

             

            *******FATAL ERROR*******

            PROGRAM : (/u03/ebt1/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPCutOverPhaseCtrlScript.pl)

            TIME    : Thu May 12 23:37:09 2016

            FUNCTION: main::forceStartupServices [ Level 1 ]

            ERRORMSG: 1

                [UNEXPECTED]Error occurred running "perl /u03/ebt1/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPCutOverPhaseCtrlScript.pl -

            contextfile=/u03/ebt1/fs1/inst/apps/EBT1_erp1/appl/admin/EBT1_erp1.xml -patchcontextfile=/u03/ebt1/fs2/inst/apps/EBT1_erp1/appl/admin/EBT1_erp1.xml -promptmsg=hide -console=off -sessionid=4 -

            timestamp=20160512_231105 -outdir=/u03/ebt1/fs_ne/EBSapps/log/adop/4/cutover_20160512_231105/EBT1_erp1 -action=forcestartup"

                [UNEXPECTED]Cutover phase has failed.

             

             

            [STATEMENT] Please run adopscanlog utility, using the command

             

            "adopscanlog -latest=yes"

             

            to get the list of the log files along with snippet of the error message corresponding to each log file.

             

             

            adop exiting with status = 1 (Fail)

            You have new mail in /var/spool/mail/applebt1

            [applebt1@erp1][/home/applebt1]

             

             

             

             

             

            $ adopscanlog -latest=yes

             

            Scanning /u03/ebt1/fs_ne/EBSapps/log/adop/4/ directory ...

             

            /u03/ebt1/fs_ne/EBSapps/log/adop/4/adop_20160512_231105.log:

            ------------------------------------------------------------

             

            Lines #(506-510):

            [EVENT]     Starting application tier services.

                [PROCEDURE] [START 2016/05/12 23:34:33] Starting Middle Tier Services

                [UNEXPECTED]Error occurred running "perl /u03/ebt1/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPCutOverPhaseCtrlScript.pl -

            contextfile=/u03/ebt1/fs1/inst/apps/EBT1_erp1/appl/admin/EBT1_erp1.xml -patchcontextfile=/u03/ebt1/fs2/inst/apps/EBT1_erp1/appl/admin/EBT1_erp1.xml -promptmsg=hide -console=off -sessionid=4 -

            timestamp=20160512_231105 -outdir=/u03/ebt1/fs_ne/EBSapps/log/adop/4/cutover_20160512_231105/EBT1_erp1 -action=forcestartup"

                [UNEXPECTED]Cutover phase has failed.

                [PROCEDURE] [START 2016/05/12 23:37:09] Updating session timestamps

             

            Lines #(507-511):

                [PROCEDURE] [START 2016/05/12 23:34:33] Starting Middle Tier Services

                [UNEXPECTED]Error occurred running "perl /u03/ebt1/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPCutOverPhaseCtrlScript.pl -

            contextfile=/u03/ebt1/fs1/inst/apps/EBT1_erp1/appl/admin/EBT1_erp1.xml -patchcontextfile=/u03/ebt1/fs2/inst/apps/EBT1_erp1/appl/admin/EBT1_erp1.xml -promptmsg=hide -console=off -sessionid=4 -

            timestamp=20160512_231105 -outdir=/u03/ebt1/fs_ne/EBSapps/log/adop/4/cutover_20160512_231105/EBT1_erp1 -action=forcestartup"

                [UNEXPECTED]Cutover phase has failed.

                [PROCEDURE] [START 2016/05/12 23:37:09] Updating session timestamps

                [STATEMENT] CUTOVER Phase END TIME: 12-05-2016 23:37:10

             

            /u03/ebt1/fs_ne/EBSapps/log/adop/4/cutover_20160512_231105/EBT1_erp1/TXK_CTRL_forcestartup/txkADOPCutOverPhaseCtrlScript.log:

            -----------------------------------------------------------------------------------------------------------------------------

             

            Lines #(73-75):

            TIME    : Thu May 12 23:37:09 2016

            FUNCTION: main::forceStartupServices [ Level 1 ]

            ERRORMSG: 1[applebt1@erp1][/home/applebt1]

            $

             

            Regards.

            John.

            • 19. Re: adop phase=cutover / ORA-20010: ERROR: Unable to acquire lock on ad_adop_sessions table
              user11952526

              Hi Jason,

               

              getting this from adop -status...

               

              [applebt1@erp1][/u03/ebt1/fs2/inst/apps/EBT1_erp1/admin/scripts]
              $ adop -status


              Enter the APPS password:
              Connected.

              ==============================================================
              ADOP (C.Delta.7)
              Session Id: 4
              Command: status
              Output: /u03/ebt1/fs_ne/EBSapps/log/status_20160513_000441/adzdshowstatus.out
              ===============================================================

              Node Name       Node Type  Phase           Status          Started              Finished             Elapsed
              --------------- ---------- --------------- --------------- -------------------- -------------------- ------------
              erp1            master     PREPARE         COMPLETED       2016/04/24 17:30:05  2016/05/01 15:47:22  166:17:17
                                         APPLY           COMPLETED
                                         FINALIZE        COMPLETED       2016/05/12 23:11:56  2016/05/12 23:12:15  0:00:19
                                         CUTOVER         FAILED          2016/05/12 23:11:43  2016/05/12 23:37:10  0:25:27
                                         CLEANUP         NOT STARTED

               

              File System Synchronization Type: Full

               

               

              adop exiting with status = 0 (Success)
              [applebt1@erp1][/u03/ebt1/fs2/inst/apps/EBT1_erp1/admin/scripts]

               

              Regards.

              JOhn

              • 20. Re: adop phase=cutover / ORA-20010: ERROR: Unable to acquire lock on ad_adop_sessions table
                Jason Brincat

                Hi John,

                Looks like services are failing to start at the end of cutover

                At the end of cutover we switch file systems - this has happened

                Then right at the very end we start services on the new run file system. This is what has failed.

                 

                lets get 2 things.

                1/ run the diagnostics again and upload the results

                2/ use this script from a note I created when I was at oracle and send the log files. This gathers log files from the applications technology stack

                12.2 Ebusiness Suite - Collecting Fusion Middleware Log Files Note 1362900.1


                regards

                Jason

                • 21. Re: adop phase=cutover / ORA-20010: ERROR: Unable to acquire lock on ad_adop_sessions table
                  user11952526

                  Hi Jason,

                   

                  I have uploaded the below files.

                   

                  diagnostic_info_erp1.tar.bz2

                  erp1.ebs.com_051316.1402_OPMN_OHS.zip

                  erp1.ebs.com_051316.1403_FMW.zip

                  mzAppsLogFiles_erp1.ebs.com_051316.zip

                   

                  Regards.

                  John.

                  • 22. Re: adop phase=cutover / ORA-20010: ERROR: Unable to acquire lock on ad_adop_sessions table
                    Jason Brincat

                    AdminServer Error

                    May 12, 2016 11:55:39 PM ClientCommunicatorAdmin restart

                    WARNING: Failed to restart: java.rmi.NoSuchObjectException: The object identified by: '284' could not be found.  Either it was has not been exported or it has been collected by the distributed garbage collector.

                     

                    Error

                    --> Process (index=1,uid=1458447087,pid=12955)

                      failed to start a managed process after the maximum retry limit

                      Log:

                      /u03/ebt1/fs2/FMW_Home/webtier/instances/EBS_web_EBT1_OHS1/diagnostics/logs/OHS/EBS_web_EBT1/console~OHS~1.log

                     

                    OHS

                    --------

                    16/05/12 23:34:46 Start process

                    --------

                    /u03/ebt1/fs2/FMW_Home/webtier/ohs/bin/apachectl startssl: execing httpd

                    ModSecurity: WARNING Using transformations in SecDefaultAction is deprecated (/u03/ebt1/fs2/FMW_Home/webtier/instances/EBS_web_EBT1_OHS1/config/OHS/EBS_web_EBT1/security2.conf:62).

                    ModSecurity: WARNING Using transformations in SecDefaultAction is deprecated (/u03/ebt1/fs2/FMW_Home/webtier/instances/EBS_web_EBT1_OHS1/config/OHS/EBS_web_EBT1/security2.conf:84).

                    [Thu May 12 23:34:46 2016] [warn] Errors will be logged into /u03/ebt1/fs2/FMW_Home/webtier/instances/EBS_web_EBT1_OHS1/diagnostics/logs/OHS/EBS_web_EBT1/EBS_web_EBT1.log

                     

                     

                    [2016-05-12T23:34:46.4300+05:30] [OHS] [INCIDENT_ERROR:32] [OHS-9999] [core.c] [host_id: erp1.ebs.com] [host_addr: 192.168.0.10] [pid: 12552] [tid: 140726662326064] [user: applebt1] [VirtualHost: main] (98)Address already in use:  make_sock: could not bind to address 0.0.0.0:8010

                     

                    ACTION PLAN

                    ===========

                     

                    Looks like we have some port bidning issues.

                    Shutdown all application tier services

                    Make sure no appplmgr processes are running on the fs1 or fs2 file system - this is VERY important

                    e.g. run

                    ps -ef | grep applmgr

                    (assumes applmgr is the owner of the applications tier file system)

                     

                     

                    Now source the run file system and re-run cutover

                     

                    regards

                    Jason

                    • 23. Re: adop phase=cutover / ORA-20010: ERROR: Unable to acquire lock on ad_adop_sessions table
                      user11952526

                      Hi Jason,

                       

                      Thanks a lot!

                      My issue is fixed.

                       

                      But git into new issue...while running adop phase=fs_clone, getting the below error...

                       

                       

                      Checking for existing adop sessions.
                          No pending session exists.
                          Starting new adop session.

                      ===========================================================================
                      ADOP (C.Delta.7)
                      Session ID: 5
                      Node: erp1
                      Phase: fs_clone
                      Log: /u03/ebt1/fs_ne/EBSapps/log/adop/5/adop_20160516_114223.log
                      ===========================================================================

                      Validating configuration on node: [erp1].
                          Log: /u03/ebt1/fs_ne/EBSapps/log/adop/5/fs_clone_20160516_114223/EBT1_erp1
                              [WARNING]: There could be issues while validating the ports used for E-Business Suite instance against ports used in /etc/services. Refer the log file for more details.

                      Checking for pending cleanup actions.
                          Running cleanup as it was not run in previous patching cycle.

                      Processing cleanup actions in parallel.
                          Log: /u03/ebt1/fs_ne/EBSapps/log/adop/5/fs_clone_20160516_114223/EBT1_erp1/log/cleanup.log

                      AutoPatch error:
                      adptaskStoreTimInfoToHist: Error while inserting into AD_PROGRAM_RUN_TASK_JOBS.

                      AutoPatch error:
                      ORA-01578: ORACLE data block corrupted (file # 395, block # 110633)
                      ORA-01110: data file 395: '/u03/ebt1/data/a_int01.dbf'


                      AutoPatch error:
                      adpmrp: Error calling adptaskStoreTimInfoToHist(2).


                      You should check the file
                      /u03/ebt1/fs_ne/EBSapps/log/adop/5/fs_clone_20160516_114223/EBT1_erp1/log/cleanup.log

                      for errors.

                          [UNEXPECTED]Error occurred in CLEANUP.

                          [UNEXPECTED]Cleanup completed with errors or warnings. Review log files
                          Stopping services on patch file system.
                          Stopping admin server.

                      You are running adadminsrvctl.sh version 120.10.12020000.10


                      The AdminServer is already shutdown

                      AdminServer logs are located at /u03/ebt1/fs1/FMW_Home/user_projects/domains/EBS_domain_EBT1/servers/AdminServer/logs

                      adadminsrvctl.sh: exiting with status 2

                      adadminsrvctl.sh: check the logfile /u03/ebt1/fs1/inst/apps/EBT1_erp1/logs/appl/admin/log/adadminsrvctl.txt for more information ... 

                          Stopping node manager.

                      You are running adnodemgrctl.sh version 120.11.12020000.12


                      The Node Manager is already shutdown

                      NodeManager log is located at /u03/ebt1/fs1/FMW_Home/wlserver_10.3/common/nodemanager/nmHome1

                      adnodemgrctl.sh: exiting with status 2

                      adnodemgrctl.sh: check the logfile /u03/ebt1/fs1/inst/apps/EBT1_erp1/logs/appl/admin/log/adnodemgrctl.txt for more information ... 

                          [UNEXPECTED]fs_clone has failed.


                      [STATEMENT] Please run adopscanlog utility, using the command

                      "adopscanlog -latest=yes"

                      to get the list of the log files along with snippet of the error message corresponding to each log file.


                      adop exiting with status = 2 (Fail)
                      [applebt1@erp1][/home/applebt1]
                      $

                       

                      It seems database block corruption... shall I start a new discussion for database block cirruption ?

                       

                      Mon May 16 11:44:33 2016

                      Hex dump of (file 395, block 110633) in trace file /u03/ebt1/11.2.0/admin/EBT1_erp1/diag/rdbms/ebt1/EBT1/trace/EBT1_ora_1

                      8440.trc

                      Corrupt block relative dba: 0x62c1b029 (file 395, block 110633)

                      Bad check value found during multiblock buffer read

                      Data in bad block:

                      type: 6 format: 2 rdba: 0x62c1b029

                      last change scn: 0x056c.de301ed5 seq: 0x1 flg: 0x06

                      spare1: 0x0 spare2: 0x0 spare3: 0x0

                      consistency value in tail: 0x1ed50601

                      check value in block header: 0x502a

                      computed block checksum: 0x1000

                      Reading datafile '/u03/ebt1/data/a_int01.dbf' for corruption at rdba: 0x62c1b029 (file 395, block 110633)

                      Reread (file 395, block 110633) found same corrupt data (no logical check)

                      Mon May 16 11:44:33 2016

                      Corrupt Block Found

                               TSN = 384, TSNAME = APPS_TS_INTERFACE

                               RFN = 395, BLK = 110633, RDBA = 1656860713

                               OBJN = 22606, OBJD = 258366, OBJECT = AD_TASK_TIMING, SUBOBJECT =

                               SEGMENT OWNER = APPLSYS, SEGMENT TYPE = Table Segment

                      Errors in file /u03/ebt1/11.2.0/admin/EBT1_erp1/diag/rdbms/ebt1/EBT1/trace/EBT1_ora_18440.trc  (incident=88537):

                      ORA-01578: ORACLE data block corrupted (file # 395, block # 110633)

                      ORA-01110: data file 395: '/u03/ebt1/data/a_int01.dbf'

                      Incident details in: /u03/ebt1/11.2.0/admin/EBT1_erp1/diag/rdbms/ebt1/EBT1/incident/incdir_88537/EBT1_ora_18440_i88537.tr

                      c

                      Mon May 16 11:47:24 2016

                      Non critical error ORA-48913 caught while writing to trace file "/u03/ebt1/11.2.0/admin/EBT1_erp1/diag/rdbms/ebt1/EBT1/in

                      cident/incdir_88537/EBT1_ora_18440_i88537.trc"

                      Error message: ORA-48913: Writing into trace file failed, file size limit [10485760] reached

                      Writing to the above trace file is disabled for now on...

                       

                      Regards.

                      John.

                      • 24. Re: adop phase=cutover / ORA-20010: ERROR: Unable to acquire lock on ad_adop_sessions table
                        Jason Brincat

                        Yes, please start a new thread. It may even pay to post it to a DBA forum as consider the block corruption a seperate issue completely.

                        Once that is fixed, then come back and re-run fs_clone. If that fails when the block corruption is fixed then please post a qn to this forum again

                         

                        Glad I could be of assistance

                         

                        regards

                        Jason

                        • 25. Re: adop phase=cutover / ORA-20010: ERROR: Unable to acquire lock on ad_adop_sessions table
                          user11952526

                          Hi Jason,

                           

                          Block corruption got fixed.

                          fs_clone also completed successfully.

                          Thanks for the help and support...

                           

                          Regards

                          John.

                          • 26. Re: adop phase=cutover / ORA-20010: ERROR: Unable to acquire lock on ad_adop_sessions table
                            Jason Brincat

                            Brilliant John,

                            thank you for letting me know. Glad that is all fixed now

                             

                            if you have time to make some comments as useful or solution provided that would be appreciated

                             

                            best regards

                            Jason

                            1 2 Previous Next