0 Replies Latest reply on Aug 18, 2015 9:54 AM by 2893577

    adop fs_clone failing on 2nd node during ports validation

    2893577

      Hi,

       

      We are running fs_clone on a 2 node Apps tier (R12.2.4). It is failing on 2nd node during ports validation:

       

      [apthdr2i@vmohshdrc024 <RUN> ]$ adopscanlog -latest=yes

       

      Scanning /thdr2i/applmgr/fs_ne/EBSapps/log/adop/80/ directory ...

       

      /thdr2i/applmgr/fs_ne/EBSapps/log/adop/80/adop_20150818_030521.log:

      -------------------------------------------------------------------

       

      Lines #(128-132):

          Executing "txkADOPValidation" action in all nodes

           Remote execution results xml output file = /thdr2i/applmgr/fs_ne/EBSapps/log/adop/80/fs_clone_20150818_030521/THDR2I_vmohshdrc023/remote_execution_result_20150818_030824.xml

          [ERROR]     Node: "vmohshdrc024"  Status: "failed"

          [ERROR]     Remote execution failed on Node: "vmohshdrc024".

          [PROCEDURE] [START 2015/08/18 03:10:26] Checking if adop can continue with available nodes

       

      Lines #(129-133):

           Remote execution results xml output file = /thdr2i/applmgr/fs_ne/EBSapps/log/adop/80/fs_clone_20150818_030521/THDR2I_vmohshdrc023/remote_execution_result_20150818_030824.xml

          [ERROR]     Node: "vmohshdrc024"  Status: "failed"

          [ERROR]     Remote execution failed on Node: "vmohshdrc024".

          [PROCEDURE] [START 2015/08/18 03:10:26] Checking if adop can continue with available nodes

          [PROCEDURE] Calling: /thdr2i/applmgr/fs2/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPEvalSrvStatus.pl

       

      Lines #(133-137):

          [PROCEDURE] Calling: /thdr2i/applmgr/fs2/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPEvalSrvStatus.pl

          [EVENT]     Log: /thdr2i/applmgr/fs_ne/EBSapps/log/adop/80/fs_clone_20150818_030521/THDR2I_vmohshdrc023

          [UNEXPECTED]Error occurred while executing "perl /thdr2i/applmgr/fs2/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPEvalSrvStatus.pl -contextfile=/thdr2i/inst/fs2/inst/apps/THDR2I_vmohshdrc023/appl/admin/THDR2I_vmohshdrc023.xml -patchcontextfile=/thdr2i/inst/fs1/inst/apps/THDR2I_vmohshdrc023/appl/admin/THDR2I_vmohshdrc023.xml -phase=fs_clone -promptmsg=hide -nodelist=vmohshdrc023 -console=off -sessionid=80 -timestamp=20150818_030521 -outdir=/thdr2i/applmgr/fs_ne/EBSapps/log/adop/80/fs_clone_20150818_030521/THDR2I_vmohshdrc023"

          [UNEXPECTED]Error occurred while Checking if adop can continue with available nodes using command: "perl /thdr2i/applmgr/fs2/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPEvalSrvStatus.pl -contextfile=/thdr2i/inst/fs2/inst/apps/THDR2I_vmohshdrc023/appl/admin/THDR2I_vmohshdrc023.xml -patchcontextfile=/thdr2i/inst/fs1/inst/apps/THDR2I_vmohshdrc023/appl/admin/THDR2I_vmohshdrc023.xml -phase=fs_clone -promptmsg=hide -nodelist=vmohshdrc023 -console=off -sessionid=80 -timestamp=20150818_030521 -outdir=/thdr2i/applmgr/fs_ne/EBSapps/log/adop/80/fs_clone_20150818_030521/THDR2I_vmohshdrc023".

          Previous tasks have failed or are incomplete on nodes: ",vmohshdrc024"

       

      Lines #(134-138):

          [EVENT]     Log: /thdr2i/applmgr/fs_ne/EBSapps/log/adop/80/fs_clone_20150818_030521/THDR2I_vmohshdrc023

          [UNEXPECTED]Error occurred while executing "perl /thdr2i/applmgr/fs2/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPEvalSrvStatus.pl -contextfile=/thdr2i/inst/fs2/inst/apps/THDR2I_vmohshdrc023/appl/admin/THDR2I_vmohshdrc023.xml -patchcontextfile=/thdr2i/inst/fs1/inst/apps/THDR2I_vmohshdrc023/appl/admin/THDR2I_vmohshdrc023.xml -phase=fs_clone -promptmsg=hide -nodelist=vmohshdrc023 -console=off -sessionid=80 -timestamp=20150818_030521 -outdir=/thdr2i/applmgr/fs_ne/EBSapps/log/adop/80/fs_clone_20150818_030521/THDR2I_vmohshdrc023"

          [UNEXPECTED]Error occurred while Checking if adop can continue with available nodes using command: "perl /thdr2i/applmgr/fs2/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPEvalSrvStatus.pl -contextfile=/thdr2i/inst/fs2/inst/apps/THDR2I_vmohshdrc023/appl/admin/THDR2I_vmohshdrc023.xml -patchcontextfile=/thdr2i/inst/fs1/inst/apps/THDR2I_vmohshdrc023/appl/admin/THDR2I_vmohshdrc023.xml -phase=fs_clone -promptmsg=hide -nodelist=vmohshdrc023 -console=off -sessionid=80 -timestamp=20150818_030521 -outdir=/thdr2i/applmgr/fs_ne/EBSapps/log/adop/80/fs_clone_20150818_030521/THDR2I_vmohshdrc023".

          Previous tasks have failed or are incomplete on nodes: ",vmohshdrc024"

          [UNEXPECTED]Unable to continue with other completed nodes: "vmohshdrc023".

       

      Lines #(136-140):

          [UNEXPECTED]Error occurred while Checking if adop can continue with available nodes using command: "perl /thdr2i/applmgr/fs2/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPEvalSrvStatus.pl -contextfile=/thdr2i/inst/fs2/inst/apps/THDR2I_vmohshdrc023/appl/admin/THDR2I_vmohshdrc023.xml -patchcontextfile=/thdr2i/inst/fs1/inst/apps/THDR2I_vmohshdrc023/appl/admin/THDR2I_vmohshdrc023.xml -phase=fs_clone -promptmsg=hide -nodelist=vmohshdrc023 -console=off -sessionid=80 -timestamp=20150818_030521 -outdir=/thdr2i/applmgr/fs_ne/EBSapps/log/adop/80/fs_clone_20150818_030521/THDR2I_vmohshdrc023".

          Previous tasks have failed or are incomplete on nodes: ",vmohshdrc024"

          [UNEXPECTED]Unable to continue with other completed nodes: "vmohshdrc023".

          [UNEXPECTED]Error: While executing "txkADOPValidation" on all other nodes.

          [UNEXPECTED]Execution failed when performing remote action.

       

      Lines #(137-141):

          Previous tasks have failed or are incomplete on nodes: ",vmohshdrc024"

          [UNEXPECTED]Unable to continue with other completed nodes: "vmohshdrc023".

          [UNEXPECTED]Error: While executing "txkADOPValidation" on all other nodes.

          [UNEXPECTED]Execution failed when performing remote action.

       

       

      Lines #(138-142):

          [UNEXPECTED]Unable to continue with other completed nodes: "vmohshdrc023".

          [UNEXPECTED]Error: While executing "txkADOPValidation" on all other nodes.

          [UNEXPECTED]Execution failed when performing remote action.

       

      [EVENT]     Summary report for current adop run -

       

      /thdr2i/applmgr/fs_ne/EBSapps/log/adop/80/fs_clone_20150818_030521/THDR2I_vmohshdrc023/remote_execution_result_20150818_030607.xml:

      -----------------------------------------------------------------------------------------------------------------------------------

       

      /thdr2i/applmgr/fs_ne/EBSapps/log/adop/80/fs_clone_20150818_030521/THDR2I_vmohshdrc023/remote_execution_result_20150818_030824.xml:

      -----------------------------------------------------------------------------------------------------------------------------------

      Status of remote command on Node vmohshdrc024 is failed.

      Error string was found in the above file.

      Please check the adoplog files created in vmohshdrc024 for more details.

       

       

      /thdr2i/applmgr/fs_ne/EBSapps/log/adop/80/fs_clone_20150818_030521/THDR2I_vmohshdrc024/ADOPValidations_detailed.log:

      --------------------------------------------------------------------------------------------------------------------

       

      Lines #(341-345):

         ONS Request Port available:  Port Value = 6500

      Checking  Forms Port on vmohshdrc024:  Port Value = 9000

      RC-50204: Error: - Forms Port in use: Port Value = 9000

      Checking  Metrics Server Data Port on vmohshdrc024:  Port Value = 9100

         Metrics Server Data Port available:  Port Value = 9100

       

      Lines #(377-381):

       

      -----------------------------

      ERROR: The following required ports are in use:

      -----------------------------

      9000 : Forms Port

       

      /thdr2i/applmgr/fs_ne/EBSapps/log/adop/80/fs_clone_20150818_030521/THDR2I_vmohshdrc024/txkADOPValidations.log:

      --------------------------------------------------------------------------------------------------------------

       

      Lines #(21-25):

              [WARNING]: There could be issues while validating the ports used for E-Business Suite instance against ports used in /etc/services. Refer the log file for more details.

      ValidatePatchFSPortPool:

              [ERROR]: Some of the ports specified for the patch file system are not available.

      ValidateETCHosts:

              [SUCCESS]: All required entries exist in /etc/hosts.

      [apthdr2i@vmohshdrc024 <RUN> ]$

       

      *****************************************************************************************************************************************************************************************************************

       

      When I check directly on both nodes for port 9000:

       

      On the 1st node where validation checks have passed it is showing below:

      [apthdr2i@vmohshdrc024 <RUN> ]$ netstat -putan|grep 900

      (Not all processes could be identified, non-owned process info

        will not be shown, you would have to be root to see it all.)

      tcp 0 0 129.155.128.225:9001 0.0.0.0:* LISTEN 21817/frmsrv

      tcp 0 0 129.155.128.225:9001 199.168.246.111:53243 ESTABLISHED 50206/frmweb

       

      On the 2nd node where validation checks have failed:

       

      [apthdr2i@vmohshdrc024 <RUN> ]$ netstat -putan|grep 900

      (Not all processes could be identified, non-owned process info

        will not be shown, you would have to be root to see it all.)

      tcp 0 0 129.155.128.226:9000 0.0.0.0:* LISTEN 36993/frmsrv

       

       

      2nd node is showing forms services up on 9000 and 1st node shows 9001.

      Forms port should be same on both the nodes.

       

       

      adop current status:

       

      [apthdr2i@vmohshdrc024 <RUN> ]$ adop -status -details

       

      Enter the APPS password:

       

      ==============================================================

      ADOP (C.Delta.6)

      Session Id: 80

      Command: status

      Output: /thdr2i/applmgr/fs_ne/EBSapps/log/status_20150818_045341/adzdshowstatus.out

      ===============================================================

       

      Node Name       Node Type  Phase           Status          Started              Finished             Elapsed

      --------------- ---------- --------------- --------------- -------------------- -------------------- ------------

      vmohshdrc023    master     FS_CLONE        NOT STARTED     2015/08/17 09:28:34

       

      vmohshdrc024    slave      CONFIG_CLONE    NOT STARTED     2015/08/17 09:28:34

       

       

       

      File System Synchronization Type: Full

       

      INFORMATION: Patching cycle aborted, so fs_clone will run automatically on vmohshdrc023 node in prepare phase of next patching cycle.

       

       

      Generating full ADOP Status Report at location: /thdr2i/applmgr/fs_ne/EBSapps/log/status_20150818_045341/adzdshowstatus.out

       

      adop exiting with status = 0 (Success)

      [apthdr2i@vmohshdrc024 <RUN> ]$

       

       

       

      Kindly suggest.

       

       

       

      Regards,

      Purnima Johari