4 Replies Latest reply on Jun 21, 2008 4:45 AM by Madrid

    End of file communication error Service Request 7010160.992

    user00726
      Hi,

      Day before yesterday,we were facing a strange problem.I want to share with you.Our server was running properly,but when users try to connect the application at the client site,they couldn;t able to login to the application system was generating an error i.e end of file communication channel.For detailed problem do visit metalink and is Service Request 7010160.992

      Message was edited by:
      user00726
        • 1. Re: Service Request 7010160.992  pease follow the metalink
          Madrid
          Unfortunately there is no way any one here in the forum is able to access your Metalink SR. If you want to share it here in the forum some comments could be issued.


          ~ Madrid
          http://hrivera99.blogspot.com/
          • 2. Re: Service Request 7010160.992  pease follow the metalink
            user00726
            i think if u have metalink account no. then u can access.....
            • 3. Re: Service Request 7010160.992  pease follow the metalink
              user00726
              OK, i AM GIVING HERE A DETAILED DESCRIPTION OF PROBELM HERE,
              Abstract

              ACTIVE: SQLPLUS FAILING WITH ORA-3113 
                   

              Resolution History


              19-JUN-08 07:22:06 GMT

              The following files were uploaded for SR number 7010160.992:

              File Name: NEWTRACE.TXT
              File Name: TRC.LOG
              File Name: alert_neosoft.log

              This SR was saved as a draft on: 19-JUN-2008 00:21:38

              ### Platform and O/S version, including patch set or service pack level? ###wINDOWS 2003 ADVANCED SERVER
              ### What version and patch set level of the database are you running? ###VERSION AND
              PATCH 9.2.0.7
              ### Are you running the most recent patch set? ###WE HAD INSTALLED 9.2.0.7 PACHSET INSTALLED
              ### Problem Description and its duration: ###THE PROBLE WAS STARTED AROUND 9:30 AM,WHEN USER TRY TO LOGIN INTO THE
              APPLICATION AS SOON AS USER CLICK ON THE APPLICATION IT GENERATES AN ERROR
              "ORA-03113 END OF FILE COMMUNICATION CHANNEL".AT THE CLIENT SITE, WE HAVE
              TRACED THE PROBLEM AND LOG IN WHICH IT IS DESCRIBIBGTHE PROBLEM RELATED TO TNS:
              IE.

              Time: 19-JUN-2008 12:06:49
              Tracing to file: D:\oracle\ora90\network\ADMIN\TRACE\NEWTRACE.TXT
              Tns error struct:
              ns main err code: 12560
              TNS-12560: TNS:proto adapter error
              ns secondary err code: 0
              nt main err code: 530
              TNS-00530: Protocol adapter error
              nt secondary err code: 126
              nt OS err code: 0

              ### If you are receiving errors, please list the exact error messages and message text.### reporting
              NS-12560 error as ORA-12560
              ### Did the error generate a trace file? ###Yes
              ### List and describe files that you plan to upload: ###
              CONTENTS OF TRACE FILE GENERATED AT THE CLIENT SITE USING SQLNET.ORA

              [19-JUN-2008 12:23:18:458]
              --- TRACE CONFIGURATION INFORMATION FOLLOWS ---
              [19-JUN-2008 12:23:18:458] New trace stream is D:\oracle\ora90\network\ADMIN\TRACE\NEWTRACE.TXT
              [19-JUN-2008 12:23:18:458] New trace level is 16
              [19-JUN-2008 12:23:18:458] --- TRACE CONFIGURATION INFORMATION ENDS ---
              [19-JUN-2008 12:23:18:458]
              --- PARAMETER SOURCE INFORMATION FOLLOWS ---
              [19-JUN-2008 12:23:18:458] Attempted load of system pfile source d:\oracle\ora90\network\admin\sqlnet.ora
              [19-JUN-2008 12:23:18:458] Parameter source loaded successfully
              [19-JUN-2008 12:23:18:458]
              [19-JUN-2008 12:23:18:458] Attempted load of local pfile source C:\Documents and Settings\tsingh\sqlnet.ora
              [19-JUN-2008 12:23:18:458] Parameter source was not loaded
              [19-JUN-2008 12:23:18:458]
              [19-JUN-2008 12:23:18:458] -> PARAMETER TABLE LOAD RESULTS FOLLOW <-
              [19-JUN-2008 12:23:18:458] Successful parameter table load
              [19-JUN-2008 12:23:18:458] -> PARAMETER TABLE HAS THE FOLLOWING CONTENTS <-
              [19-JUN-2008 12:23:18:458] LOG_DIRECTORY_CLIENT = D:\oracle\ora90\network\ADMIN\LOG
              [19-JUN-2008 12:23:18:458] TRACE_LEVEL_CLIENT = 16
              [19-JUN-2008 12:23:18:458] NAMES.DIRECTORY_PATH = (TNSNAMES, ONAMES, HOSTNAME)
              [19-JUN-2008 12:23:18:458] TRACE_FILE_CLIENT = D:\oracle\ora90\network\ADMIN\TRACE\NEWTRACE.TXT
              [19-JUN-2008 12:23:18:458] SQLNET.AUTHENTICATION_SERVICES = (NTS)
              [19-JUN-2008 12:23:18:458] LOG_FILE_CLIENT = D:\oracle\ora90\network\ADMIN\LOG\TRC.LOG
              [19-JUN-2008 12:23:18:458] TRACE_DIRECTORY_CLIENT =
              D:\oracle\ora90\network\ADMIN\TRACE
              [19-JUN-2008 12:23:18:458] TRACE_UNIQUE_CLIENT = ON
              [19-JUN-2008 12:23:18:458] NAMES.DEFAULT_DOMAIN = mahd.com
              [19-JUN-2008 12:23:18:458] --- PARAMETER SOURCE INFORMATION ENDS ---
              [19-JUN-2008 12:23:18:458]
              --- LOG CONFIGURATION INFORMATION FOLLOWS ---
              [19-JUN-2008 12:23:18:458] Log stream will be "D:\oracle\ora90\network\ADMIN\LOG\TRC.LOG"
              [19-JUN-2008 12:23:18:458] Log stream validation not requested
              [19-JUN-2008 12:23:18:458] --- LOG CONFIGURATION INFORMATION ENDS ---

              [19-JUN-2008 12:23:18:458] ntvllt: entry
              [19-JUN-2008 12:23:18:458] ntvllt: exit
              [19-JUN-2008 12:23:18:458] nigini: entry
              [19-JUN-2008 12:23:18:458] nigini: Count in NI global area now: 1
              [19-JUN-2008 12:23:18:458] nigini: Count in NI global area now: 1
              [19-JUN-2008 12:23:18:458] nrigbi: entry
              [19-JUN-2008 12:23:18:458] nrigbni: entry
              [19-JUN-2008 12:23:18:458] nrigbni: Unable to get data from navigation file tnsnav.ora
              [19-JUN-2008 12:23:18:458] nrigbni: exit
              [19-JUN-2008 12:23:18:458] nrigbi: exit
              [19-JUN-20

              19-JUN-08 07:22:10 GMT

              Text continued from previous action...

              ### What was being done at time of error/any changes since this last worked? ###WHEN THE ERROR OCCURED USER JUST
              TRIED TO OPEN THE APPLICATION
              ### Error can be generated if SQL statement is run in SQL*Plus/Server Mgr: ###Unknown
              ### What is the frequency of the error? ###Consistently
              ### What is the impact to your business because of this problem? ###CRTICAL,NO OF USERS AFFECTED -200
              ### Are you running any third-party applications? ###YES
              ### Has the third-party Vendor been contacted? ###No
              Can you easily recover from, bypass or work around the problem? No
              Does your system or application continue normally after the problem occurs? No
              Are the standard features of the system or application still available; is the loss
              of service minor?Yes
              Error : NS-12560
              Contact me via : Telephone -> 9811657928


              19-JUN-08 07:25:34 GMT

              TAR has been assigned to an analyst -- Sending email.

              19-JUN-08 08:10:10 GMT

              .
              ############# SEVERITY 1 REQUEST TEMPLATE ###################

              Business Justification:

              This is a hospital and all their services are down

              Can you easily recover from, bypass or work around the problem?
              No
              Does your system or application continue normally after the problem occurs?
              No
              Are the standard features of the system or application still available; is the loss of service minor?
              No
              Is this a Data Corruption issue?
              No
              Do you have a Complete Loss of Service?
              Yes
              Is this a Test or Production Environment?
              Prod
              Are you unable to proceed with implementation and the go live date is in less than 3 days?


              ALL PHONE NUMBERS SHOULD INCLUDE THE COUNTRY CODE AND AREA CODE.

              Customers Management name:
              Customer's Management 24x7 contact number:
              Primary Customer contact name: LOKESH AGGARWAL
              Current Customer 24x7 voice phone number: +91 98 1165 7928 or +91 98 1057 7271
              Current Customer Email address: aggarwalokesh@gmail.com
              Current Customer Pager/Fax number:
              Secondary Contact Name: Srivatsan
              Secondary Contact Phone number: +91 98 1165 7928 or +91 98 1057 7271
              Secondary Contact Email:



              19-JUN-08 08:14:18 GMT

              UPDATE
              ======
              Hi Lokesh,

              Thank you for using MetaLink. We are currently reviewing/researching the situation and will update the Service Request (SR) or call yo
              u as soon as we have relevant information. Thank you for your patience.

              Best Regards,
              ShashiKirAN
              Global Customer Services

              STATUS
              =======

              @WIP -- Work In Progress



              19-JUN-08 08:35:27 GMT

              .
              ISSUE CLARIFICATION
              ====================

              Called customer +91 98 1165 7928

              Sqlplus is failing with ORA-3113.
              This setup was working fine before. The problem started to occur since this morning...They are using a third party application t
              o connect the
              database. No firewall between the client and server . tnsping works. Local (Beq) connections to the DB works.

              .
              ISSUE VERIFICATION
              ===================

              Not yet.

              Spoke to ct, they will be uploading the client sqlnet trace.

              .
              UPDATE
              =======
              Hi lokesh,

              Waiting for you to upload the client trace.
              Also upload the alert log from the database server...

              Thanks,
              ShashiKirAN
              GCS



              19-JUN-08 08:35:37 GMT

              Email Update button has been pressed: Sending email to aggarwalokesh@gmail.com.

              19-JUN-08 08:48:05 GMT

              The customer : aggarwalokesh@gmail.com : has uploaded the following file via MetaLink:
              D:\oracle\ora90\network\ADMIN\TRACE\NEWTRACE.TXT


              19-JUN-08 09:28:00 GMT

              .
              UPDATE
              =======
              Hi Lokesh,

              I'm currently downloading the client trace file and analyzing it. Meanwhile , I would like to have one more clarification
              on this issue :

              + Does all the clients fail with ORA-3113 ? OR only this particular client is failing with ORA-3113?

              Thanks,
              --SK



              19-JUN-08 09:28:12 GMT

              Email Update button has been pressed: Sending email to aggarwalokesh@gmail.com.

              19-JUN-08 10:01:45 GMT

              .
              DATA COLLECTED
              ===============

              Client trace : (NEWTRACE.TXT)
              ------------------------

              [19-JUN-2008 13:48:13:566] nspsend: packet dump
              [19-JUN-2008 13:48:13:566] nspsend: 00 25 00 00 06 00 00 00 |.%......|
              [19-JUN-2008 13:48:13:566] nspsend: 00 00 01 06 05 04 03 02 |........|
              [19-JUN-2008 13:48:13:566] nspsend: 01 00 49 42 4D 50 43 2F |..IBMPC/|
              [19-JUN-2008 13:48:13:566] nspsend: 57 49 4E 5F 4E 54 2D 38 |WIN_NT-8|
              [19-JUN-2008 13:48:13:566] nspsend: 2E 31 2E 30 00 |.1.0. |
              .......
              .........
              [19-JUN-2008 13:48:13:566] nsrdr: recving a packet
              [19-JUN-2008 13:48:13:566] nsprecv: entry
              [19-JUN-2008 13:48:13:566] nsprecv: reading from transport...
              .........
              [19-JUN-2008 13:48:13:566] ntt2err: soc 264 error - operation=5, ntresnt[0]=530, ntresnt[1]=53, ntresnt[2]=0
              [19-JUN-2008 13:48:13:566] ntt2err: exit
              [19-JUN-2008 13:48:13:566] nttrd: exit
              [19-JUN-2008 13:48:13:566] nsprecv: transport read error
              [19-JUN-2008 13:48:13:566] nsprecv: error exit
              [19-JUN-2008 13:48:13:566] nserror: entry
              [19-JUN-2008 13:48:13:566] nserror: nsres: id=0, op=68, ns=12570, ns2=12560; nt[0]=530, nt[1]=53, nt[2]=
              0; ora[0]=0, ora[1]=0, ora[2]=0

              Observation/Analysis:
              ------------------------------

              + TNS-12570 -- TNS:packet reader failure . An error occurred during a data receive.
              + nt[1]=53

              WSAECONNABORTED 53 10053
              ---------------------------------------------------------
              Software caused connection abort. An established connection was aborted by the
              software in your host machine, possibly due to a data transmission timeout or
              protocol error.

              ==============================

              .
              UPDATE
              =======

              Hi lokesh,

              Thank you for supplying the trace information.

              Please clarify :

              + Does all the clients fail with ORA-3113 ? OR only this particular client is failing with ORA-3113?

              The main part we are looking at is the following line in the trace :


              [19-JUN-2008 13:48:13:566] nserror: nsres: id=0, op=68, ns=12570, ns2=12560; nt[0]=530, nt[1]=53, nt[2]=0; or
              a[0]=0, ora[1]=0, ora[2]=0

              and the real error here is the following << nt[1]=53 >> . This is a non oracle error that is forcing the connections to fail this er
              ror is comming from the O/S or network and causing the error

              - At the time of the problem, do you see any errors in the application log / system logs (event v
              iewer) related to network problems.
              Suggest to obtain assistance from Sysadmin / network adminstrators for this to check up server's NIC / bad network cable /
              switch/hub port settings etc if you seem to get errors from many clients and che
              ck in the system logs for any network related errors.


              .
              ACTION PLAN
              ============
              1# I am supplying this action to aid you in to find the problem but based on the error and trace information the oracle error 12570 is not the root cause an
              d is only the end result. You will need to take a closer look at your Network an
              d O/S configuration, Intermittent SQL*Net TCP/IP read and write errors are somet
              imes encountered on NT. The underlying reasons of these errors are a synchroniza
              tion error in the TCP/IP layer on NT. To help prevent this kind of errors, a few
              things can be adjusted to help the synchronization:


              A#. TCP.NODELAY parameter

              Try adding the below parameter in the client sqlnet.ora :
              tcp.nodelay = yes

              In most cases, TCP/IP info send across the network is buffered till at least a complete network packet can be send. This means that in certain cas
              es commands are not issued directly, and kept buffered until some other info can
              be send as well.This has the potential to generate timeouts and errors.To avoid
              this, this delay can be switched off.

              tcp.nodelay = yes

              B#. Disabling AUTOMATIC_IPC on Clients
              On client PC's, checking for IPC connections is pointless as there is never a database installed on them. So, in order to save some time
              during the connections phase, set AUTOMATIC_IPC=OFF in the "SQLNET.ORA" file.

              C#. Add the following to the Server and Client sqlnet.ora file

              Disable_OOB = ON

              D#.Increase TcpMaxDataRetransmissions parameter in the Windows Registry


              On Windows NT/2000:

              HKEY_LOCAL_MACHINE
              SYSTEM
              CurrentControlSet
              Services
              Tcpip
              Parameters

              add a key "TcpMaxDataRetransmissions" (REG_DWORD) and set its value to 10.

              On the Edit menu, click Add Value, and then add the following:

              Value Name: TcpMaxDataRetransmissions
              Value Type: REG_DWORD - Number
              Valid Range: 0 - 0xFFFFFFFF
              Default Value: 5 Decimal
              New Value: 10 to Decimal

              The value may vary as it is dynamically adjusted.

              E#:
              If you not using NTS authetication for client-server then comment out the SQLNET.AUTHENTICATI
              ON_SERVICES parameter
              in the client side sqlnet.ora.

              F#:
              Check whether the listener service and database service are started as same domain user / local acco
              unt as applicable.
              If this is a MS AD environment, then try starting with Domain account.

              G#:
              Check whether there is any firewall between the client and server. if so, check whether the all the ports are reachable for the client. for eg.
              for the uploaded client trace port used is 1521.

              C:\> telnet <server hostname/ ip > 1521

              if this fails then the firewall /network device is blocking the connection and hence contact your network / firewall administrator to
              take necessary actions.


              Kind Regards,
              ShashiKirAN
              GCS


              19-JUN-08 10:01:53 GMT

              Email Update button has been pressed: Sending email to aggarwalokesh@gmail.com.

              19-JUN-08 11:42:41 GMT

              ===========================================
              ** HUB **
              CUSTOMER UPDATE

              Customer contact name (if not same as in SR header): Lokesh Aggarwal


              Verify preferred method of contact: Phone
              Phone#: +91 9811657928
              Email: aggarwalokesh@gmail.com


              Comments for this update:

              Customer would like to talk to the analyst, please call +91 9811657928 asap.

              Thank you and kind regards,
              Danijela Mustar
              EMEA HUB

              @ contacted CTC Khalid Rachidi on IM


              19-JUN-08 12:49:20 GMT

              .
              UPDATE
              =======
              Hi Lokesh,

              Which points ion the action plan given by my colleague did you follow ? What is the result so far ?

              Thanks,
              Bogdan



              19-JUN-08 12:49:32 GMT

              Email Update button has been pressed: Sending email to aggarwalokesh@gmail.com.

              19-JUN-08 12:54:28 GMT

              .
              DATA COLLECTED
              ===============
              ++ From client trace new trace.txt

              [19-JUN-2008 13:48:13:566] ntt2err: entry
              [19-JUN-2008 13:48:13:566] ntt2err: soc 264 error - operation=5, ntresnt[0]=530, ntresnt[1]=53, ntresnt[2]=0
              [19-JUN-2008 13:48:13:566] ntt2err: exit
              [19-JUN-2008 13:48:13:566] nttrd: exit
              [19-JUN-2008 13:48:13:566] nsprecv: transport read error
              [19-JUN-2008 13:48:13:566] nsprecv: error exit
              [19-JUN-2008 13:48:13:566] nserror: entry
              [19-JUN-2008 13:48:13:566] nserror: nsres: id=0, op=68, ns=12570, ns2=12560; nt[0]=530, nt[1]=53, nt[2]=0; ora[0]=
              0, ora[1]=0, ora[2]=0
              [19-JUN-2008 13:48:13:566] nsrdr: error exit
              [19-JUN-2008 13:48:13:566] nsdo: nsctxrnk=0
              [19-JUN-2008 13:48:13:566] nsdo: error exit
              [19-JUN-2008 13:48:13:566] nioqrc: recv: packet reader error -> translated to IFCR_EO
              F
              [19-JUN-2008 13:48:13:566] nioqrs: entry
              [19-JUN-2008 13:48:13:566] nioqrs: state = interrupted (1)
              [19-JUN-2008 13:48:13:566] nioqsm: entry
              [19-JUN-2008 13:48:13:566] nioqsm: Sending break packet (1)...
              [19-JUN-2008 13:48:13:566] nttwr: entry
              [19-JUN-2008 13:48:13:566] ntt2err: entry
              [19-JUN-2008 13:48:13:566] ntt2err: soc 264 error - operation=6, ntresnt[0]=530, ntresnt[1]=53, ntresnt[2]=0
              [19-JUN-2008 13:48:13:566] ntt2err: exit
              [19-JUN-2008 13:48:13:566] nttwr: exit
              [19-JUN-2008 13:48:13:566] nserror: entry
              [19-JUN-2008 13:48:13:566] nserror: nsres: id=5060, op=152, ns=12571, ns2=12560; nt[0]=0, nt[1]=0, nt[2]=0; ora[0]=0, o
              ra[1]=0, ora[2]=0
              [19-JUN-2008 13:48:13:566] nioqsm: send-break: failed to send break...
              [19-JUN-2008 13:48:13:566] nioqper: error from send-marker
              [19-JUN-2008 13:48:13:566] nioqper: nr err code: 0
              [19-JUN-2008 13:48:13:566] nioqper: ns main err code: 12570
              [19-JUN-2008 13:48:13:566] nioqper: ns (2) err code: 12560
              [19-JUN-2008 13:48:13:566] nioqper: nt main err code: 530
              [19-JUN-2008 13:48:13:566] nioqper: nt (2) err code: 53
              [19-JUN-2008 13:48:13:566] nioqper: nt OS err code: 0
              [19-JUN-2008 13:48:13:566] nioqsm: exit
              [19-JUN-2008 13:48:13:566] nioqer: entry
              [19-JUN-2008 13:48:13:566] nioqer: incoming err = 12152
              [19-JUN-2008 13:48:13:566] nioqce: entry
              [19-JUN-2008 13:48:13:566] nioqce: exit
              [19-JUN-2008 13:48:13:566] nioqer: returning err = 3113
              [19-JUN-2008 13:48:13:566] nioqer: exit
              [19-JUN-2008 13:48:13:566] nioqrs: nioqrs: Couldn't send break. returning 3113
              [19-JUN-2008 13:48:13:566] nioqrs: exit


              19-JUN-08 16:38:06 GMT

              New info : aggarwalokesh@gmail.com
              We have to deinstalled the oracle s/w from the server and installed it again.Then, we resoted the database from the cold
              backup.We had followed the steps you had emaild, but it did not work.


              19-JUN-08 18:39:59 GMT


              ACTION PLAN
              ============
              Ok with the fact that a reinstall occurred I will need the following.

              1. RDA for the serrver node.
              Please download and run the Remote Diagnostic Agent (RDA) on the instance
              where you are experiencing the issue and upload the output file to MetaLink.
              Instructions for downloading and running RDA are available via the following
              Metalink notes:

              Note 314422.1 "Remote Diagnostic Agent (RDA) 4 - Overview and User Guide"
              Note 330363.1 "Remote Diagnostic Agent (RDA) 4 - FAQ"
              Note 330344.1 "Remote Diagnostic Agent (RDA) 4 - Training"
              Note 330362.1 "Remote Diagnostic Agent (RDA) 4 - Troubleshooting Guide"
              Note 359395.1 "Remote Diagnostic Agent (RDA) 4 - RAC Cluster Guide"
              Note 359388.1 "Remote Diagnostic Agent (RDA) 4 - RAC Cluster FAQ"


              2. Client and server matching trace of the connection error taking place.
              Place these parameters in the sqlnet.ora file on the client node.
              TRACE_LEVEL_CLIENT=16
              TRACE_FILE_CLIENT=CLIENT
              TRACE_DIRECTORY_CLIENT=full path to directory where you want the trace file created.
              TRACE_TIMESTAMP_CLIENT=on
              TRACE_UNIQUE_CLIENT=on


              Place These parameters in the sqlnet.ora file on the server node.
              TRACE_LEVEL_SERVER=16
              TRACE_FILE_SERVER=SERVER
              TRACE_DIRECTORY_SERVER=full path to directory where you want the trace file created.
              TRACE_TIMESTAMP_SERVER=on

              Note 374116.1:How to Match Oracle Net Client and Server Trace Files

              3. Please provide the alert.log file for AFTER the connection attempt is issued that generate
              s the errors.

              STATUS
              =====
              @cus


              • 4. Re: Service Request 7010160.992  pease follow the metalink
                Madrid
                Metalink doesn't allow access to any service request but my own.

                On the other hand, would you mind killing your caps lock key?
                OK, i AM GIVING HERE A DETAILED DESCRIPTION OF PROBELM HERE,
                ~ Madrid
                http://hrivera99.blogspot.com/