9 Replies Latest reply on Jan 7, 2010 7:42 PM by Lakmal Rajapakse

    Linux Error: 32: Broken Pipe

    745105
      I'm having an issue where the connection fails only on Red Hat 5. We have an installer that configures everything the same way for both Red Hat 4 and Red Hat 5 (configuration files). The bits are for the appropriate version of Red Hat. I'm seeing the problem on Red Hat Enterprise Linux Server release 5.3 (Tikanga). I assume it affects all versions of Red Hat 5.

      I've included the listener.log below. You see that it is working fine and then the errors occur. This segment of the log below is taken during our installation of our product.

      The confusing part is that after I kill the install, the listener seems to start working again. So some of the connections appear to be working, then it fails, then everything starts working again.

      I'm looking for some things to try. Any suggestions would be appreciated.


      Oracle versions:
      ------------------------------------------------------------------------------------------------------------------
      Oracle RDBMS 10.2.0.4.0-16
      Instantclient 11.1.0.6-0


      listener.ora
      ------------------------------------------------------------------------------------------------------------------
      LISTENER =
      (ADDRESS_LIST=
      (ADDRESS=
      (PROTOCOL=tcp)
      (HOST=nc02.example.com)
      (PORT=1521)
      )
      (ADDRESS=
      (PROTOCOL=ipc)
      (KEY=PNPKEY)
      )
      )
      SID_LIST_LISTENER=
      (SID_LIST=
      (SID_DESC=
      (SID_NAME=cmdb)
      (ORACLE_HOME=/u01/app/oracle/product/10.2.0/db_1)
      )
      )

      LOG_DIRECTORY_LISTENER=/u01/app/oracle/product/10.2.0/db_1/network/log
      LOG_FILE_LISTENER=listener.log
      TRACE_DIRECTORY_LISTENER=/u01/app/oracle/product/10.2.0/db_1/network/trace
      TRACE_FILE_LISTENER=listener.trc
      INBOUND_CONNECT_TIMEOUT_LISTENER=120
      #LOCAL_OS_AUTHENTICATION_LISTENER=OFF
      SAVE_CONFIG_ON_STOP_LISTENER = ON
      PASSWORDS_LISTENER = CEE9A6452943F605

      tnsnames.ora
      ------------------------------------------------------------------------------------------------------------------
      cmdb=(DESCRIPTION=
      (ADDRESS=
      (HOST=nc02.example.com)
      (PORT=1521)
      (PROTOCOL=tcp)
      )
      (CONNECT_DATA=
      (SERVICE_NAME=cmdb.example.com)
      )




      listener.log
      ------------------------------------------------------------------------------------------------------------------

      TIMESTAMP * CONNECT DATA [* PROTOCOL INFO] * EVENT [* SID] * RETURN CODE
      WARNING: Subscription for node down event still pending
      07-JAN-2010 06:29:12 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=nc02.example.com)(USER=oracle))(COMMAND=status)(ARGUMENTS=64)(PASSWORD=xxxxxxxx)(SERVICE=LISTENER)(VERSION=169870336)) * status * 0
      07-JAN-2010 06:29:15 * service_register * cmdb * 0
      07-JAN-2010 06:29:21 * service_update * cmdb * 0
      07-JAN-2010 06:29:30 * service_update * cmdb * 0
      07-JAN-2010 06:29:33 * service_update * cmdb * 0
      07-JAN-2010 06:31:09 * service_update * cmdb * 0
      WARNING: Subscription for node down event still pending
      07-JAN-2010 06:31:23 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=nc02.example.com)(USER=oracle))(COMMAND=status)(ARGUMENTS=64)(SERVICE=LISTENER)(VERSION=169870336)) * status * 0
      07-JAN-2010 06:31:52 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=nc02.example.com)(USER=oracle))(COMMAND=services)(ARGUMENTS=64)(SERVICE=LISTENER)(VERSION=169870336)) * services * 0
      07-JAN-2010 06:35:00 * service_update * cmdb * 0
      07-JAN-2010 06:35:51 * 12502
      TNS-12502: TNS:listener received no CONNECT_DATA from client
      07-JAN-2010 06:36:35 * ping * 0
      07-JAN-2010 06:36:35 * version * 0
      07-JAN-2010 06:36:36 * service_died * cmdb * 12537
      07-JAN-2010 06:36:39 * service_register * cmdb * 0
      07-JAN-2010 06:36:45 * service_update * cmdb * 0
      07-JAN-2010 06:36:48 * service_update * cmdb * 0
      07-JAN-2010 06:37:19 * (CONNECT_DATA=(SID=cmdb)(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.101.34.4)(PORT=27312)) * establish * cmdb * 12518
      TNS-12518: TNS:listener could not hand off client connection
      TNS-12547: TNS:lost contact
      TNS-12560: TNS:protocol adapter error
      TNS-00517: Lost contact
      Linux Error: 32: Broken pipe
      07-JAN-2010 06:37:20 * (CONNECT_DATA=(SID=cmdb)(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.101.34.4)(PORT=27313)) * establish * cmdb * 12518
      TNS-12518: TNS:listener could not hand off client connection
      TNS-12547: TNS:lost contact
      TNS-12560: TNS:protocol adapter error
      TNS-00517: Lost contact
      Linux Error: 32: Broken pipe
      07-JAN-2010 06:37:47 * (CONNECT_DATA=(SID=cmdb)(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.101.34.4)(PORT=27317)) * establish * cmdb * 12518
      TNS-12518: TNS:listener could not hand off client connection
      TNS-12547: TNS:lost contact
      TNS-12560: TNS:protocol adapter error
      TNS-00517: Lost contact
      Linux Error: 32: Broken pipe
      07-JAN-2010 06:37:51 * (CONNECT_DATA=(SID=cmdb)(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.101.34.4)(PORT=27320)) * establish * cmdb * 12518
      TNS-12518: TNS:listener could not hand off client connection
      TNS-12547: TNS:lost contact
      TNS-12560: TNS:protocol adapter error
      TNS-00517: Lost contact
      Linux Error: 32: Broken pipe
      07-JAN-2010 06:37:53 * (CONNECT_DATA=(SID=cmdb)(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.101.34.4)(PORT=27321)) * establish * cmdb * 12518
      TNS-12518: TNS:listener could not hand off client connection
      TNS-12547: TNS:lost contact
      TNS-12560: TNS:protocol adapter error
      TNS-00517: Lost contact
      Linux Error: 32: Broken pipe
      07-JAN-2010 06:37:55 * (CONNECT_DATA=(SID=cmdb)(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.101.34.4)(PORT=27322)) * establish * cmdb * 12518
      TNS-12518: TNS:listener could not hand off client connection
      TNS-12547: TNS:lost contact
      TNS-12560: TNS:protocol adapter error
      TNS-00517: Lost contact
      Linux Error: 32: Broken pipe
      07-JAN-2010 06:37:56 * (CONNECT_DATA=(SID=cmdb)(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.101.34.4)(PORT=27324)) * establish * cmdb * 12518
      TNS-12518: TNS:listener could not hand off client connection
      TNS-12547: TNS:lost contact
      TNS-12560: TNS:protocol adapter error
      TNS-00517: Lost contact
      Linux Error: 32: Broken pipe
      07-JAN-2010 06:37:59 * (CONNECT_DATA=(SID=cmdb)(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.101.34.4)(PORT=27326)) * establish * cmdb * 12518
      TNS-12518: TNS:listener could not hand off client connection
      TNS-12547: TNS:lost contact
      TNS-12560: TNS:protocol adapter error
      TNS-00517: Lost contact
      Linux Error: 32: Broken pipe
      07-JAN-2010 06:39:36 * service_update * cmdb * 0
      07-JAN-2010 06:49:39 * service_update * cmdb * 0
      07-JAN-2010 06:59:42 * service_update * cmdb * 0
      07-JAN-2010 07:09:45 * service_update * cmdb * 0
        • 1. Re: Linux Error: 32: Broken Pipe
          sb92075
          Is any type of Operating System virtualization installed on this system?
          If so, what?
          • 2. Re: Linux Error: 32: Broken Pipe
            745105
            This operating system is running as a guest inside a VMware ESX server (version 3.5).
            • 3. Re: Linux Error: 32: Broken Pipe
              sb92075
              This operating system is running as a guest inside a VMware ESX server (version 3.5).
              networking is mis-configured some place between virtual world & real world & root cause is not associated with Oracle.
              Oracle is the victim; not the culprit.

              Please don't respond that because some networking operations succeed, therefore no problem exists.
              The root cause of these errors is outside of Oracle.
              • 4. Re: Linux Error: 32: Broken Pipe
                745105
                I'm not sure I understand. Let me give some more background. We have lots of ESX servers. Our Red Hat 4 guests run on these same ESX servers. QA has ESX servers and they have seen this issue too.

                Are you thinking that it is a networking problem inside the Red Hat 5 guests? I'm not sure what the virtualization aspect has to do with the issue.

                Are you aware of any network configuration changes in Red Hat 5 that are different from Red Hat 4?
                • 5. Re: Linux Error: 32: Broken Pipe
                  sb92075
                  I'm not sure what the virtualization aspect has to do with the issue.
                  These errors do not occur when no virtualization is involved.

                  If you contend that virtualization is blameless & not contributing to these errors, you are free to do so.
                  I am not going to argue with you.

                  Enjoy your mystery & errors
                  • 6. Re: Linux Error: 32: Broken Pipe
                    745105
                    If anyone else would care to help out, I would appreciate it. I'm not argumentative.

                    The logic of the previous solution doesn't make sense in this case. We have Red Hat 4 machines running on the same ESX servers as guests and they work fine. That leads me to believe that something is different with either the environment on Red Hat 5 or the oracle bits. Unfortunately, I'm unable to test whether or not the above suggestion works because we don't have dedicated hardware available for testing.
                    • 7. Re: Linux Error: 32: Broken Pipe
                      Lakmal Rajapakse
                      This sort of indicates that listener could not create a dedicated server process to hand the client request over to. Did you check you the alert log for the cmdb database to see if there are any errors?
                      • 8. Re: Linux Error: 32: Broken Pipe
                        sb92075
                        CUT lines below & PASTE into TERMINAL owned by OS user that owns Oracle s/w tree

                        date
                        env | sort
                        cat /etc/hosts
                        uname -a
                        lsnrctl status
                        lsnrctl service
                        date

                        CUT commands & results from TERMINAL window and PASTE back here.
                        • 9. Re: Linux Error: 32: Broken Pipe
                          Lakmal Rajapakse
                          I am logging off for the day but these are my guesses as to why you are getting the error:

                          1. You have reached the max processes limit set on the database and therefore cannot spawn a new server process

                          2. You are trying the connect to the database while the database is still in the process of coming up

                          3. You have hit some memory limit or some other OS issue that is preventing the server process from been created.