13 Replies Latest reply: Jun 20, 2014 3:14 AM by ora_tech RSS

    ASM instances restarting, all database restart too

    user1138812

      Hi,

       

      i'm facing a problem that i can't understand: asm instances on 2 nodes are restarting for the same reason and all the databases too after the ASM  crash (obviously), here the alert log of ASM.

      This is a grid installation (11.2.0.4) on 2 windows 2012 server nodes:

       

      Unexpected return code (6) from the Cluster Synchronization Service (LCK0)

      Please check the CSS log file for more detail

      Errors in file D:\APP\ORACLE\diag\asm\+asm\+asm1\trace\+asm1_lck0_8628.trc:

      ORA-29701: unable to connect to Cluster Synchronization Service

      LCK0 (ospid: 8628): terminating the instance due to error 29701

      Tue Jun 17 23:42:07 2014

      Errors in file D:\APP\ORACLE\diag\asm\+asm\+asm1\trace\+asm1_diag_3772.trc:

      ORA-27508: IPC error sending a message

      ORA-27300: OS system dependent operation:IPCSOCK_Send failed with status: 10054

      ORA-27301: OS failure message: An existing connection was forcibly closed by the remote host.

      ORA-27302: failure occurred at: send_3

      System state dump requested by (instance=1, osid=8628 (LCK0)), summary=[abnormal instance termination].

      System State dumped to trace file D:\APP\ORACLE\diag\asm\+asm\+asm1\trace\+asm1_diag_3772_20140617234207.trc

      Dumping diagnostic data in directory=[cdmp_20140617234207], requested by (instance=1, osid=8628 (LCK0)), summary=[abnormal instance termination].

      Tue Jun 17 23:42:08 2014

      opiodr aborting process unknown ospid (3824) as a result of ORA-1092

      Tue Jun 17 23:42:08 2014

      ORA-1092 : opitsk aborting process

      Tue Jun 17 23:42:12 2014

      NOTE: ASMB process exiting, either shutdown is in progress

      NOTE: or foreground connected to ASMB was killed.

      Tue Jun 17 23:42:14 2014

      ORA-1092 : opitsk aborting process

      Tue Jun 17 23:42:14 2014

      License high water mark = 6

      Tue Jun 17 23:42:19 2014

      Instance terminated by LCK0, pid = 8628

      Tue Jun 17 23:42:48 2014

      MEMORY_TARGET defaulting to 1128267776.

      * instance_number obtained from CSS = 1, checking for the existence of node 0...

      * node 0 does not exist. instance_number = 1

       

      then i've inspected the 2 traces:

       

      here an extract of +asm1_lck0_8628.trc:

       

      KSXPCINI: kjxgnpub KSXP 4.1.0LCK0 10, 0

      2014-06-17 23:42:07.020: [ CSSCLNT]clsssRecvMsg: got a disconnect from the server while waiting for message type 1

       

      *** 2014-06-17 23:42:07.020

      2014-06-17 23:42:07.020: [ CSSCLNT]clssgsGroupGetStatus:  communications failed (0/3/-1)

      2014-06-17 23:42:07.020: [ CSSCLNT]clssgsGroupGetStatus: returning 8

      kgxgnpstat: received ABORT event from CLSS

      ksimpoll: kgxgnpstat returns ABORT for group (IG+ASMSYS$BACKGROUND)

      error 29701 detected in background process

      ORA-29701: unable to connect to Cluster Synchronization Service

      kjzduptcctx: Notifying DIAG for crash event

      ----- Abridged Call Stack Trace -----

      ksedsts()+585<-kjzdssdmp()+329<-kjzduptcctx()+288<-kjzdicrshnfy()+99<-ksuitm()+1525<-ksbrdp()+4578<-opirip()+853<-opidrv()+906<-sou2o()+98<-opimai_real()+280<-opimai()+191<-BackgroundThreadStart()+646<-000007FDD92C1832<-000007FDDBE5D609

      ----- End of Abridged Call Stack Trace -----

       

       

      *** 2014-06-17 23:42:07.138

      LCK0 (ospid: 8628): terminating the instance due to error 29701

      Dump file d:\app\oracle\diag\asm\+asm\+asm1\trace\+asm1_lck0_8628.trc

      Tue Jun 17 23:42:12 2014

      ORACLE V11.2.0.4.0 - 64bit Production vsnsta=0

      vsnsql=16 vsnxtr=3

      *** 2014-06-17 23:42:12.359 ssthreadkillsafe: about to forcefully terminate tid=8540, dumping call stack:

       

      here an extract of +asm1_diag_3772.trc:

       

      *** ACTION NAME:() 2014-06-17 19:42:51.874

       

      Node id: 0

      List of nodes: 0, 1,

       

       

      *** 2014-06-17 19:42:51.876

      Reconfiguration starts [incarn=0]

       

       

      *** 2014-06-17 19:42:51.876

      I'm the master node

      Group reconfiguration cleanup

       

       

      *** 2014-06-17 19:42:51.878

      A rcfg proposal from node 1 is received

      vote_map :3000000000000000000000000000000000000000000000000000000000000000

      expect_map :3000000000000000000000000000000000000000000000000000000000000000

      reply_map :2000000000000000000000000000000000000000000000000000000000000000

      vote->incar_num 8, ctx->incar_num 0, commit_incar 1, prop_incar 1

       

       

      *** 2014-06-17 19:42:51.878

      Reconfiguration completes [incarn=8]

      kjzdattdlm: Can not attach to DLM (LMON up=[FALSE], DB mounted=[FALSE]).

       

       

      *** 2014-06-17 19:42:51.878

      kjzdm_rac: enabling heartbeat after reconfig

      kjzdattdlm: Can not attach to DLM (LMON up=[TRUE], DB mounted=[FALSE]).

      kjzdattdlm: Can not attach to DLM (LMON up=[TRUE], DB mounted=[FALSE]).

      kjzdattdlm: Can not attach to DLM (LMON up=[TRUE], DB mounted=[FALSE]).

      kjzdattdlm: Can not attach to DLM (LMON up=[TRUE], DB mounted=[FALSE]).

       

       

      *** 2014-06-17 19:42:52.879

      kjzdattdlm: Can not attach to DLM (LMON up=[TRUE], DB mounted=[FALSE]).

      kjzdattdlm: Can not attach to DLM (LMON up=[TRUE], DB mounted=[FALSE]).

      kjzdattdlm:A rcfg proposal from node 1 is received Can not attach to DLM (LMON up=[TRUE], DB mounted=[FALSE]).

      kjzdattdlm: Can not attach to DLM (LMON up=[TRUE], DB mounted=[FALSE]).

      kjzdattdlm: Can not attach to DLM (LMON up=[TRUE], DB mounted=[FALSE]).

       

       

      *** 2014-06-17 19:42:53.905

      DIAG attached to DLM

      2014-06-17 23:42:05.840: [ CSSCLNT]clsssRecvMsg: got a disconnect from the server while waiting for message type 1

       

       

      *** 2014-06-17 23:42:05.840

      2014-06-17 23:42:05.840: [ CSSCLNT]clssgsGroupGetStatus:  communications failed (0/3/-1)

      2014-06-17 23:42:05.840: [ CSSCLNT]clssgsGroupGetStatus: returning 8

      kgxgnpstat: received ABORT event from CLSS

      CM problem, please abort

       

       

      *** 2014-06-17 23:42:05.840

      Node monitor becomes unavailable for service

      2014-06-17 23:42:06.040: [ CSSCLNT]clsssRecvMsg: got a disconnect from the server while waiting for message type 1

      2014-06-17 23:42:06.040: [ CSSCLNT]clssgsGroupGetStatus:  communications failed (0/3/-1)

      2014-06-17 23:42:06.040: [ CSSCLNT]clssgsGroupGetStatus: returning 8

      kgxgnpstat: received ABORT event from CLSS

      CM problem, please abort

       

      Any idea???

       

      Thanks in advance!

       

       

      EDIT:

       

      In the event log of windows i find this accordint to the time when asm instances crash:

       

      TCP/IP failed to establish an outgoing connection because the selected local endpoint was recently used to connect to the same remote endpoint. This error typically occurs when outgoing connections are opened and closed at a high rate, causing all available local ports to be used and forcing TCP/IP to reuse a local port for an outgoing connection. To minimize the risk of data corruption, the TCP/IP standard requires a minimum time period to elapse between successive connections from a given local endpoint to a given remote endpoint.

       

      for now i've disabled all other processes not releated to oracle.

        • 1. Re: ASM instances restarting, all database restart too
          ora_tech

          Hi,

           

          Any recent changes on the system?

           

          Make sure private all IPS among cluster nodes are able to ping each other.

           

          Also refer MOS tech note:

           

           

           

          Troubleshooting ORA-27300 ORA-27301 ORA-27302 errors (Doc ID 579365.1)

           

          thanks,

          X A H E E R

          • 2. Re: ASM instances restarting, all database restart too
            user1138812

            ora_tech ha scritto:

             

            Hi,

             

            Any recent changes on the system?

             

            Make sure private all IPS among cluster nodes are able to ping each other.

             

            Also refer MOS tech note:

             

             

             

            Troubleshooting ORA-27300 ORA-27301 ORA-27302 errors (Doc ID 579365.1)

             

            thanks,

            X A H E E R

             

            The system is a brand new installation not yet in production.

             

            I've checked the note that you suggest but in this case i didn't find any suitable behaviour like this.

            All interconnects ip among the cluster nodes are able to ping each other.

            I've edited the post with this information that can be useful:

             

            In the event log of windows i find this accordint to the time when asm instances crash:

             

            TCP/IP failed to establish an outgoing connection because the selected local endpoint was recently used to connect to the same remote endpoint. This error typically occurs when outgoing connections are opened and closed at a high rate, causing all available local ports to be used and forcing TCP/IP to reuse a local port for an outgoing connection. To minimize the risk of data corruption, the TCP/IP standard requires a minimum time period to elapse between successive connections from a given local endpoint to a given remote endpoint.

             

            for now i've disabled all other processes not releated to oracle.

            • 3. Re: ASM instances restarting, all database restart too
              ora_tech

              Hi,

               

              What is speed of interconnect and what type of interfaces are used for interconnect?

               

              Switches for interconnect are shared with other networks too?

               

              regards,

              X A H E E R

              • 4. Re: ASM instances restarting, all database restart too
                user1138812

                Hi,

                 

                Here the interface specs:

                 

                Connection-specific DNS Suffix:

                Description: Broadcom BCM57840 NetXtreme II 10 Gigabit Ethernet (NDIS VBD Client) #85

                Physical Address: F8-DB-88-D4-41-B8

                DHCP Enabled: No

                IPv4 Address: 10.10.20.1

                IPv4 Subnet Mask: 255.255.255.240

                IPv4 Default Gateway:

                IPv4 DNS Server:

                IPv4 WINS Server:

                NetBIOS over Tcpip Enabled: Yes

                 

                The interconnect subnet 10.10.20.0 is only used for those 2 nodes.

                 

                The oracle nodes physically are 2 blades on a chassis that use 2 blade switches, then they are not dedicated but they are isolated on a dedicated vlan.

                 

                Thanks in advance!

                • 5. Re: ASM instances restarting, all database restart too
                  ora_tech

                  Hi,

                   

                  Interconnect configuration is OK.

                   

                  Could you run orachk utility to check overall status of your cluster.

                   

                  Download the latest version of orachk refer "ORAchk - Health Checks for the Oracle Stack (Doc ID 1268927.2)"

                   

                  thanks,

                  X A H E E R

                  • 6. Re: ASM instances restarting, all database restart too
                    user1138812

                    unfortunately this tool is not supported on windows 64 (server 2012)

                     

                    REQUIREMENTS (BACK TO TOP)

                    Supported Platforms

                    • Linux x86-64* (Enterprise Linux, RedHat and SuSE 9, SuSE 10 & SuSE 11)
                    • Oracle Solaris SPARC (Solaris 10 and 11)
                    • Oracle Solaris x86-64 (Solaris 10 and 11)
                    • AIX **
                    • HPUX**

                    * 32-bit platforms not supported, no planned support for Linux Itanium
                    **Requires BASH Shell 3.2 or higher to be installed

                    Supported Oracle Releases

                    • 10gR2
                    • 11gR1
                    • 11gR2
                    • 12cR1

                     

                    i think that is not possible to run this tool from a linux machine targeting the windows oracle rac...

                    • 7. Re: ASM instances restarting, all database restart too
                      ora_tech

                      Hi,

                       

                      Sorry i did not notice that this tool is not supported on windows.

                       

                      i think that is not possible to run this tool from a linux machine targeting the windows oracle rac..

                      No its not possible.

                       

                      Is there any specific reason you've opted windows for RAC. I faced many node eviction issues on windows compare to linux/unix bases OS.

                       

                      Please try to install OS watcher and check logs after running for 1 day atleast.

                       

                       

                       

                      OS Watcher For Windows (OSWFW) User Guide (Doc ID 433472.1)

                      How to get OSWatcher Analyzer to work on Windows Platform (Doc ID 1469970.1)

                       

                      How frequently nodes are getting evicted?

                       

                      regards,

                      X A H E E R

                      • 8. Re: ASM instances restarting, all database restart too
                        user1138812

                        The reason is that an application that will run on this cluster is not certified under linux (but working):

                         

                        the customer don't want to loose application suupport for this reason.....

                         

                        I know that oracle on linux is more stable due to os architecture and other.

                         

                        I'll install the tools that you suggest, the eviction append about every 7/8 hours.

                        All the asm instances crash and get restarted succesfully by the cluster.

                         

                        I'll let you know !

                         

                        Many Thanks!

                        • 9. Re: ASM instances restarting, all database restart too
                          ora_tech

                          Hi,

                           

                          Yes Install the tool and try to extract the logs when the node is evicting.

                           

                          Please share the logs after collection to investigate further.

                           

                          thanks,

                          X A H E E R

                          • 10. Re: ASM instances restarting, all database restart too
                            user1138812

                            reading the notes you linked i've found this:

                             

                            ABSTRACT

                            Note: OS Watcher for Windows is no longer supported.  Please use the tool Cluster Health Monitor instead.  Please see Document ID: 736752.1 for more information, and how to download, the Cluster Health Monitor.

                            OS Watcher for Windows (OSWFW) is several batch files that run the Windows utility logman and schtasks. The logman utility collects various Operating System counters. It archives these metrics to aid diagnosing performance and Operating System issues. OSWFW has segmented these counter collections into various categories. The schtasks utility is used to run a batch file that cleans up the archive files to keep only 24 hours of data. It is also used if Oracle's Real Application Cluster (RAC) is involved, to run a batch file that checks the RAC Interconnect. OSWFW can be downloaded from this note. Installation instructions for OSWFW are provided in this User Guide.

                             

                            then i've checked and the tool is istalled, then this is the output of this check 1 minutes before asm instance crashed down (14:21):

                             

                            ----------------------------------------

                            Node: inigfora01 Clock: '06-18-14 14.20.17' SerialNo:3011

                            ----------------------------------------

                             

                             

                            SYSTEM:

                            #pcpus: 6 #vcpus: 12 cpuht: Y chipname: PROCESSOR_AMD_X8664 cpu: 10.81 cpuq: 0 physmemfree: 59914484 physmemtotal: 67062312 mcache: 218284 swapfree: 8704 swaptotal: 8704 ior: 80 iow: 83 ios: 16 swpin: 0 swpout: 0 pgin: 3 pgout: 0 netr: 93.234 netw: 21.257 procs: 120 rtprocs: 0 #fds: 44416 #sysfdlimit: 4294967295 #disks: 10 #nics: 4  nicErrors: 0

                             

                             

                            TOP CONSUMERS:

                            topcpu: 'svchost.exe(448) 44.3' topprivmem: 'oracle.exe(5640) 3434712' topworkingset: 'oracle.exe(5640) 959148' topfd: 'dns.exe(1776) 5245' topthread: 'System(4) 191'

                             

                             

                            PROCESSES:

                             

                             

                            name: 'svchost.exe' pid: 448 #procfdlimit: 4294967295 cpuusage: 44.3 privmem: 53864 workingset: 70308 #fd: 1822 #threads: 45 priority: 8 nice: 0

                            name: 'WmiPrvSE.exe' pid: 3552 #procfdlimit: 4294967295 cpuusage: 27.79 privmem: 12936 workingset: 20336 #fd: 241 #threads: 9 priority: 8 nice: 0

                            name: 'WmiPrvSE.exe' pid: 3544 #procfdlimit: 4294967295 cpuusage: 25.92 privmem: 33460 workingset: 39056 #fd: 395 #threads: 14 priority: 8 nice: 0

                            name: 'System' pid: 4 #procfdlimit: 4294967295 cpuusage: 16.55 privmem: 124 workingset: 260 #fd: 1314 #threads: 191 priority: 8 nice: 0

                            name: 'svchost.exe' pid: 1120 #procfdlimit: 4294967295 cpuusage: 1.24 privmem: 21000 workingset: 25948 #fd: 628 #threads: 21 priority: 8 nice: 0

                            name: 'orarootagent.exe' pid: 7140 #procfdlimit: 4294967295 cpuusage: 0.93 privmem: 20392 workingset: 31748 #fd: 329 #threads: 17 priority: 8 nice: 0

                            name: 'ocssd.exe' pid: 4332 #procfdlimit: 4294967295 cpuusage: 0.62 privmem: 38900 workingset: 44820 #fd: 555 #threads: 19 priority: 13 nice: 0

                            name: 'WmiApSrv.exe' pid: 8520 #procfdlimit: 4294967295 cpuusage: 0.62 privmem: 1652 workingset: 5484 #fd: 124 #threads: 5 priority: 8 nice: 0

                            name: 'oraagent.exe' pid: 6720 #procfdlimit: 4294967295 cpuusage: 0.62 privmem: 42052 workingset: 47188 #fd: 517 #threads: 29 priority: 8 nice: 0

                            name: 'services.exe' pid: 656 #procfdlimit: 4294967295 cpuusage: 0.31 privmem: 8912 workingset: 12648 #fd: 380 #threads: 7 priority: 9 nice: 0

                            name: 'oracle.exe' pid: 5640 #procfdlimit: 4294967295 cpuusage: 0.31 privmem: 3434712 workingset: 959148 #fd: 1155 #threads: 48 priority: 8 nice: 0

                            name: 'oracle.exe' pid: 2228 #procfdlimit: 4294967295 cpuusage: 0.31 privmem: 1874036 workingset: 738828 #fd: 1065 #threads: 50 priority: 8 nice: 0

                            name: 'gpnpd.exe' pid: 5300 #procfdlimit: 4294967295 cpuusage: 0.31 privmem: 23632 workingset: 28908 #fd: 337 #threads: 6 priority: 8 nice: 0

                            name: 'nscp.exe' pid: 1920 #procfdlimit: 4294967295 cpuusage: 0.31 privmem: 9780 workingset: 21100 #fd: 274 #threads: 29 priority: 8 nice: 0

                            name: 'csrss.exe' pid: 508 #procfdlimit: 4294967295 cpuusage: 0.31 privmem: 3152 workingset: 5392 #fd: 672 #threads: 11 priority: 13 nice: 0

                            name: 'oracle.exe' pid: 8444 #procfdlimit: 4294967295 cpuusage: 0.31 privmem: 812808 workingset: 365024 #fd: 808 #threads: 35 priority: 8 nice: 0

                            name: 'svchost.exe' pid: 876 #procfdlimit: 4294967295 cpuusage: 0.31 privmem: 4208 workingset: 8184 #fd: 392 #threads: 7 priority: 8 nice: 0

                            name: 'lsass.exe' pid: 684 #procfdlimit: 4294967295 cpuusage: 0.31 privmem: 9748 workingset: 18904 #fd: 1179 #threads: 8 priority: 9 nice: 0

                            name: 'oraagent.exe' pid: 8812 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 54304 workingset: 44844 #fd: 537 #threads: 28 priority: 8 nice: 0

                            name: 'conhost.exe' pid: 8432 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 560 workingset: 2444 #fd: 30 #threads: 1 priority: 8 nice: 0

                            name: 'octssd.exe' pid: 5008 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 21564 workingset: 28188 #fd: 1318 #threads: 12 priority: 8 nice: 0

                            name: 'conhost.exe' pid: 6832 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 560 workingset: 2444 #fd: 30 #threads: 1 priority: 8 nice: 0

                            name: 'orarootagent.exe' pid: 7952 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 40756 workingset: 34976 #fd: 375 #threads: 20 priority: 8 nice: 0

                            name: 'dns.exe' pid: 1776 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 244332 workingset: 233368 #fd: 5245 #threads: 33 priority: 8 nice: 0

                            name: 'conhost.exe' pid: 7104 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 580 workingset: 2484 #fd: 30 #threads: 1 priority: 8 nice: 0

                            name: 'mDNSResponder.exe' pid: 8248 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 19188 workingset: 23668 #fd: 320 #threads: 13 priority: 8 nice: 0

                            name: 'conhost.exe' pid: 6376 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 568 workingset: 2448 #fd: 30 #threads: 1 priority: 8 nice: 0

                            name: 'cssdagent.exe' pid: 4424 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 17276 workingset: 24920 #fd: 243 #threads: 13 priority: 13 nice: 0

                            name: 'conhost.exe' pid: 9620 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 564 workingset: 2444 #fd: 30 #threads: 1 priority: 8 nice: 0

                            name: 'cssdmonitor.exe' pid: 5076 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 17172 workingset: 24832 #fd: 243 #threads: 13 priority: 13 nice: 0

                            name: 'java.exe' pid: 9144 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 177712 workingset: 180132 #fd: 945 #threads: 51 priority: 8 nice: 0

                            name: 'cmd.exe' pid: 5040 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 1612 workingset: 2236 #fd: 28 #threads: 1 priority: 8 nice: 0

                            name: 'dllhost.exe' pid: 8412 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 3776 workingset: 10772 #fd: 188 #threads: 11 priority: 8 nice: 0

                            name: 'conhost.exe' pid: 8188 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 1720 workingset: 10096 #fd: 50 #threads: 2 priority: 8 nice: 0

                            name: 'cmd.exe' pid: 9068 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 1640 workingset: 2712 #fd: 20 #threads: 1 priority: 8 nice: 0

                            name: 'TNSLSNR.EXE' pid: 7760 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 17820 workingset: 21760 #fd: 214 #threads: 6 priority: 8 nice: 0

                            name: 'TNSLSNR.EXE' pid: 5276 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 17312 workingset: 21592 #fd: 213 #threads: 6 priority: 8 nice: 0

                            name: 'TNSLSNR.EXE' pid: 8052 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 17604 workingset: 21576 #fd: 213 #threads: 6 priority: 8 nice: 0

                            name: 'ons.exe' pid: 7960 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 3752 workingset: 6412 #fd: 134 #threads: 15 priority: 8 nice: 0

                            name: 'ons.exe' pid: 7956 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 2492 workingset: 4804 #fd: 78 #threads: 1 priority: 8 nice: 0

                            name: 'java.exe' pid: 7420 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 204516 workingset: 130636 #fd: 820 #threads: 42 priority: 8 nice: 0

                            name: 'cmd.exe' pid: 7460 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 1492 workingset: 1844 #fd: 25 #threads: 1 priority: 8 nice: 0

                            name: 'conhost.exe' pid: 1416 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 572 workingset: 2528 #fd: 31 #threads: 1 priority: 8 nice: 0

                            name: 'RemoteExecService.exe' pid: 1064 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 1720 workingset: 4456 #fd: 61 #threads: 3 priority: 8 nice: 0

                            name: 'jusched.exe' pid: 6904 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 2412 workingset: 8928 #fd: 325 #threads: 4 priority: 8 nice: 0

                            name: 'msseces.exe' pid: 3348 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 4292 workingset: 14452 #fd: 305 #threads: 7 priority: 8 nice: 0

                            name: 'BACSTray.exe' pid: 6292 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 1080 workingset: 4976 #fd: 57 #threads: 1 priority: 8 nice: 0

                            name: 'BrowseMark.BrowserAdapter.exe' pid: 6420 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 6848 workingset: 19732 #fd: 287 #threads: 9 priority: 8 nice: 0

                            name: 'svchost.exe' pid: 6224 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 1060 workingset: 4360 #fd: 84 #threads: 2 priority: 8 nice: 0

                            name: 'explorer.exe' pid: 4768 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 40556 workingset: 88260 #fd: 1164 #threads: 34 priority: 8 nice: 0

                            name: 'rdpclip.exe' pid: 5260 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 1832 workingset: 7276 #fd: 195 #threads: 6 priority: 8 nice: 0

                            name: 'taskhostex.exe' pid: 4696 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 7972 workingset: 10912 #fd: 211 #threads: 7 priority: 8 nice: 0

                            name: 'dwm.exe' pid: 6108 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 15464 workingset: 42260 #fd: 167 #threads: 8 priority: 13 nice: 0

                            name: 'winlogon.exe' pid: 3008 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 1192 workingset: 4816 #fd: 141 #threads: 3 priority: 13 nice: 0

                            name: 'csrss.exe' pid: 4724 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 1936 workingset: 23296 #fd: 285 #threads: 11 priority: 13 nice: 0

                            name: 'conhost.exe' pid: 6052 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 564 workingset: 2456 #fd: 31 #threads: 1 priority: 8 nice: 0

                            name: 'ologgerd.exe' pid: 5856 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 24984 workingset: 36384 #fd: 310 #threads: 12 priority: 13 nice: 0

                            name: 'conhost.exe' pid: 6128 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 560 workingset: 2452 #fd: 30 #threads: 1 priority: 8 nice: 0

                            name: 'nvstatsmngr.exe' pid: 2144 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 2956 workingset: 6108 #fd: 87 #threads: 2 priority: 8 nice: 0

                            name: 'msdtc.exe' pid: 756 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 2880 workingset: 7160 #fd: 150 #threads: 10 priority: 8 nice: 0

                            name: 'conhost.exe' pid: 6100 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 560 workingset: 2440 #fd: 30 #threads: 1 priority: 8 nice: 0

                            name: 'osysmond.exe' pid: 5688 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 36740 workingset: 43792 #fd: 533 #threads: 17 priority: 13 nice: 0

                            name: 'conhost.exe' pid: 6184 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 564 workingset: 2424 #fd: 30 #threads: 1 priority: 8 nice: 0

                            name: 'scriptagent.exe' pid: 10188 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 16480 workingset: 24040 #fd: 203 #threads: 9 priority: 8 nice: 0

                            name: 'conhost.exe' pid: 2224 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 568 workingset: 2448 #fd: 30 #threads: 1 priority: 8 nice: 0

                            name: 'dwm.exe' pid: 964 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 20800 workingset: 32924 #fd: 168 #threads: 5 priority: 13 nice: 0

                            name: 'dsm_om_shrsvc64.exe' pid: 1236 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 1816 workingset: 6320 #fd: 148 #threads: 9 priority: 8 nice: 0

                            name: 'BrowseMark.PurBrowse64.exe' pid: 5200 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 2292 workingset: 4292 #fd: 76 #threads: 2 priority: 8 nice: 0

                            name: 'svchost.exe' pid: 2888 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 9012 workingset: 15276 #fd: 551 #threads: 55 priority: 8 nice: 0

                            name: 'dsm_om_connsvc64.exe' pid: 4100 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 189244 workingset: 140152 #fd: 415 #threads: 31 priority: 8 nice: 0

                            name: 'emagent.exe' pid: 3984 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 35728 workingset: 44980 #fd: 497 #threads: 12 priority: 8 nice: 0

                            name: 'nvpmgr.exe' pid: 1804 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 3320 workingset: 6740 #fd: 152 #threads: 7 priority: 8 nice: 0

                            name: 'winlogon.exe' pid: 640 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 1140 workingset: 5248 #fd: 111 #threads: 3 priority: 13 nice: 0

                            name: 'perl.exe' pid: 1512 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 16356 workingset: 20768 #fd: 35 #threads: 1 priority: 8 nice: 0

                            name: 'cmd.exe' pid: 1580 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 1688 workingset: 2336 #fd: 22 #threads: 1 priority: 8 nice: 0

                            name: 'vmware-usbarbitrator64.exe' pid: 2664 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 1912 workingset: 6740 #fd: 147 #threads: 3 priority: 8 nice: 0

                            name: 'svchost.exe' pid: 2992 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 4448 workingset: 8516 #fd: 149 #threads: 23 priority: 8 nice: 0

                            name: 'vmware-converter.exe' pid: 2508 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 14012 workingset: 27876 #fd: 168 #threads: 7 priority: 8 nice: 0

                            name: 'vmware-converter.exe' pid: 2376 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 19388 workingset: 34060 #fd: 216 #threads: 12 priority: 8 nice: 0

                            name: 'vmware-converter-a.exe' pid: 2256 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 13444 workingset: 26476 #fd: 173 #threads: 7 priority: 8 nice: 0

                            name: 'utilBrowseMark.exe' pid: 2920 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 52076 workingset: 52224 #fd: 709 #threads: 20 priority: 8 nice: 0

                            name: 'updateBrowseMark.exe' pid: 2688 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 16280 workingset: 17528 #fd: 461 #threads: 8 priority: 8 nice: 0

                            name: 'svchost.exe' pid: 2708 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 15884 workingset: 15868 #fd: 437 #threads: 20 priority: 8 nice: 0

                            name: 'SMagent.exe' pid: 2844 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 35564 workingset: 17452 #fd: 250 #threads: 17 priority: 8 nice: 0

                            name: 'Vizioncore.PhysicalAppliance.Ser' pid: 2452 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 1412 workingset: 6172 #fd: 75 #threads: 4 priority: 8 nice: 0

                            name: 'svchost.exe' pid: 2176 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 1132 workingset: 4276 #fd: 105 #threads: 4 priority: 8 nice: 0

                            name: 'ohasd.exe' pid: 2212 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 46104 workingset: 56848 #fd: 2553 #threads: 34 priority: 8 nice: 0

                            name: 'conhost.exe' pid: 2288 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 628 workingset: 2668 #fd: 30 #threads: 2 priority: 8 nice: 0

                            name: 'nvwcshadower.exe' pid: 2280 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 2636 workingset: 5688 #fd: 125 #threads: 5 priority: 8 nice: 0

                            name: 'conhost.exe' pid: 2156 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 628 workingset: 2640 #fd: 30 #threads: 2 priority: 8 nice: 0

                            name: 'svchost.exe' pid: 1020 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 16376 workingset: 19864 #fd: 566 #threads: 15 priority: 8 nice: 0

                            name: 'conhost.exe' pid: 2120 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 640 workingset: 2704 #fd: 36 #threads: 2 priority: 8 nice: 0

                            name: 'nmesrvc.exe' pid: 2080 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 824 workingset: 2884 #fd: 45 #threads: 3 priority: 8 nice: 0

                            name: 'conhost.exe' pid: 6412 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 572 workingset: 2432 #fd: 30 #threads: 1 priority: 8 nice: 0

                            name: 'LogonUI.exe' pid: 956 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 8444 workingset: 25884 #fd: 303 #threads: 11 priority: 13 nice: 0

                            name: 'WmiPrvSE.exe' pid: 6868 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 4452 workingset: 8356 #fd: 195 #threads: 6 priority: 8 nice: 0

                            name: 'MsMpEng.exe' pid: 916 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 2904 workingset: 9528 #fd: 401 #threads: 11 priority: 8 nice: 0

                            name: 'evmlogger.exe' pid: 1420 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 20480 workingset: 17320 #fd: 196 #threads: 1 priority: 8 nice: 0

                            name: 'dsm_sa_datamgr64.exe' pid: 1752 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 12936 workingset: 18408 #fd: 337 #threads: 17 priority: 8 nice: 0

                            name: 'dsm_sa_eventmgr64.exe' pid: 1716 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 2552 workingset: 7540 #fd: 148 #threads: 6 priority: 8 nice: 0

                            name: 'svchost.exe' pid: 1596 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 3184 workingset: 7380 #fd: 95 #threads: 7 priority: 8 nice: 0

                            name: 'spoolsv.exe' pid: 1556 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 3364 workingset: 9020 #fd: 337 #threads: 12 priority: 8 nice: 0

                            name: 'svchost.exe' pid: 1328 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 9124 workingset: 11744 #fd: 372 #threads: 17 priority: 8 nice: 0

                            name: 'svchost.exe' pid: 792 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 2884 workingset: 8804 #fd: 374 #threads: 7 priority: 8 nice: 0

                            name: 'svchost.exe' pid: 904 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 8004 workingset: 13936 #fd: 739 #threads: 16 priority: 8 nice: 0

                            name: 'csrss.exe' pid: 576 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 1360 workingset: 3572 #fd: 82 #threads: 9 priority: 13 nice: 0

                            name: 'conhost.exe' pid: 5956 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 564 workingset: 2440 #fd: 30 #threads: 1 priority: 8 nice: 0

                            name: 'conhost.exe' pid: 5208 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 620 workingset: 2712 #fd: 30 #threads: 2 priority: 8 nice: 0

                            name: 'conhost.exe' pid: 5484 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 564 workingset: 2428 #fd: 31 #threads: 1 priority: 8 nice: 0

                            name: 'conhost.exe' pid: 8912 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 572 workingset: 2432 #fd: 30 #threads: 1 priority: 8 nice: 0

                            name: 'wininit.exe' pid: 584 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 888 workingset: 3568 #fd: 82 #threads: 2 priority: 13 nice: 0

                            name: 'OracleOBJService.exe' pid: 832 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 10284 workingset: 14304 #fd: 159 #threads: 4 priority: 8 nice: 0

                            name: 'smss.exe' pid: 380 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 384 workingset: 1076 #fd: 53 #threads: 3 priority: 11 nice: 0

                            name: 'omtsreco.exe' pid: 2100 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 10244 workingset: 16768 #fd: 175 #threads: 5 priority: 8 nice: 0

                            name: 'conhost.exe' pid: 5744 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 564 workingset: 2412 #fd: 30 #threads: 1 priority: 8 nice: 0

                            name: 'crsd.exe' pid: 9584 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 45560 workingset: 53448 #fd: 1660 #threads: 43 priority: 8 nice: 0

                            name: 'gipcd.exe' pid: 5872 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 14256 workingset: 17712 #fd: 233 #threads: 3 priority: 8 nice: 0

                            name: 'evmd.exe' pid: 3564 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 37784 workingset: 35704 #fd: 1491 #threads: 17 priority: 8 nice: 0

                            name: 'conhost.exe' pid: 9748 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 560 workingset: 2420 #fd: 30 #threads: 1 priority: 8 nice: 0

                            DEVICES:

                            9 ior: 0.999 iow: 0.499 ios: 2 qlen: 0 wait: 0 type: VOTING[ONLINE]

                            0_C:_D: ior: 12.797 iow: 76.984 ios: 8 qlen: 0 wait: 0 type: SWAP

                            D: ior: 12.797 iow: 69.686 ios: 6 qlen: 0 wait: 0 type: SYS

                            C: ior: 0.000 iow: 7.298 ios: 1 qlen: 0 wait: 0 type: SWAP

                            8 ior: 0.000 iow: 0.000 ios: 0 qlen: 0 wait: 0 type: SYS

                            7 ior: 31.993 iow: 0.000 ios: 1 qlen: 0 wait: 0 type: SYS

                            6 ior: 6.398 iow: 3.199 ios: 0 qlen: 0 wait: 0 type: SYS

                            5 ior: 28.794 iow: 3.199 ios: 1 qlen: 0 wait: 0 type: SYS

                            4 ior: 0.000 iow: 0.000 ios: 0 qlen: 0 wait: 0 type: SYS

                            3 ior: 0.000 iow: 0.000 ios: 0 qlen: 0 wait: 0 type: SYS

                            2 ior: 0.000 iow: 0.000 ios: 0 qlen: 0 wait: 0 type: SYS

                            1 ior: 0.000 iow: 0.000 ios: 0 qlen: 0 wait: 0 type: SYS

                            NICS:

                            Broadcom BCM57840 NetXtreme II 10 Gigabit Ethernet (NDIS VBD Client) #84 netrr: 1.160  netwr: 0.323  neteff: 1.483  nicerrors: 0 pktsin: 11  pktsout: 2  errsin: 0  errsout: 0  indiscarded: 0  outdiscarded: 0  inunicast: 2  innonunicast: 8  type: PUBLIC

                            Broadcom BCM57840 NetXtreme II 10 Gigabit Ethernet (NDIS VBD Client) #85 netrr: 6.676  netwr: 8.864  neteff: 15.540  nicerrors: 0 pktsin: 20  pktsout: 21  errsin: 0  errsout: 0  indiscarded: 0  outdiscarded: 0  inunicast: 20  innonunicast: 0  type: PRIVATE

                            Broadcom BCM57840 NetXtreme II 10 Gigabit Ethernet (NDIS VBD Client) #82 netrr: 0.012  netwr: 0.000  neteff: 0.012  nicerrors: 0 pktsin: 0  pktsout: 0  errsin: 0  errsout: 0  indiscarded: 0  outdiscarded: 0  inunicast: 0  innonunicast: 0  type: PUBLIC

                            Broadcom BCM57840 NetXtreme II 10 Gigabit Ethernet (NDIS VBD Client) #83 netrr: 85.385  netwr: 12.069  neteff: 97.454  nicerrors: 0 pktsin: 65  pktsout: 27  errsin: 0  errsout: 0  indiscarded: 0  outdiscarded: 0  inunicast: 65  innonunicast: 0  type: PUBLIC

                             

                             

                            FILESYSTEMS:

                            name: Logical Disk mount: D: type: NTFS total: 247289852 used: 54090948 available: 193198904 used%: 21 mft%: -1 [ORACLE_HOME ]

                             

                             

                            name: Logical Disk mount: C: type: NTFS total: 41943036 used: 36498336 available: 5444700 used%: 87 mft%: -1 []

                             

                             

                            PROTOCOL ERRORS:

                            IPHdrErr: 0 IPAddrErr: 4864 IPUnkProto: 0 IPReasFail: 0 IPFragFail: 0 TCPFailedConn: 3679 TCPEstRst: 8652 TCPRetraSeg: 9184 UDPUnkPort: 10544 UDPRcvErr: 0

                             

                             

                            ----------------------------------------

                            Node: inigfora02 Clock: '06-18-14 14.20.20' SerialNo:2615

                            ----------------------------------------

                             

                             

                            SYSTEM:

                            #pcpus: 6 #vcpus: 12 cpuht: Y chipname: PROCESSOR_AMD_X8664 cpu: 10.23 cpuq: 0 physmemfree: 60218392 physmemtotal: 67062312 mcache: 139337 swapfree: 8704 swaptotal: 8704 ior: 100 iow: 60 ios: 10 swpin: 0 swpout: 0 pgin: 4 pgout: 0 netr: 97.465 netw: 32.518 procs: 92 rtprocs: 0 #fds: 31812 #sysfdlimit: 4294967295 #disks: 6 #nics: 4  nicErrors: 0

                             

                             

                            TOP CONSUMERS:

                            topcpu: 'svchost.exe(452) 47.76' topprivmem: 'oracle.exe(5584) 3528720' topworkingset: 'oracle.exe(5584) 1437676' topfd: 'ohasd.exe(1032) 2573' topthread: 'System(4) 170'

                             

                             

                            PROCESSES:

                             

                             

                            name: 'svchost.exe' pid: 452 #procfdlimit: 4294967295 cpuusage: 47.76 privmem: 28588 workingset: 42964 #fd: 1578 #threads: 42 priority: 8 nice: 0

                            name: 'WmiPrvSE.exe' pid: 2120 #procfdlimit: 4294967295 cpuusage: 28.40 privmem: 10880 workingset: 17696 #fd: 212 #threads: 7 priority: 8 nice: 0

                            name: 'WmiPrvSE.exe' pid: 2104 #procfdlimit: 4294967295 cpuusage: 18.73 privmem: 11076 workingset: 16660 #fd: 233 #threads: 9 priority: 8 nice: 0

                            name: 'System' pid: 4 #procfdlimit: 4294967295 cpuusage: 16.85 privmem: 124 workingset: 256 #fd: 1126 #threads: 170 priority: 8 nice: 0

                            name: 'svchost.exe' pid: 1084 #procfdlimit: 4294967295 cpuusage: 1.24 privmem: 20276 workingset: 24936 #fd: 595 #threads: 22 priority: 8 nice: 0

                            name: 'orarootagent.exe' pid: 5140 #procfdlimit: 4294967295 cpuusage: 0.93 privmem: 21176 workingset: 33240 #fd: 305 #threads: 16 priority: 8 nice: 0

                            name: 'ocssd.exe' pid: 5012 #procfdlimit: 4294967295 cpuusage: 0.93 privmem: 41044 workingset: 47008 #fd: 553 #threads: 19 priority: 13 nice: 0

                            name: 'svchost.exe' pid: 1016 #procfdlimit: 4294967295 cpuusage: 0.31 privmem: 11400 workingset: 14680 #fd: 403 #threads: 12 priority: 8 nice: 0

                            name: 'osysmond.exe' pid: 4692 #procfdlimit: 4294967295 cpuusage: 0.31 privmem: 34568 workingset: 41956 #fd: 517 #threads: 17 priority: 13 nice: 0

                            name: 'oraagent.exe' pid: 4380 #procfdlimit: 4294967295 cpuusage: 0.31 privmem: 53868 workingset: 44836 #fd: 546 #threads: 28 priority: 8 nice: 0

                            name: 'oracle.exe' pid: 472 #procfdlimit: 4294967295 cpuusage: 0.31 privmem: 1938480 workingset: 902588 #fd: 1123 #threads: 52 priority: 8 nice: 0

                            name: 'oracle.exe' pid: 5584 #procfdlimit: 4294967295 cpuusage: 0.31 privmem: 3528720 workingset: 1437676 #fd: 1360 #threads: 59 priority: 8 nice: 0

                            name: 'oraagent.exe' pid: 5156 #procfdlimit: 4294967295 cpuusage: 0.31 privmem: 40668 workingset: 48428 #fd: 547 #threads: 27 priority: 8 nice: 0

                            name: 'winlogon.exe' pid: 3452 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 1268 workingset: 4860 #fd: 141 #threads: 3 priority: 13 nice: 0

                            name: 'conhost.exe' pid: 1344 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 568 workingset: 2448 #fd: 30 #threads: 1 priority: 8 nice: 0

                            name: 'evmlogger.exe' pid: 3700 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 20520 workingset: 17372 #fd: 196 #threads: 1 priority: 8 nice: 0

                            name: 'mmc.exe' pid: 3904 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 12680 workingset: 29420 #fd: 315 #threads: 9 priority: 8 nice: 0

                            name: 'conhost.exe' pid: 3908 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 572 workingset: 2456 #fd: 30 #threads: 1 priority: 8 nice: 0

                            name: 'crsd.exe' pid: 704 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 60196 workingset: 71120 #fd: 1742 #threads: 47 priority: 8 nice: 0

                            name: 'conhost.exe' pid: 1768 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 572 workingset: 2444 #fd: 30 #threads: 1 priority: 8 nice: 0

                            name: 'ologgerd.exe' pid: 2852 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 21532 workingset: 30912 #fd: 245 #threads: 6 priority: 13 nice: 0

                            name: 'oracle.exe' pid: 1104 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 815348 workingset: 368372 #fd: 835 #threads: 34 priority: 8 nice: 0

                            name: 'conhost.exe' pid: 4316 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 568 workingset: 2448 #fd: 30 #threads: 1 priority: 8 nice: 0

                            name: 'evmd.exe' pid: 1212 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 37544 workingset: 35856 #fd: 1502 #threads: 17 priority: 8 nice: 0

                            name: 'conhost.exe' pid: 3144 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 568 workingset: 2432 #fd: 31 #threads: 1 priority: 8 nice: 0

                            name: 'octssd.exe' pid: 4300 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 21912 workingset: 28508 #fd: 1327 #threads: 13 priority: 8 nice: 0

                            name: 'conhost.exe' pid: 5020 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 564 workingset: 2444 #fd: 30 #threads: 1 priority: 8 nice: 0

                            name: 'SMagent.exe' pid: 2700 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 35620 workingset: 17608 #fd: 249 #threads: 17 priority: 8 nice: 0

                            name: 'conhost.exe' pid: 4940 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 568 workingset: 2440 #fd: 30 #threads: 1 priority: 8 nice: 0

                            name: 'cssdagent.exe' pid: 4932 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 17488 workingset: 25376 #fd: 251 #threads: 13 priority: 13 nice: 0

                            name: 'conhost.exe' pid: 4776 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 564 workingset: 2440 #fd: 30 #threads: 1 priority: 8 nice: 0

                            name: 'cssdmonitor.exe' pid: 4768 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 17220 workingset: 25088 #fd: 245 #threads: 13 priority: 13 nice: 0

                            name: 'conhost.exe' pid: 4700 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 568 workingset: 2444 #fd: 30 #threads: 1 priority: 8 nice: 0

                            name: 'conhost.exe' pid: 904 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 628 workingset: 2640 #fd: 30 #threads: 2 priority: 8 nice: 0

                            name: 'conhost.exe' pid: 4620 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 572 workingset: 2432 #fd: 30 #threads: 1 priority: 8 nice: 0

                            name: 'gipcd.exe' pid: 4612 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 14260 workingset: 17596 #fd: 234 #threads: 3 priority: 8 nice: 0

                            name: 'conhost.exe' pid: 4600 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 564 workingset: 2436 #fd: 30 #threads: 1 priority: 8 nice: 0

                            name: 'orarootagent.exe' pid: 4588 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 40832 workingset: 35572 #fd: 401 #threads: 21 priority: 8 nice: 0

                            name: 'conhost.exe' pid: 4504 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 572 workingset: 2448 #fd: 30 #threads: 1 priority: 8 nice: 0

                            name: 'gpnpd.exe' pid: 4496 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 22744 workingset: 28440 #fd: 324 #threads: 6 priority: 8 nice: 0

                            name: 'conhost.exe' pid: 4448 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 580 workingset: 2468 #fd: 30 #threads: 1 priority: 8 nice: 0

                            name: 'mDNSResponder.exe' pid: 4440 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 19236 workingset: 23556 #fd: 327 #threads: 16 priority: 8 nice: 0

                            name: 'conhost.exe' pid: 4388 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 572 workingset: 2440 #fd: 30 #threads: 1 priority: 8 nice: 0

                            name: 'dsm_om_shrsvc64.exe' pid: 1840 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 1820 workingset: 6324 #fd: 148 #threads: 9 priority: 8 nice: 0

                            name: 'msdtc.exe' pid: 1800 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 2712 workingset: 6960 #fd: 158 #threads: 10 priority: 8 nice: 0

                            name: 'ohasd.exe' pid: 1032 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 46392 workingset: 56964 #fd: 2573 #threads: 35 priority: 8 nice: 0

                            name: 'jusched.exe' pid: 3228 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 1032 workingset: 4176 #fd: 65 #threads: 1 priority: 8 nice: 0

                            name: 'msseces.exe' pid: 2676 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 4032 workingset: 13292 #fd: 300 #threads: 7 priority: 8 nice: 0

                            name: 'svchost.exe' pid: 864 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 3012 workingset: 6516 #fd: 324 #threads: 8 priority: 8 nice: 0

                            name: 'explorer.exe' pid: 1968 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 36996 workingset: 85240 #fd: 1259 #threads: 34 priority: 8 nice: 0

                            name: 'rdpclip.exe' pid: 4060 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 1692 workingset: 6668 #fd: 190 #threads: 6 priority: 8 nice: 0

                            name: 'taskhostex.exe' pid: 2436 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 7360 workingset: 8412 #fd: 190 #threads: 7 priority: 8 nice: 0

                            name: 'OracleOBJService.exe' pid: 820 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 9920 workingset: 14036 #fd: 159 #threads: 4 priority: 8 nice: 0

                            name: 'spoolsv.exe' pid: 1544 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 3124 workingset: 8808 #fd: 334 #threads: 10 priority: 8 nice: 0

                            name: 'csrss.exe' pid: 3472 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 2028 workingset: 24968 #fd: 195 #threads: 9 priority: 13 nice: 0

                            name: 'svchost.exe' pid: 2564 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 7360 workingset: 14368 #fd: 536 #threads: 54 priority: 8 nice: 0

                            name: 'dsm_om_connsvc64.exe' pid: 2068 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 191668 workingset: 137824 #fd: 415 #threads: 31 priority: 8 nice: 0

                            name: 'svchost.exe' pid: 780 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 2764 workingset: 8348 #fd: 369 #threads: 7 priority: 8 nice: 0

                            name: 'svchost.exe' pid: 796 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 5352 workingset: 10324 #fd: 419 #threads: 15 priority: 8 nice: 0

                            name: 'svchost.exe' pid: 2732 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 14896 workingset: 12164 #fd: 304 #threads: 15 priority: 8 nice: 0

                            name: 'lsass.exe' pid: 672 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 7692 workingset: 16736 #fd: 1083 #threads: 8 priority: 9 nice: 0

                            name: 'Vizioncore.PhysicalAppliance.Ser' pid: 2632 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 1424 workingset: 6184 #fd: 75 #threads: 4 priority: 8 nice: 0

                            name: 'svchost.exe' pid: 2204 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 1116 workingset: 4296 #fd: 106 #threads: 4 priority: 8 nice: 0

                            name: 'services.exe' pid: 644 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 4816 workingset: 8656 #fd: 283 #threads: 6 priority: 9 nice: 0

                            name: 'nvwcshadower.exe' pid: 2036 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 2632 workingset: 5656 #fd: 125 #threads: 5 priority: 8 nice: 0

                            name: 'omtsreco.exe' pid: 1912 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 10240 workingset: 16784 #fd: 169 #threads: 4 priority: 8 nice: 0

                            name: 'conhost.exe' pid: 1904 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 636 workingset: 2628 #fd: 30 #threads: 2 priority: 8 nice: 0

                            name: 'nvstatsmngr.exe' pid: 1868 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 2952 workingset: 6100 #fd: 87 #threads: 2 priority: 8 nice: 0

                            name: 'MsMpEng.exe' pid: 908 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 2912 workingset: 9496 #fd: 401 #threads: 11 priority: 8 nice: 0

                            name: 'nscp.exe' pid: 1736 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 9508 workingset: 20892 #fd: 268 #threads: 29 priority: 8 nice: 0

                            name: 'nvpmgr.exe' pid: 1652 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 3292 workingset: 6704 #fd: 152 #threads: 7 priority: 8 nice: 0

                            name: 'TNSLSNR.EXE' pid: 5752 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 17720 workingset: 21520 #fd: 214 #threads: 6 priority: 8 nice: 0

                            name: 'dsm_sa_eventmgr64.exe' pid: 1588 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 2512 workingset: 7492 #fd: 148 #threads: 6 priority: 8 nice: 0

                            name: 'wininit.exe' pid: 572 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 888 workingset: 3560 #fd: 82 #threads: 2 priority: 13 nice: 0

                            name: 'svchost.exe' pid: 1364 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 8248 workingset: 10348 #fd: 348 #threads: 17 priority: 8 nice: 0

                            name: 'smss.exe' pid: 380 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 312 workingset: 1016 #fd: 53 #threads: 3 priority: 11 nice: 0

                            name: 'csrss.exe' pid: 564 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 1908 workingset: 3920 #fd: 83 #threads: 9 priority: 13 nice: 0

                            name: 'conhost.exe' pid: 4228 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 640 workingset: 2732 #fd: 30 #threads: 2 priority: 8 nice: 0

                            name: 'csrss.exe' pid: 492 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 2012 workingset: 4496 #fd: 572 #threads: 10 priority: 13 nice: 0

                            name: 'dwm.exe' pid: 960 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 20848 workingset: 32860 #fd: 166 #threads: 5 priority: 13 nice: 0

                            name: 'LogonUI.exe' pid: 952 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 8172 workingset: 25256 #fd: 300 #threads: 11 priority: 13 nice: 0

                            name: 'winlogon.exe' pid: 628 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 1136 workingset: 5232 #fd: 110 #threads: 3 priority: 13 nice: 0

                            name: 'ons.exe' pid: 5896 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 3812 workingset: 6452 #fd: 134 #threads: 16 priority: 8 nice: 0

                            name: 'conhost.exe' pid: 5164 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 568 workingset: 2456 #fd: 31 #threads: 1 priority: 8 nice: 0

                            name: 'BACSTray.exe' pid: 3148 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 1084 workingset: 4908 #fd: 57 #threads: 1 priority: 8 nice: 0

                            name: 'dsm_sa_datamgr64.exe' pid: 1612 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 13620 workingset: 19088 #fd: 337 #threads: 17 priority: 8 nice: 0

                            name: 'ons.exe' pid: 5880 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 2492 workingset: 4796 #fd: 78 #threads: 1 priority: 8 nice: 0

                            name: 'conhost.exe' pid: 5848 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 1108 workingset: 8824 #fd: 48 #threads: 2 priority: 8 nice: 0

                            name: 'dwm.exe' pid: 3748 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 12260 workingset: 41384 #fd: 157 #threads: 7 priority: 13 nice: 0

                            name: 'TNSLSNR.EXE' pid: 5700 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 17656 workingset: 21576 #fd: 213 #threads: 6 priority: 8 nice: 0

                            name: 'cmd.exe' pid: 5840 #procfdlimit: 4294967295 cpuusage: 0.0 privmem: 1472 workingset: 2120 #fd: 20 #threads: 1 priority: 8 nice: 0

                            DEVICES:

                            5_E: ior: 0.999 iow: 1.299 ios: 3 qlen: 0 wait: 0 type: VOTING[ONLINE]

                            0_C:_D: ior: 19.195 iow: 50.890 ios: 1 qlen: 0 wait: 2 type: SWAP

                            D: ior: 19.195 iow: 50.890 ios: 1 qlen: 0 wait: 2 type: SYS

                            C: ior: 0.000 iow: 0.000 ios: 0 qlen: 0 wait: 0 type: SWAP

                            4 ior: 0.000 iow: 0.800 ios: 0 qlen: 0 wait: 0 type: SYS

                            3 ior: 38.392 iow: 0.000 ios: 2 qlen: 0 wait: 0 type: SYS

                            2 ior: 9.598 iow: 3.199 ios: 0 qlen: 0 wait: 0 type: SYS

                            1 ior: 31.993 iow: 3.999 ios: 2 qlen: 0 wait: 0 type: SYS

                            NICS:

                            Broadcom BCM57840 NetXtreme II 10 Gigabit Ethernet (NDIS VBD Client) #85 netrr: 1.653  netwr: 0.397  neteff: 2.051  nicerrors: 0 pktsin: 16  pktsout: 2  errsin: 0  errsout: 0  indiscarded: 0  outdiscarded: 0  inunicast: 3  innonunicast: 13  type: PUBLIC

                            Broadcom BCM57840 NetXtreme II 10 Gigabit Ethernet (NDIS VBD Client) #84 netrr: 10.021  netwr: 21.347  neteff: 31.367  nicerrors: 0 pktsin: 22  pktsout: 30  errsin: 0  errsout: 0  indiscarded: 0  outdiscarded: 0  inunicast: 22  innonunicast: 0  type: PRIVATE

                            Broadcom BCM57840 NetXtreme II 10 Gigabit Ethernet (NDIS VBD Client) #83 netrr: 85.765  netwr: 10.756  neteff: 96.521  nicerrors: 0 pktsin: 15  pktsout: 17  errsin: 0  errsout: 0  indiscarded: 0  outdiscarded: 0  inunicast: 15  innonunicast: 0  type: PUBLIC

                            Broadcom BCM57840 NetXtreme II 10 Gigabit Ethernet (NDIS VBD Client) #82 netrr: 0.024  netwr: 0.018  neteff: 0.042  nicerrors: 0 pktsin: 0  pktsout: 0  errsin: 0  errsout: 0  indiscarded: 0  outdiscarded: 0  inunicast: 0  innonunicast: 0  type: PUBLIC

                             

                             

                            FILESYSTEMS:

                            name: Logical Disk mount: D: type: NTFS total: 247289852 used: 7359556 available: 239930296 used%: 2 mft%: -1 [ORACLE_HOME ]

                             

                             

                            name: Logical Disk mount: C: type: NTFS total: 41943036 used: 31709212 available: 10233824 used%: 75 mft%: -1 []

                             

                             

                            PROTOCOL ERRORS:

                            IPHdrErr: 0 IPAddrErr: 4148 IPUnkProto: 0 IPReasFail: 0 IPFragFail: 0 TCPFailedConn: 478 TCPEstRst: 2378 TCPRetraSeg: 7065 UDPUnkPort: 8702 UDPRcvErr: 0

                             

                             

                            And maybe this is the problem:

                             

                            PROTOCOL ERRORS:

                            IPHdrErr: 0 IPAddrErr: 4864

                             

                            Maybe a duplicated ip? I think that could be the problem...

                             

                            Thanks in advance

                             

                             


                            EDIT:

                             

                            this is the command used to produce the output:

                             

                            oclumon.bat dumpnodeview -allnodes -v -last "14:12:00"  > D:\ANALISYS\test.log

                            • 11. Re: ASM instances restarting, all database restart too
                              user1138812

                              i've found another thing that could be the cause of all of this:

                               

                              in the installation note was written to setup the "metric" on the network interface to:

                               

                              100 for cluster interconnect

                              300 for public network

                               

                              but i've also other 2 nic on the machine:

                               

                              management + iscsi

                               

                              then i set 310 & 310 for those 2 nics on both nodes.

                               

                              The stat collecting is started waiting for another crash...

                               

                               

                              Bye!

                              • 12. Re: ASM instances restarting, all database restart too
                                user1138812

                                Update:

                                 

                                after the modification on interface metric only one node is now crashin' every 8/9 hours.

                                 

                                The asm instance crushin' node is crushin with the same exact errors.

                                • 13. Re: ASM instances restarting, all database restart too
                                  ora_tech

                                  Hi,

                                   

                                  Could you post the OS watches logs and ASM Instace logs at the time of crash.

                                   

                                  thanks,

                                  X A H E E R