1 2 3 4 Previous Next 46 Replies Latest reply: May 23, 2011 9:26 AM by 834371 RSS

    Long running update

    Colin Thiner
      Hello,
      Can anyone explain why this simple update statement against a single partition in large table (~300,000 rows, ~1GB in size for the single partition) is taking very long time. The most unusual thing I see in the stats are HUGE number of buffer gets.

      Table def is below, and there are 25 local b-tree indexes also on this table (too much to paste here), each on a single column residing in seperate tablespace than the table.

      I don't have a trace and will not be able to get one. Any theories as to the high buffer gets? A simple table scan (which occurs many times in our batch) against a single partition takes usually between 30-60 seconds. Sometimes the table scan goes haywire and I see these huge buffer gets, somewhat higher disk reads, and much longer execution time. There are less than 3 million rows in the partition being acted on, and only updating a couple columns, I simply cannot understand why Oracle would be getting a block (whether it was in cache already or not) over 1 BILLION times to perform this update.

      This is Oracle 11g 11.1.0.7 on RHL 5.3, 2 node RAC but all processing on instance 1 and instance 2 shut down at this point to avoid any possibility of cache fusuion issues.


      Elapsed
      SQL Id Time (ms)

      0np3ccxhf9jmc 1.79E+07
      UPDATE ESRODS.EXTR_CMG_TRANSACTION_HISTORY SET RULE_ID_2 = '9285', REPORT_CODE =
      'MMKT' WHERE EDIT_CUSIP_NUM = '19766G868' AND PROCESS_DATE BETWEEN '01-JAN-201
      0' AND '31-JAN-2010' AND RULE_ID_2 IS NULL

      Plan Statistics
      -> % Total DB Time is the Elapsed Time of the SQL statement divided
      into the Total Database Time multiplied by 100

      Stat Name Statement Per Execution % Snap
      ---------------------------------------- ---------- -------------- -------
      Elapsed Time (ms) 1.79E+07 17,915,656.1 2.3
      CPU Time (ms) 1.18E+07 11,837,756.4 2.5
      Executions 1 N/A N/A
      Buffer Gets 1.09E+09 1.089168E+09 3.3
      Disk Reads 246,267 246,267.0 0.0
      Parse Calls 1 1.0 0.0
      Rows 326,843 326,843.0 N/A
      User I/O Wait Time (ms) 172,891 N/A N/A
      Cluster Wait Time (ms) 0 N/A N/A
      Application Wait Time (ms) 0 N/A N/A
      Concurrency Wait Time (ms) 504,047 N/A N/A
      Invalidations 0 N/A N/A
      Version Count 21 N/A N/A
      Sharable Mem(KB) 745 N/A N/A

      Execution Plan
      ------------------------------------------------------------------------------------------------------------------------
      | Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time | Pstart| Pstop |
      ------------------------------------------------------------------------------------------------------------------------
      | 0 | UPDATE STATEMENT | | | | 36029 (100)| | | |
      | 1 | UPDATE | EXTR_CMG_TRANSACTION_HISTORY | | | | | | |
      | 2 | PARTITION RANGE SINGLE| | 305K| 21M| 36029 (1)| 00:05:16 | 62 | 62 |
      | 3 | TABLE ACCESS FULL | EXTR_CMG_TRANSACTION_HISTORY | 305K| 21M| 36029 (1)| 00:05:16 | 62 | 62 |
      ------------------------------------------------------------------------------------------------------------------------



      Full SQL Text

      SQL ID SQL Text
      ------------ -----------------------------------------------------------------
      0np3ccxhf9jm UPDATE ESRODS.EXTR_CMG_TRANSACTION_HISTORY SET RULE_ID_2 = '9285'
      ', REPORT_CODE = 'MMKT' WHERE EDIT_CUSIP_NUM = '19766G868' AND PR
      OCESS_DATE BETWEEN '01-JAN-2010' AND '31-JAN-2010' AND RULE_ID_2
      IS NULL




      Table def:
      CREATE TABLE EXTR_CMG_TRANSACTION_HISTORY
      (
      TRANSACTION_ID NUMBER(15) NOT NULL,
      CREATE_DATE DATE,
      CREATE_USER VARCHAR2(80 BYTE),
      MODIFY_DATE DATE,
      MODIFY_USER VARCHAR2(80 BYTE),
      EXCEPTION_FLG CHAR(1 BYTE),
      SOURCE_SYSTEM VARCHAR2(20 BYTE),
      SOURCE_TYPE VARCHAR2(32 BYTE),
      TRANSACTION_STATUS VARCHAR2(8 BYTE),
      FUND_ID NUMBER(15),
      FUND_UNIT_ID NUMBER(15),
      FROM_FUND_ID NUMBER(15),
      FROM_FUND_UNIT_ID NUMBER(15),
      EXECUTING_DEALER_ID NUMBER(15),
      EXECUTING_BRANCH_ID NUMBER(15),
      CLEARING_DEALER_ID NUMBER(15),
      CLEARING_BRANCH_ID NUMBER(15),
      BRANCH_PERSON_MAP_ID NUMBER(15),
      BP_REP_MAP_ID NUMBER(15),
      REP_ID NUMBER(15),
      PERSON_ID NUMBER(15),
      TPA_DEALER_ID NUMBER(15),
      TRUST_DEALER_ID NUMBER(15),
      TRANS_CODE_ID NUMBER(15),
      EDIT_DEALER_NUM VARCHAR2(30 BYTE),
      EDIT_BRANCH_NUM VARCHAR2(50 BYTE),
      EDIT_REP_NUM VARCHAR2(100 BYTE),
      EDIT_CUSIP_NUM VARCHAR2(9 BYTE),
      TRANS_TYPE VARCHAR2(80 BYTE),
      TRANSACTION_CD VARCHAR2(8 BYTE),
      TRANSACTION_SUFFIX VARCHAR2(8 BYTE),
      SHARE_BALANCE_IND VARCHAR2(2 BYTE),
      PROCESS_DATE DATE,
      BATCH_DATE DATE,
      SUPER_SHEET_DATE DATE,
      CONFIRM_DATE DATE,
      TRADE_DATE DATE,
      SETTLE_DATE DATE,
      PAYMENT_DATE DATE,
      AM_PM_CD VARCHAR2(2 BYTE),
      TRUST_DEALER_NUM VARCHAR2(7 BYTE),
      TPA_DEALER_NUM VARCHAR2(7 BYTE),
      TRUST_COMPANY_NUM VARCHAR2(10 BYTE),
      DEALER_NUM VARCHAR2(25 BYTE),
      BRANCH_NUM VARCHAR2(50 BYTE),
      REP_NUM VARCHAR2(100 BYTE),
      DEALER_NAME VARCHAR2(80 BYTE),
      REP_NAME VARCHAR2(80 BYTE),
      SOCIAL_SECURITY_NUMBER VARCHAR2(9 BYTE),
      ACCT_NUMBER_CD VARCHAR2(6 BYTE),
      ACCT_NUMBER VARCHAR2(20 BYTE),
      ACCT_SHORT_NAME VARCHAR2(80 BYTE),
      FROM_TO_ACCT_NUM VARCHAR2(20 BYTE),
      EXTERNAL_ACCT_NUM VARCHAR2(14 BYTE),
      NAV_ACCT VARCHAR2(1 BYTE),
      MANAGEMENT_CD VARCHAR2(16 BYTE),
      PRODUCT VARCHAR2(80 BYTE),
      SUBSET_PRODUCT VARCHAR2(3 BYTE),
      FUND_NAME VARCHAR2(80 BYTE),
      FUND_NUM VARCHAR2(7 BYTE),
      FUND_CUSIP_NUM VARCHAR2(9 BYTE),
      TICKER_SYMBOL VARCHAR2(10 BYTE),
      APL_FUND_TYPE VARCHAR2(10 BYTE),
      LOAD_INDICATOR VARCHAR2(50 BYTE),
      FROM_TO_FUND_NUM VARCHAR2(7 BYTE),
      FROM_TO_FUND_CUSIP_NUM VARCHAR2(9 BYTE),
      CUM_DISCNT_NUM VARCHAR2(9 BYTE),
      NSCC_CONTROL_CD VARCHAR2(15 BYTE),
      NSCC_NAV_REASON_CD VARCHAR2(1 BYTE),
      BATCH_NUMBER VARCHAR2(20 BYTE),
      ORDER_NUMBER VARCHAR2(16 BYTE),
      CONFIRM_NUMBER VARCHAR2(9 BYTE),
      AS_OF_REASON_CODE VARCHAR2(3 BYTE),
      SOCIAL_CODE VARCHAR2(3 BYTE),
      NETWORK_MATRIX_LEVEL VARCHAR2(1 BYTE),
      SHARE_PRICE NUMBER(15,4),
      GROSS_AMOUNT NUMBER(15,2),
      GROSS_SHARES NUMBER(15,4),
      NET_AMOUNT NUMBER(15,2),
      NET_SHARES NUMBER(15,4),
      DEALER_COMMISSION_CODE CHAR(1 BYTE),
      DEALER_COMMISSION_AMOUNT NUMBER(15,2),
      UNDRWRT_COMMISSION_CODE CHAR(1 BYTE),
      UNDRWRT_COMMISSION_AMOUNT NUMBER(15,2),
      DISCO-stupid spam filter- UNT_CATEGORY VARCHAR2(2 BYTE),
      LOI_NUMBER VARCHAR2(9 BYTE),
      RULE_ID_1 NUMBER(15),
      RULE_ID_2 NUMBER(15),
      OMNIBUS_FLG CHAR(1 BYTE),
      MFA_FLG CHAR(1 BYTE),
      REPORT_CODE VARCHAR2(80 BYTE),
      TERRITORY_ADDR_CODE VARCHAR2(3 BYTE),
      ADDRESS_ID NUMBER(15),
      POSTAL_CODE_ID NUMBER(15),
      CITY VARCHAR2(50 BYTE),
      STATE_PROVINCE_CODE VARCHAR2(5 BYTE),
      POSTAL_CODE VARCHAR2(12 BYTE),
      COUNTRY_CODE VARCHAR2(5 BYTE),
      LOB_ID NUMBER(15),
      CHANNEL_ID NUMBER(15),
      REGION_ID NUMBER(15),
      TERRITORY_ID NUMBER(15),
      EXCEPTION_NOTES VARCHAR2(4000 BYTE),
      SOURCE_RECORD_ID NUMBER(15),
      LOAD_ID NUMBER(15),
      BIN VARCHAR2(20),
      SHARE_CLASS VARCHAR2(50),
      ACCT_PROD_ID NUMBER,
      ORIGINAL_FUND_NUM VARCHAR2(7),
      ORIGINAL_FROM_TO_FUND_NUM VARCHAR2(7),
      ACCT_PROD_REGISTRATION_ID NUMBER,
      REGISTRATION_LINE_1 VARCHAR2(60),
      REGISTRATION_LINE_2 VARCHAR2(60),
      REGISTRATION_LINE_3 VARCHAR2(60),
      REGISTRATION_LINE_4 VARCHAR2(60),
      REGISTRATION_LINE_5 VARCHAR2(60),
      REGISTRATION_LINE_6 VARCHAR2(35),
      REGISTRATION_LINE_7 VARCHAR2(35),
      SECONDARY_LOB_ID NUMBER(15,0),
      SECONDARY_CHANNEL_ID NUMBER(15,0),
      SECONDARY_REGION_ID NUMBER(15,0),
      SECONDARY_TERRITORY_ID NUMBER(15,0),
      ACCOUNT_OVERRIDE_PRIORITY_CODE NUMBER(3,0)
      )
      TABLESPACE P_ESRODS_EXTR_TRANS_LARGE_DAT
      PCTUSED 0
      PCTFREE 25
      INITRANS 1
      MAXTRANS 255
      NOLOGGING
      PARTITION BY RANGE (PROCESS_DATE)
      (
      PARTITION P_ESRODS_EXTR_CMG_TRAN_PRE2005 VALUES LESS THAN (TO_DATE(' 2005-01-01 00:00:00', 'SYYYY-MM-DD HH24:MI:SS', 'NLS_CALENDAR=GREGORIAN'))
      LOGGING
      COMPRESS
      TABLESPACE P_ESRODS_EXTR_TRANS_LARGE_DAT
      PCTFREE 25
      INITRANS 100
      MAXTRANS 255
      STORAGE (
      INITIAL 5M
      NEXT 5M
      MINEXTENTS 1
      MAXEXTENTS 2147483645
      PCTINCREASE 0
      BUFFER_POOL DEFAULT
      ),
      .
      .
      .
      .
      PARTITION P_ESRODS_EXTR_CMG_TRAN_201105 VALUES LESS THAN (TO_DATE(' 2011-06-01 00:00:00', 'SYYYY-MM-DD HH24:MI:SS', 'NLS_CALENDAR=GREGORIAN'))
      LOGGING
      COMPRESS
      TABLESPACE P_ESRODS_EXTR_TRANS_LARGE_DAT
      PCTFREE 25
      INITRANS 100
      MAXTRANS 255
      STORAGE (
      INITIAL 5M
      NEXT 5M
      MINEXTENTS 1
      MAXEXTENTS 2147483645
      PCTINCREASE 0
      BUFFER_POOL DEFAULT
      ),
      PARTITION P_ESRODS_EXTR_CMG_TRAN_201106 VALUES LESS THAN (TO_DATE(' 2011-07-01 00:00:00', 'SYYYY-MM-DD HH24:MI:SS', 'NLS_CALENDAR=GREGORIAN'))
      LOGGING
      COMPRESS
      TABLESPACE P_ESRODS_EXTR_TRANS_LARGE_DAT
      PCTFREE 25
      INITRANS 100
      MAXTRANS 255
      STORAGE (
      INITIAL 5M
      NEXT 5M
      MINEXTENTS 1
      MAXEXTENTS 2147483645
      PCTINCREASE 0
      BUFFER_POOL DEFAULT
      )
      )
      NOCACHE
      NOPARALLEL;


      ALTER TABLE EXTR_CMG_TRANSACTION_HISTORY ADD (
      CONSTRAINT PK_EXTR_CMG_TRANSACTION_HIST PRIMARY KEY (TRANSACTION_ID)
      USING INDEX
      TABLESPACE P_ESRODS_EXTR_TRANS_LARGE_IDX
      PCTFREE 25
      INITRANS 2
      MAXTRANS 255
      STORAGE (
      INITIAL 5M
      NEXT 5M
      MINEXTENTS 1
      MAXEXTENTS 2147483645
      PCTINCREASE 0
      FREELISTS 1
      FREELIST GROUPS 1
      ));

      Edited by: 855802 on May 1, 2011 6:46 AM
        • 1. Re: Long running update
          sb92075
          Full Table Scan is to be expected with SQL below
          UPDATE esrods.extr_cmg_transaction_history
          SET    rule_id_2 = '9285',
                 report_code = 'MMKT'
          WHERE  edit_cusip_num = '19766G868'
                 AND process_date BETWEEN '01-JAN-2010' AND '31-JAN-2010'
                 AND rule_id_2 IS NULL  
          about how many rows in the table?
          about how many rows get UPDATEd?
          • 2. Re: Long running update
            781417
            You say that you have 25 local b-tree indexes on this table, but are any of them on the only column that this statement could use, namely 'edit_cusip_num'?

            Have you had a look at the wait events for this running session, since you indicate that you are using RAC, are you seeing any GC issues?
            • 3. Re: Long running update
              Colin Thiner
              Agreed that full table scan is expected (with partition pruning).

              Table:
              ~86 million rows
              ~10.5 million blocks

              Jan 2010 partition:
              ~ 2 million rows
              ~ 248,000 blocks

              On this update about 320,000 rows are updated. Execution time is almost 5 hours. Disk reads and I/O waits seem normal. Overall AWR report shows pretty good response times from ASM/File reads. Large CPU wait and large buffer gets don't make sense to me still.
              • 4. Re: Long running update
                Colin Thiner
                No GC issues at all, I'll have to look up the wait events, I believe CPU and DB File Seq Read. Although, there wasn't an abnormal amount, I just see the buffer gets climbing as I watch the session.


                The update is updating 320,000 records, I don't think we want to use an index as the table fetches would be too many, direct read full table scan is much faster. However, yes one of the indexes is on the edit_cusip_num field. Also on Rule_ID_2 is indexed. The rest of the columns on this update are not indexed.
                • 5. Re: Long running update
                  orawiss
                  855802 wrote:
                  Agreed that full table scan is expected (with partition pruning).

                  Table:
                  ~86 million rows
                  ~10.5 million blocks

                  Jan 2010 partition:
                  ~ 2 million rows
                  ~ 248,000 blocks

                  On this update about 320,000 rows are updated. Execution time is almost 5 hours. Disk reads and I/O waits seem normal. Overall AWR report shows pretty good response times from ASM/File reads. Large CPU wait and large buffer gets don't make sense to me still.
                  UPDATE esrods.extr_cmg_transaction_history
                  SET    rule_id_2 = '9285',
                         report_code = 'MMKT'
                  WHERE  edit_cusip_num = '19766G868'
                         AND process_date BETWEEN '01-JAN-2010' AND '31-JAN-2010'
                         AND rule_id_2 IS NULL 
                  I will change the query as following, for example:

                  1- Create a new table, for example:
                  create new_table NOLOGGING
                     PARALLEL (DEGREE 3)
                   as select COL1, COL2, CASE WHEN edit_cusip_num = '19766G868' AND rule_id_2 IS NULL  THEN  '9285' as RULE_ID_2 ELSE RULE_ID_2 END, CASE WHEN... from   esrods.extr_cmg_transaction_history  PARTITION NAME_OF_PARTITION;
                  2- Perform an exchange partition
                  alter table esrods.extr_cmg_transaction_history 
                    exchange partition NAME_OF_PARTITION
                     with table new_table;
                  • 6. Re: Long running update
                    Colin Thiner
                    Thank you for the suggestion - that is an interesting work around. It may work faster, but unfortunately this is one of ~8000 update statements performed on this partition. The update statements are in application code, and at this point I don't think it is going to be an option to change the SQL that much. We can probably add hints, but cannot make this into a swap partition. I would like to concentrate on determining why the update is doing so much work and taking so much time.
                    • 7. Re: Long running update
                      orawiss
                      what about altering this table partition to NOLOGGING ? for less redo...
                      • 8. Re: Long running update
                        Colin Thiner
                        You cannot bypass redo logging on update statement, there are only a handfull of operations that you can skip redo logging. The table is created no-logging. Still, update of the this many rows should not be affected by the redo writes. I agree that would be a way to speed it up.

                        http://download.oracle.com/docs/cd/B19306_01/server.102/b14200/clauses005.htm#i999782

                        NOLOGGING is supported in only a subset of the locations that support LOGGING. Only the following operations support the NOLOGGING mode:
                        DML:  
                        Direct-path INSERT (serial or parallel) resulting either from an INSERT or a MERGE statement. NOLOGGING is not applicable to any UPDATE operations resulting from the MERGE statement.
                        Direct Loader (SQL*Loader)
                        DDL:  
                        CREATE TABLE ... AS SELECT
                        CREATE TABLE ... LOB_storage_clause ... LOB_parameters ... NOCACHE | CACHE READS
                        ALTER TABLE ... LOB_storage_clause ... LOB_parameters ... NOCACHE | CACHE READS (to specify logging of newly created LOB columns)
                        ALTER TABLE ... modify_LOB_storage_clause ... modify_LOB_parameters ... NOCACHE | CACHE READS (to change logging of existing LOB columns)
                        ALTER TABLE ... MOVE
                        ALTER TABLE ... (all partition operations that involve data movement)
                        ALTER TABLE ... ADD PARTITION (hash partition only)
                        ALTER TABLE ... MERGE PARTITIONS
                        ALTER TABLE ... SPLIT PARTITION
                        ALTER TABLE ... MOVE PARTITION
                        ALTER TABLE ... MODIFY PARTITION ... ADD SUBPARTITION
                        ALTER TABLE ... MODIFY PARTITION ... COALESCE SUBPARTITION
                        CREATE INDEX
                        ALTER INDEX ... REBUILD
                        ALTER INDEX ... REBUILD [SUB]PARTITION
                        ALTER INDEX ... SPLIT PARTITION
                        • 9. Re: Long running update
                          orawiss
                          855802 wrote:
                          You cannot bypass redo logging on update statement, there are only a handfull of operations that you can skip redo logging. The table is created no-logging. Still, update of the this many rows should not be affected by the redo writes. I agree that would be a way to speed it up.

                          http://download.oracle.com/docs/cd/B19306_01/server.102/b14200/clauses005.htm#i999782

                          NOLOGGING is supported in only a subset of the locations that support LOGGING. Only the following operations support the NOLOGGING mode:
                          DML:  
                          Direct-path INSERT (serial or parallel) resulting either from an INSERT or a MERGE statement. NOLOGGING is not applicable to any UPDATE operations resulting from the MERGE statement.
                          Direct Loader (SQL*Loader)
                          DDL:  
                          CREATE TABLE ... AS SELECT
                          CREATE TABLE ... LOB_storage_clause ... LOB_parameters ... NOCACHE | CACHE READS
                          ALTER TABLE ... LOB_storage_clause ... LOB_parameters ... NOCACHE | CACHE READS (to specify logging of newly created LOB columns)
                          ALTER TABLE ... modify_LOB_storage_clause ... modify_LOB_parameters ... NOCACHE | CACHE READS (to change logging of existing LOB columns)
                          ALTER TABLE ... MOVE
                          ALTER TABLE ... (all partition operations that involve data movement)
                          ALTER TABLE ... ADD PARTITION (hash partition only)
                          ALTER TABLE ... MERGE PARTITIONS
                          ALTER TABLE ... SPLIT PARTITION
                          ALTER TABLE ... MOVE PARTITION
                          ALTER TABLE ... MODIFY PARTITION ... ADD SUBPARTITION
                          ALTER TABLE ... MODIFY PARTITION ... COALESCE SUBPARTITION
                          CREATE INDEX
                          ALTER INDEX ... REBUILD
                          ALTER INDEX ... REBUILD [SUB]PARTITION
                          ALTER INDEX ... SPLIT PARTITION
                          Yes, I was thinking in consideration with my previous post using create table as select ...... But, if it's in application side then you need to think about another solution...
                          • 10. Re: Long running update
                            orawiss
                            ADDM reports does not showing any advice?
                            • 11. Re: Long running update
                              Colin Thiner
                              ADDM report basically just says to add CPU. Really, add CPU rather than determine what is really causing the problem?

                              Still I have not been able to find anyone (including our Advanced Customer Support Services people) who can give my any possible explanation of over 1 billion buffer gets for above query. I see these in the session as I monitor, as well as in the AWR reports. There is an underlying issue here, and I would like to find it. There might be a perfectly good explanation for the large number of gets, but it doesn't make sense to me.
                              • 12. Re: Long running update
                                781417
                                Have you had a look at the session using tanel poders snapper script, to see what what is actually happening at the session stats level. I'm curious if with the high volume of updates you are having to go to undo to create a read consistent image of the block many times

                                Have a look at Jonathan Lewis article: http://jonathanlewis.wordpress.com/2011/04/18/consistent-reads/
                                • 13. Re: Long running update
                                  Colin Thiner
                                  Thanks for the suggestion, I haven't used that script before and will run it during our next run and post the results.

                                  Not sure about the undo theory, the application runs N threads, each trhead updates the data for one and only one partition date range. So theoretically one one transaction at a time per partition.
                                  • 14. Re: Long running update
                                    Colin Thiner
                                    Got some trace data...

                                    Why does this show so many CR for the update line? Looks like 12,150 rows examined, but 145,649 CR.

                                    WAIT #3: nam='db file sequential read' ela= 287 file#=143 block#=1145721 blocks=1 obj#=253277 tim=1304524445783293
                                    EXEC #3:c=5306192,e=7090285,p=3620,cr=145649,cu=211430,mis=0,r=12150,dep=0,og=1,plh=3987740729,tim=1304524445799700
                                    STAT #3 id=1 cnt=0 pid=0 pos=1 obj=0 op='UPDATE EXTR_CMG_TRANSACTION_HISTORY (cr=145649 pr=3620 pw=0 time=0 us)'
                                    STAT #3 id=2 cnt=12150 pid=1 pos=1 obj=0 op='PARTITION RANGE SINGLE PARTITION: 67 67 (cr=4496 pr=3620 pw=0 time=39715 us cost=32 size=70 card=1)'
                                    STAT #3 id=3 cnt=12150 pid=2 pos=1 obj=253210 op='TABLE ACCESS BY LOCAL INDEX ROWID EXTR_CMG_TRANSACTION_HISTORY PARTITION: 67 67 (cr=4496 pr=3620 pw=0 time=39715 us cost=32 size=70 card=1)'
                                    STAT #3 id=4 cnt=12150 pid=3 pos=1 obj=255274 op='INDEX RANGE SCAN RP_CLEARING_BRANCH_ID_IDX PARTITION: 67 67 (cr=22 pr=20 pw=0 time=0 us cost=3 size=0 card=57)'
                                    XCTEND rlbk=0, rd_only=0, tim=1304524445799700
                                    WAIT #3: nam='SQL*Net message to client' ela= 5 driver id=1413697536 #bytes=1 p3=0 obj#=253277 tim=1304524445818706
                                    WAIT #3: nam='SQL*Net message from client' ela= 701 driver id=1413697536 #bytes=1 p3=0 obj#=253277 tim=1304524445819507
                                    CLOSE #3:c=0,e=0,dep=0,type=0,tim=1304524445818269
                                    1 2 3 4 Previous Next