Skip to Main Content

SQL & PL/SQL

Announcement

For appeals, questions and feedback about Oracle Forums, please email oracle-forums-moderators_us@oracle.com. Technical questions should be asked in the appropriate category. Thank you!

Fastest way to batch delete data from a table with 1 billion rows

OraCJan 30 2021 — edited Jan 30 2021

Hi,
I need some help deleting batches from a really large online transactions table (up to 1 billion records). I hope to delete around 9 million records daily. 9 million more are being added daily. I have an off-peak time window when customer usage is limited so I can try to run this optimally, but I'm also conscious of not impacting any potential customers too much by specify too high a batch size below(10,000). Its Oracle 12.2 Standard Edition so unfortunately partitioning is not an option. I've come up with the following but its just not deleting fast enough. The initial select seems to be ok, its more about my loop. Is there a more efficient way of batching this?
DECLARE
cursor cur is
select /*+ index_ffs(a,P_ITD) parallel_index(a,P_ITD,4) */ C_ID from ITD a WHERE CREATED < '27-NOV-20 12.00.00.000000 AM';
TYPE CII_TYPE IS TABLE OF NUMBER;
CII_TYPE_TBL CII_TYPE;
BEGIN
OPEN CUR;
LOOP
FETCH CUR BULK COLLECT INTO CII_TYPE_TBL LIMIT 10000;
FORALL i IN 1..CII_TYPE_TBL.COUNT
DELETE FROM ITD WHERE C_ID=CII_TYPE_TBL(i);
COMMIT;
EXIT WHEN CUR%NOTFOUND;
END LOOP;
CLOSE CUR;
END;
/
P_ITD is the primary key constraint on the ITD table on C_ID
CREATED_ON is also indexed separately.
Thanks

Comments

ORASCN

Hi ,
Use the SQLEXEC feature to execute database statements or call procs from GG processes. Please check the below,

SQLEXEC (0 Bytes)17 Customizing Oracle GoldenGate Processing (0 Bytes)
Regards,
Veera

Kremnican

Unfortunately SQLEXEC in the MAP statement executes even before the MAP is "executed" (insert / update / delete) as the result can be used in the MAP statement itself.
Standalone SQLEXEC has a timer, which means it either runs when not needed or there's no guarantee it will be run right before the COMMIT, e.g. the timing may cause to run the SQL procedure some time after COMMIT, which may cause inconsistent behaviour.

LexmanLexman-Oracle

As your requirement is ti run the SP on a subset of data, you must use sqlexec with MAP statement. Sqlexec for MAP statements have the option to run the SP on the data set before or after the mapping
Check options AFTERFILTER/BEFOREFILTER in sqlexec documentation
Also , you can choose the iteration of SP run , again option EXEC {MAP | ONCE | TRANSACTION | SOURCEROW}
https://docs.oracle.com/en/middleware/goldengate/core/19.1/reference/sqlexec.html#GUID-34A0589B-1450-4BC9-A573-683895AAA1EC

1 - 3

Post Details

Added on Jan 30 2021
25 comments
29,558 views