Trying to ETL users from AD to a table in Oracle. Having trouble with Loading into table phase and managing CDC (Change Data Capture) into a logging table. Our old process used to take ~39 hours to do this (Row-by-Row) processing. Using the UDT & Collection method I was able to decrease this time to ~+-1 minute.
I am using Oracle MERGE currently to accompish this.
Have also explored to PLSQL solutions including loops but didn't have much success and felt like the code was a bit kludge and not elegant. I feel like this is a fairly rudimentary thing of Oracle devs to have to do but there doesn't seem to be a clear cut solution on how to accomplish this.
Using this link and making a few modifications I was able to get it working for an object and collection type such as
Object Type
CREATE OR REPLACE type dbo.P_REC AS OBJECT
(
ATTR1 VARCHAR2(10 CHAR),
ATTR2 VARCHAR2(20 CHAR),
ATTR3 VARCHAR2(20 CHAR),
ATTR4 VARCHAR2(20 CHAR)
);
Collection Type
CREATE OR REPLACE type dbo.P_REC_LIST IS TABLE OF P_REC;
Log Table
CREATE TABLE LogTable
(
Date_Time DATETIME,
Name VARCHAR2(10 CHAR),
old VARCHAR2(20 CHAR),
new VARCHAR2(20 CHAR),
)
Destination Table - Has about 15-20k records in it on average.
CREATE TABLE pTable
(
Col1 VARCHAR2(20 CHAR),
Col2 VARCHAR2(10 CHAR),
Col3 VARCHAR2(20 CHAR),
Col4 VARCHAR2(20 CHAR)
)
Trigger - The trigger was done this way in order to hopefully only have changed data in the log table, no sense in having to repeat unchanged data.
CREATE OR REPLACE TRIGGER Log_Entry before
INSERT OR UPDATE OF
Col1,
Col2,
Col3,
Col4
ON pTable FOR EACH row
BEGIN
IF INSERTING THEN
INSERT INTO LogTable VALUES
(sysdate, :new.Col1, 'new record', :new.Col1);
END IF;
IF UPDATING AND (:new.Col2 != :old.Col2) THEN
INSERT INTO LogTable VALUES
(sysdate, :old.CN, :old.Col2, :new.Col2);
END IF;
IF UPDATING AND (:new.Col3 != :old.Col3) THEN
INSERT INTO LogTable VALUES
(sysdate, :old.CN, :old.Col3, :new.Col3);
END IF;
IF UPDATING AND (:new.Col4 != :old.Col4) THEN
INSERT INTO LogTable VALUES
(sysdate, :old.CN, :old.Col4, :new.Col4);
END IF;
END;
Destination table has about 95% of the data already in it we just want to insert/update/delete, new/existing(where data is different) and old records.
We are expecting ~10-20 inserts to be added to pTable daily, ~100 updates on various columns and maybe ~5-10 deletes to happen. Logging the changed data is an absolute must for auditing purposes.
Open to suggestions on the best/most efficient way to do this.
Aucun commentaire:
Enregistrer un commentaire