vendredi 15 février 2019

Oracle ETL and Logging

Trying to ETL users from AD to a table in Oracle. Having trouble with Loading into table phase and managing CDC (Change Data Capture) into a logging table. Our old process used to take ~39 hours to do this (Row-by-Row) processing. Using the UDT & Collection method I was able to decrease this time to ~+-1 minute.

I am using Oracle MERGE currently to accompish this.

Have also explored to PLSQL solutions including loops but didn't have much success and felt like the code was a bit kludge and not elegant. I feel like this is a fairly rudimentary thing of Oracle devs to have to do but there doesn't seem to be a clear cut solution on how to accomplish this.

Using this link and making a few modifications I was able to get it working for an object and collection type such as

Object Type

CREATE OR REPLACE type dbo.P_REC AS OBJECT
(
    ATTR1    VARCHAR2(10 CHAR),
    ATTR2    VARCHAR2(20 CHAR),
    ATTR3    VARCHAR2(20 CHAR),
    ATTR4    VARCHAR2(20 CHAR)
);

Collection Type

CREATE OR REPLACE type dbo.P_REC_LIST IS TABLE OF P_REC;

Log Table

CREATE TABLE LogTable
(
  Date_Time    DATETIME,
  Name         VARCHAR2(10 CHAR),
  old          VARCHAR2(20 CHAR),
  new          VARCHAR2(20 CHAR),
)

Destination Table - Has about 15-20k records in it on average.

CREATE TABLE pTable
(
  Col1    VARCHAR2(20 CHAR),
  Col2    VARCHAR2(10 CHAR),
  Col3    VARCHAR2(20 CHAR),
  Col4    VARCHAR2(20 CHAR)
)

Trigger - The trigger was done this way in order to hopefully only have changed data in the log table, no sense in having to repeat unchanged data.

CREATE OR REPLACE TRIGGER Log_Entry before
   INSERT OR UPDATE OF
     Col1,
     Col2,
     Col3,
     Col4
  ON pTable FOR EACH row 
BEGIN 

IF INSERTING THEN
   INSERT INTO LogTable VALUES
    (sysdate, :new.Col1, 'new record', :new.Col1);
END IF;

IF  UPDATING AND (:new.Col2 != :old.Col2) THEN

   INSERT INTO LogTable VALUES
    (sysdate, :old.CN, :old.Col2, :new.Col2);
END IF;

IF  UPDATING AND (:new.Col3 != :old.Col3) THEN

   INSERT INTO LogTable VALUES
    (sysdate, :old.CN, :old.Col3, :new.Col3);
END IF;

IF  UPDATING AND (:new.Col4 != :old.Col4) THEN

   INSERT INTO LogTable VALUES
    (sysdate, :old.CN, :old.Col4, :new.Col4);
END IF;
END;

Destination table has about 95% of the data already in it we just want to insert/update/delete, new/existing(where data is different) and old records.

We are expecting ~10-20 inserts to be added to pTable daily, ~100 updates on various columns and maybe ~5-10 deletes to happen. Logging the changed data is an absolute must for auditing purposes.

Open to suggestions on the best/most efficient way to do this.

Aucun commentaire:

Enregistrer un commentaire