mercredi 22 janvier 2020

Synchronizing data between Django and different data sources

I have a Django 2.2 app with using only its ORM features (no admin, views, URL routing) in which I have the two following models (in reality it's more but it's just for the sake of giving a straightforward example):

from django.db import models

class Person(models.Model):
    name = models.CharField(max_length=100)
    age = models.IntegerField()
    job = models.ForeignKey('myapp.Job')

class Job(models.Model):
    title = models.TextField()
    description = models.TextField()
    entry_date = models.DateField()

Now, I would like to be able to periodically synchronize the objects stored in my Django app with one or more external date sources (actually its concerns only a MySQL db and a REST API but it's likely I'm going to have to plug-in new data sources in some time).

The "direction" of the synchronization depends on the data source used, with one I'm only pulling data into my Django app, with the other I'm exporting data from my Django app, creating new objects when necessary, doing the updates, deletes, etc., and my with another data source I'm doing sync both ways.

All the external data sources have a different data layout and in order to import/export some fields to/from my app, I'll have to do some work ; e.g. on a data source, the value corresponding to my entry_date field on the Job model is stored in a funky timestamp and thus, it obligates me to do a conversion on it before I can use it at all.

However, I'm really struggling to figure how I could implement this in my Django app.

The first thing I've tried to design is, implementing CRUD operations for all my data sources using class abstraction but I obviously need a mechanism to have bindings between the fields of my Django app and the all different data layout of my sources. And yet, I'm seeking for maximal abstraction here, keeping all the data treatment and logic separated from the CRUD operations.

Aucun commentaire:

Enregistrer un commentaire