lundi 2 novembre 2020

Best way to pass config for multiple service integrations in a pipeline service?

So I am building a service that has a bunch of integrations like AWS S3, Databases. We use a config file to set the config for the above integrations. I have individual classes for each of integrations and my main entry point parses all the configs from the file stores it in the main pipeline class and then sends it over to the individual classes when they are used. So the code kinda looks like this

db.py

class DB:
    def __init__(self, host):
        self.host = host

s3.py

class S3:
    def __init__(self, bucket):
        self.bucket = bucket

Then there is the pipeline class which does a bunch of tasks sequentially and uses the integration

pipeline.py

class Pipeline:
    def __init__(self, db_host, s3_bucket):
        self.db_host = db_host
        self.s3_bucket = s3_bucket
    
    def run(self):
        s3 = S3(self.s3_bucket)
        s3.upload()

        db = DB(self.db_host)
        db.upload()

The main.py file parses the config file and then initializes the Pipeline class with all the configs which then makes individual integrations objects with their configs. Something like this. main.py

config = parse_config()
Pipeline = Pipeline(config.db_host, config.s3_bucket)
pipeline.run()

This way of doing things this way seems very weird because I am parsing all config at first then passing it to Pipeline which is responsible for doing tasks and then passing individual config to class when they are needed. Is there a better way of passing config to individual config to classes so that I do not have to pass a bunch of config to Pipeline? What are some good design choices here ?

An important thing to consider here is that there could be multiple configs for a particular integration. So I might need to use two DB configs for two DBs to do separate tasks

Aucun commentaire:

Enregistrer un commentaire