mardi 27 avril 2021

Running multiple ML pipelines with almost the same code base

I have the task of running multiple different Python ML models/pipelines(a/b test with small changes in data processing stage ). Pipelines are with almost the same code base(data processing stage)

The architecture of my application is as follows:

  1. pub/sub model with RabbitMQ
  2. 1 publisher push messages to multiply queues
  3. multiple subscribers consume data from queues
    enter image description here I have the following solutions in mind:
  • repeat/rewrite code for each ML pipeline
  • write and deploy subscribers from different git branches
  • some any solutions?

Which solution is optimal in terms of support and code duplication?

Aucun commentaire:

Enregistrer un commentaire