I have the task of running multiple different Python ML models/pipelines(a/b test with small changes in data processing stage ). Pipelines are with almost the same code base(data processing stage)
The architecture of my application is as follows:
- pub/sub model with RabbitMQ
- 1 publisher push messages to multiply queues
- multiple subscribers consume data from queues
I have the following solutions in mind:
- repeat/rewrite code for each ML pipeline
- write and deploy subscribers from different git branches
- some any solutions?
Which solution is optimal in terms of support and code duplication?
Aucun commentaire:
Enregistrer un commentaire