mercredi 6 janvier 2021

Best design pattern for Machine Learning pipeline

I am really new to writing library code. I am trying to create a Machine Learning pipeline and don't really know where to start. So I thought getting a design pattern suggested, would be a good start.

I have different components that I would like to run in separate processes, so that whenever one component fails the others can still keep running, while that one restarts.

My architecture looks as follows:

  1. I receive some stream of data which the Sender stores and waits to forward
  2. A Processor requests that data and passes it through the Machine Learning pipeline
  3. The Combiner takes the outputs of the ML pipeline, stores and subsequently merges all outputs (Imagine that I need multiple images before making a prediction) and sends the accumulated data to the receiver

I only know about the Observer Pattern and it looks like it could fit in this scenario.

Aucun commentaire:

Enregistrer un commentaire