mardi 30 juillet 2019

Pipeline pattern and disposable objects

Recently I started to investigate a Pipeline pattern or also known as Pipes and Filters. I thought it is a good way to structure the code and applications which just process the data. I used this article as a base for my pipeline and steps implementation (but this is not so important). As usual blog is covering simple scenario but in my case I need (or maybe not) to work on IDisposable objects which might travel through the process.

For instance Streams

Let's consider simple pipeline which should load csv file and insert its rows into some db. In simple abstraction we could implement such functions

Stream Step1(string filePath)
IEnumerable<RowType> Step2(Stream stream)
bool Step3(IEnumerable<RowType> data)

Now my question is if that is a good approach. Because if we implement that as step after step processing the Stream object leaves first step and it is easy to fall into a memory leakage problem. I know that some might say that I should have Step1 which is loading and deserialising data but we are considering simple process. We might have more complex ones where passing a Stream makes more sense.

I am wondering how can I implement such pipelines to avoid memory leaks and also avoiding loading whole file into MemoryStream (which would be safer). Should I somehow wrap each step in try..catch blocks to call Dispose() if something goes wrong? Or should I pass all IDisposable resources into Pipeline object which will be wrapped with using to dispose all resources produced during processing correctly?

Aucun commentaire:

Enregistrer un commentaire