Recently I started to investigate a Pipeline
pattern or also known as Pipes and Filters
. I thought it is a good way to structure the code and applications which just process the data. I used this article as a base for my pipeline and steps implementation (but this is not so important). As usual blog is covering simple scenario but in my case I need (or maybe not) to work on IDisposable
objects which might travel through the process.
For instance Streams
Let's consider simple pipeline which should load csv file and insert its rows into some db. In simple abstraction we could implement such functions
Stream Step1(string filePath)
IEnumerable<RowType> Step2(Stream stream)
bool Step3(IEnumerable<RowType> data)
Now my question is if that is a good approach. Because if we implement that as step after step processing the Stream
object leaves first step and it is easy to fall into a memory leakage problem. I know that some might say that I should have Step1
which is loading and deserialising data but we are considering simple process. We might have more complex ones where passing a Stream makes more sense.
I am wondering how can I implement such pipelines to avoid memory leaks and also avoiding loading whole file into MemoryStream
(which would be safer). Should I somehow wrap each step in try..catch
blocks to call Dispose()
if something goes wrong? Or should I pass all IDisposable
resources into Pipeline
object which will be wrapped with using
to dispose all resources produced during processing correctly?
Aucun commentaire:
Enregistrer un commentaire