vendredi 27 mai 2022

Functional data processing in clojure

I'm looking for some architectural solution to the problem I'm facing. I am struggling with a huge pipeline that processes data (which is a very complex structure). Generally, it can be illustrated as:

(defn process [data]
  (-> data
    do-something-1
    do-something-2
    do-something-3
    do-something-4
    ...
    do-something-20
)

where each of the do-something-* functions might be similarly complex. My problem is that there is a lot of coupling between functions in such a processing chain. For example do-something-3add something to data that later is required by do-something-9 which adds something else required by do-something-18 and so on. So essentially data is enriched by all those functions when cascading down the threading macro. It's very hard to keep the track of what is happening and when. Holding the whole processing chain in my head is just too much of a cognitive load (or at least I have too little RAM in my head). How do handle such cases? I get that there is no silver bullet but maybe there is something I'm missing (I started to learn clojure few months ago).

Aucun commentaire:

Enregistrer un commentaire