mardi 15 novembre 2022

Architectural design clarrification

I built an API in nodejs+express that allows reactjs clients to upload CSV files(maximum size is atmost 1GB) to the server.

I also wrote another API which when given the filename and row numbers in an array (ie array of row numbers ) as input, it selects the rows corresponding to the row numbers, from the previously stored files and writes it to another result file (writeStream). Then th resultant file is piped back to the client(all via streaming).

Currently as you see I am using files(basically nodejs' read and write streams) to asynchronously manage this.

But I have faced srious latency (only 2 cores are used) and some memory leak (900mb consumption) when I have 15 requests, each supplying about 600 rows to retrieve from files of size approximately 150mb.

I also have planned an alternate design.

Basically, I will store the entire file as a SQL Table with row numbers as primary indexed key.

I will convert the user inputted array of row numbrs to a another table using sql unnest and then join both these tables to get the rows needed.

Then I will supply back the resultant table as a csv file to the client.

Would this architecture be better than the previous architecture?

Any suggestions from devs is highly appreciated.

Thanks.

Aucun commentaire:

Enregistrer un commentaire