vendredi 27 mars 2015

Node express with Kue - producer consumer design pattern advise

I'm writing a node application that pops messages onto a Kue job queue from REST calls processed by Express. I then want a number of worker processes running across multiple threads and servers to consume these messages and perform their long running tasks.


I have successfully written a standalone app that seems to work OK, but I'm confused about how I can distribute this across multiple servers. I've been looking into PM2 which seems to be the best way to load balance node processes.


My questions are..



  • Would it be best that express/Kue message producer app be a separate app to the workers apps? So for instance, one Express app running on one server, then a worker app running separately on all the servers?

  • Can I stop/start all/some of the workers across different servers with a single instance of PM2? Or, do I need an instance of PM2 running on each server?

  • Can anyone offer up a better solution?


The worker processes are basically copying/moving large files to and from remote servers, hence the need for distribution across multiple servers instead of just threads, for disk i/o and bandwidth performance.


You input is much appreciated.


Aucun commentaire:

Enregistrer un commentaire