mercredi 26 janvier 2022

Python Flask for background long running jobs

API sync requests for immediate response, but we have long running (5 to 100 minutes) jobs where we cannot depend on sync request, we should implement worker nodes handles long running jobs and api just submit jobs and get status.

while I am browsing, recommended way to use cache system , queue system, worker nodes similarly like below diagram approach 1

Since Redis is cache system and writing status states relying on redis cache is good approach ?

Another approach is using persistent no sql db, queue system and worker nodes like below diagram

approach2

Above approach slows down the submissions and getting status because no cache system is using.

approach 3 I am consider to go with cache no sql and persistent no sql DB similarly like below

aproach3

I am designing first time production ready application and implementing my. I want to make sure correct design principals are used.

Load estimated per second 1000 requests average

Also how difficult implementing approach 3 ?

Below is what I am expecting.

  1. Performance
  2. tasks status api states for at least last 30 days jobs.

Programming: Python Framework: Flask

Aucun commentaire:

Enregistrer un commentaire