I have a scenario for which I am trying to figure out how to implement this queue architecture. I was looking into using RabbitMQ but am open to other queue solutions if they are better for the following scenario.
We upload a file from the frontend to the backend. The backend processes (uploads) the file to S3. I plan on creating a message and insert it into a queue that we need to process file ABC.csv. This file contains products that we will need to fetch information on from certain API’s.
The APIs have a rate limit/throttling. Therefore we cannot send all messages together.
Each client (user) provides us with an API key to use. For example, say we need to send emails on behalf of client using send grid. So the client provides us their API and we send out emails on behalf of them. But the issue is Sendgrid has rate limit so we cannot blast those calls.
Now that being said. The csv will have each line as a user to send email to. The csv May gave 50k lines.
The issue I am running into is. If I create a message for each row, then all of User 1 messages will be in front of queue with rate limit for its api key. When user 2 comes, their messages will be behind User 1’s and therefore until user 1’s msgs are not all sent the consumer will not be able to send user 2’s messages.
Question: how can setup queue archetype such that we handle the rate limit set by Sendgrid for each user’s api key and are still able process next user’s messages as they come in, instead of making them wait until the first user’s messages are sent.
Aucun commentaire:
Enregistrer un commentaire