samedi 14 décembre 2019

Wha's a scalable way to query a rate limited API?

I need to build a service which makes REST calls to 3rd party service. The 3rd party service is rate limited for every minute, day and month. Once the data is fetched, it is stored in a data store. We need approximately 60,000 calls to fetch all the data required.

Currently, I have created a SQS which stores the Job information. And Lambda picks up the task and performs an API call to the service and stores it in the datastore.

When the lambda hits the rate limit ( 429 status code ), the SQS should stop sending messages. Or is there a better way to perform this ?

How do we handle for these rate limits ?

Thanks.

Aucun commentaire:

Enregistrer un commentaire