vendredi 27 août 2021

What is the best design for scheduling regular API calls?

I have a small(ish) MongoDB collection of 100 documents. Each of these has some fields, which are used in the API call (location, timezone, etc.). Each of these belongs to a profile based on their expected volume, and would dictate how often the API should be called, for that specific item, for example:

  • Profile A: API call every 5 minutes if current local time is between 0700 and 1900, otherwise every 30 minutes
  • Profile B: API call every 15 minutes, if 0700-1900 as above, otherwise every 30 minutes
  • Profile C: Every 30 minutes

During testing, I would call the API for each of these, but now I want to be more specific. What would be an efficient way of doing this(if this were to be hosted on the cloud, I don't want it to loop constantly multiple times per second)?

  • Easiest would be a loop that constantly checks
  • My gut is telling me to use the python scheduler? Example below:
def update_one(id):
    make_api_call(id) #gets the data based on the id specified
    process_response(id) #do whatever needs to be done with the information
    current_local = get_local_time(id) #get the local time for this place
    #find the next time this location is to be polled in the API
    next_task = get_next_call(id,current_local) 
    add_to_queue(id,next_task)

Or, should I rely on existing tools from cloud providers to schedule these things?

Aucun commentaire:

Enregistrer un commentaire