jeudi 21 novembre 2019

Concurrent async requests - keep track of necessary timeout

With Nominatim requests I need to keep a timeout of 1500ms between every requests. How do I handle that if multiple concurrent running processes keep firing on that function?

Code:

osmService:

export const getLocation = async (zipCode, street) => {
  if (!zipCode) {
    return {
      address: null,
      cacheHit: false,
    };
  }

  const cacheData = await getAddress(zipCode, street);

  if (cacheData && cacheData.length === 1) {
    return { address: cacheData[0].address, cacheHit: true };
  } if (cacheData && cacheData.length > 1) {
    logger.error('Found multiple address, this should not be', zipCode + street);
  }


  try {
    const responseObj = await getNominatimLocation(zipCode, street);
    if (responseObj) {
      const returnObj = {
        ...responseObj.address,
        lat: responseObj.lat,
        lon: responseObj.lon,
        displayName: responseObj.display_name,
      };
      await insertAddress(zipCode, street, null, returnObj);
      return {
        address: returnObj,
        cacheHit: false,
      };
    }
    return {
      address: null,
      cacheHit: false,
    };
  } catch (ex) {
    logger.error(`Error getting location from ${zipCode} ${street}`, ex);
  }
  return {
    address: null,
    cacheHit: false,
  };
};

As you can see I have a caching layer in between. So when the request was hit by cache I do not need to wait 1500ms.

export const getNominatimLocation = async (zipCode, street, query, retries = 0) => {
  if (retries > 5) {
    return null;
  }
  try {
    const qs = {
      format: 'json',
      q: query,
      postalcode: zipCode,
      addressdetails: 1,
      country: 'Deutschland',
      street,
      'accept-language': 'de',
    };
    const response = await requestPromise()
      .get({
        url: OSM_SEARCH,
        qs,
        timeout: 12000,
        headers: {
          'User-Agent': 'xxxxxxx',
        },
      });
    return JSON.parse(response)[0];
  } catch (ex) {
    logger.info(`Nominatim timed out - retry ${retries}`, ex);
    await timeout(9000);
    return await getNominatimLocation(zipCode, street, query, retries + 1);
  }
};

Since Nominatim often times out I need to do this recursive call (does not need to be recursive - was just easier).

Now let's suppose I have following jobs (backend engine) that want to get locations

const jobA = asnyc () => {
  const {address, cacheHit} = await getLocation(10243);
  if(!cacheHit){
    await timeout(1500)
  }
}

const jobB = asnyc () => {
  const {address, cacheHit} = await getLocation(10245);
  if(!cacheHit){
    await timeout(1500)
  }
}

const startJobs = async () => {
 Promise.all([jobA(),jobB()])
 console.log('all jobs done');
}

The jobs partially represent my current code structure. In my code the jobs do more (call other services etc.).

Now when I have this layout - how can I make sure to keep 1500ms between every Nominatim call when there is no cacheHit?

Aucun commentaire:

Enregistrer un commentaire