mercredi 5 juillet 2023

What pattern to use to make a IO-bound NodeJS process work at maximum efficiency (saturate CPU)?

I need to perform a process which (in an abstract way) goes roughly like this:

while (true) {
    try {
        const item = await getItemFromQueue();
        const request = transformItemToRequest(item);
        await callExternalService(request);
        await logItemComplete(item);
    } catch ( e ) {
        log(e);
        await returnItemToQueue(item);
    }
}

As you can see, this is a mainly IO-bound process. Getting the item from queue and logging completion are expected to be fast operations, since they only involve a local DB with a permanently open connection. The external service request however can be lengthy process. Some external services might even time out if there's a problem on the other end. We currently set a 10s timeout which is very generous, but we really need to get the data out there so we try to give it some time.

Items get added to the queue fairly rapidly, although it really varies depending on the time. At peak we currently have a little over 4000 items per hour. As the business grows, this number is expected to go up even more.

Obviously I'd like to make my process as efficient as possible. If an external service is being slow, I want to use the async nature of NodeJS to fire off more requests in parallel (either to the same or other external services). If for some reason our DB is bogged down and getting an item takes ages, the running requests can still be serviced. Etc. Idle CPU is a sin and should only be allowed when there really is no more work to do.

I also want to be able to run several instances of this program, in case a single CPU core is not enough to handle a burst of items.

But how to structure my code so that this would be the case? Plugging in awaits serializes the whole thing and I'm just doing a single item at a time. I could just start a new fetch-send-log iteration on every event loop tick (setImmediate() or process.nextTick()), but I'm afraid that the process might bite off more than it can chew. Memory is not an issue, but what if it spends most time fetching and preparing new items, while the stack of replies from external services that needs to be processed will continues to grow? It should fetch a new item only if there is nothing else to do, I think. But how to detect this?

Aucun commentaire:

Enregistrer un commentaire