lundi 20 juin 2022

Make microservice application resilient to db downtime

We have a microservice application which is saving the data into an Oracle Db.

So far the DB is our single point of failure which we want to improve (we are using a single Oracle DB with a cold failover instance).

Now the company is asking us to upgrade the oracle DB, the issue is that it requires downtime. For that reason we were thinking about:

  1. add a global/geo replicated cache layer (e.g redis) between the microservice and the DB
  2. for each new record that should be saved on the db:
    1. Add the record in the cache (storing the entries on the HD in case the whole cache layer crashes)
    2. throw an event to a queue (we have RabbitMQ). On the other side of the queue we can create a new service to consume the events and add them to the DB in an asynch way.

It's basically adding a write-behind cache layer.

In the above scenario we are confident that we can save easily 1 week data in the cache or more.

If the DB is down the new service which is listening to the queue will simply re-trying adding the rows in the DB, as soon as an event is added to the Db then the event can be ack and the next one will be consumed. In this way, if the DB is down or if we have to do some maintenance, it should not affect the main application: the users can still "save" the data and retrieve it (with the 1 week max constraint whenever the db is down).

The down side is that the architecture is more complex and we can have now data eventual consistency.

Is there another design pattern to better deal with database downtime without having the users feel that something is wrong?

Do you know any already-existing tools that we can use to automatically read an event from Rabbit and save it in the db? (we are already doing it with logstash to automatically forwards some rabbit events to elastic).

The next step would be to have a cluster of DB (cassandra,mongo etc) but for now we do not have the capacity for that.

Aucun commentaire:

Enregistrer un commentaire