vendredi 6 février 2015

Best practice for implementation of decoupled services and dependencies

I have a software component; it is a simple thing which reads from an MQ and does a DB read-write. The component itself can be seen as a client which has two threads



  • one thread that perpetually consumes from an MQ and writes the message into a DB

  • another thread that polls the same DB and does an action (HTTP-POST to some external target based on what's mentioned in the message read from DB). The schematic of the component is below.




+------------------------------------------------------+
| |
| Component |
| +----------------------+ |
| | | |
| +-------+ | Multi-threaded | +-------+ |
| | +----->T1 gluing client T2+-----> | |
| | DB <-----+ which would be <-----+ MQ | |
| | | | a daemon | | | |
| +-------+ +----------------------+ +-------+ |
+-+-------+-----+----------------------+-----+-------+-+


This client would be a long running process, so it would have to be daemonized. Now, I have a dilemma in terms of the design. The options on the table are




  1. Add check points in the daemon startup to ensure that the backing services (DB, MQ) are available and usable (able to bind to the queue, able to connect to the required DB etc). And in case if the DB or MQ goes down, the daemon should stop and cease to run. All of these would be logged as well. In summary, if the daemon is running, it implies that daemon<->DB and daemon<->MQ connection links are up as well.




  2. Make this daemon totally independent and implement wait/reconnect logic for MQ/DB reader threads. Irrespective of daemon<->DB and daemon<->MQ connection being up or not, the daemon keeps running. In case of unavailability of the backing services, the threads would wait and try to reconnect, infinitely.




Points to ponder:



  • With option #1, checks being done at startup would guarantee that if this client is running fine, it would guarantee that the backing DB and MQ are also up and running.

  • But with option #1, since the design is to stop the daemon in case any of the backing services went down, it would mean that this daemon should also be restarted when the backing services are restarted.

  • Option #2 would make services totally independent and thus avoid the need to restart this daemon if any of the backing services were either restarted or got disconnect even for a short while.


Which of the options #1 and #2 would be better? This component is a very critical component in the infrastructure and lot of other services depend on this component to work fine on production. Please share your advice/inputs on this design and the best practice that could be adopted.


Thanks!


Aucun commentaire:

Enregistrer un commentaire