Positional Anchor System

So just for fun I try to implement an anchor system for positionable objects, being the basis for a drag and drop UI system.

As such an object has a relative position (x,y) to a parent, if that parent exists. Else the given position is absolute.

When coming down to describing an area with (x,y,w,h) and having it's 5 + relevant anchors (corners + center and maybe center of edge anchors) I already get a load of problems:

If I want to reposition the object, I need to change all anchors. I can of course make the position of all anchors relative to another anchor that represents the position of the area, but if I move a single anchor I also want to move the object itself. Features like "snapping anchors together" and the like come to mind. But the interaction between all these objects starts to get really complicated really quickly. Connecting each anchor with the right others and not having messages being sent between each other on every occasion or getting into loops within the system seems like a daunting task.

Since looking for information about imlementing such an anchor system seems really tedious since I get 100% HTML Anchor questions in search results, I am asking: How do I start my journey into this nightmare? Are there well working patterns that I am already missing?