Bidirectional association, what's better?

Not really a problem, but I wondered many times about how to implement double association in code...

Let's have an example. A man car wear a single hat and a hat can be on a single man. But if the man wears the hat, the hat's owner have to be that man.

Here is how I used to do it (Java) :

In Character :

public void setHat (Hat hat) {
    Hat former = this.hat;
    this.hat = hat;
    if (former != null && former.getOwner() == this)
        former.setOwner(null);
    if (hat != null && hat.getOwner() != this)
        hat.setOwner (this);
}

And in Hat :

public void setOwner (Character c) {
    Character former = this.owner;
    this.owner = c;
    if (former != null && former.getHat() == this)
        former.setHat(null);
    if (c != null && c.getHat() != this)
        c.setHat (this);
}

This way, the man's hat and the hat's owner are always coordinated.

But... that sounds a bit clumsy to me. What's the better way to implement this kind of conditional setter, please ? It's very useful to me.

(if I made mistakes, that's just because baguette ok? ^^)