Adding Map property in a Builder class

I have an existing Builder where I want to add a Map property. I was wondering whether I should always set it using

Collections.unmodifiableMap(map), or

this.map = map;.

I have seen the latter few times but somehow I don't feel it is right. Can someone shed some light over the convention here?