Declaring Fields in the most Specific Way [duplicate]

This question already has an answer here:

• What does it mean to “program to an interface”? 27 answers

Most of the time when we learn to classes in Java, we learn to declare a field in the most abstract way, such as this:

private Set<String> id_values;

I recently heard a convincing argument for declaring fields in the most specific way possible, such as this:

private LinkedHashSet<String> id_values;

The reasoning behind this is that by declaring your field in the most specific way possible, it makes it clear that your class was written with that very particular object in mind. If you return a reference through a method or something, that could still be abstracted out, such as this:

public AbstractSet<String> getIDs() { return id_values; }

In this way, if you refactor the class in the future and change from a LinkedHashSet to HashSet, you don't have to refactor every reference that calls the public methods, but you do have to think through every method that touches the original class. This adds a bit of complexity if a refactor is needed. For instance, the declaration needs to be changed, and all of the methods that touch id_values would have to be manually checked. However, that seems like a pretty decent defensive programming practice.

My question is this, aside from the additional overhead of refactoring the declaration, what are the other caveats of declaring fields in the most specific way possible? I was always taught to declare things in the most abstract way possible, but this argument was pretty convincing.