Mapping subjects to their observers - Observer pattern GoF book

In the GoF design patterns book, when it comes to the implementation part of the Observer pattern, it is stated:

Mapping subjects to their observers The simplest way for a subject to keep track of the observers it should notify is to store references to them explicitly in the subject. However, such storage may be too expensive when there are many subjects and few observers. One solution is to trade space for time by using an associative look-up (e.g., a hash table) to maintain the subject-to-observer mapping. Thus a subject with no observers does not incur storage overhead. On the other hand, this approach increases the cost of accessing the observers.

I fail to see how using hash table would improve storage capacity. In Java, for every subject we could have a list of observers List<Observer>. If there are no observers attached to this subject, the list would be empty. If we use hash table, Map<Subject, List<Observer>, we still have the list, but we also have a reference to the subject, so this way is a bit more memory inefficient. I don't know whether it is relevant, but the languages used for implementation in the Gof book are Smalltalk and C++.