How does using the factory design pattern stop a class having to anticipate the class of objects it must create

As I understand it the factory design pattern allows objects to be created through the use of a separate object that's sole aim is to create the first one. Different types of factory can be used to create different types of object.

I understand that this hides the instantiation of the primary object however surely this is just replaced by the instantiation of the factory?

A common advantage for this design pattern is that it stops a class having to anticipate the class of objects it must create. However surely if the factory is supposed to create a specific class the main class still needs to anticipate what kind of factory to use?

I assume I'm misunderstanding the main purpose of a factory?