Should every class have a factory class?

It's my understanding that the main purpose for a factory class is to prevent classes from hardcoding references to each other. So instead of writing new Processor(options) I can call _processorFactory.GetInstance(options), which can be injected as a dependency.

Now when I make classes, I find I make the class, a factory class, an interface, and an interface for the factory, which seems a bit excessive. I have even had multiple implementations for a factory and found myself wanting to create a factory factory. What am I doing wrong? When should a class have a factory?