One-off creation with the Factory Pattern

I'm putting together a game whilst trying to conform to various design patterns, the factory design pattern being one of them. There are certain game elements that get created constantly e.g. enemies, bullets. Then there are elements that only get created the initial one time like the background element for instance.

In cases when your only doing one-off creations of objects is there any point in using the factory design pattern? or would it be more appropriate to have a mixture of directly created objects and factory created ones?