Should we always use a factory instead of creating an object using the 'new' keyword?

I've read at many places that object creation using the 'new' keyword should be avoided in general. Is there any case where creating an object directly in the client code is a better idea than using a factory class?