Does holding information in a Factory instead of passing it through calling code make sense?

I have a factory which creates objects that are widely used through my application. Some of the objects the factory creates require another dependency. This optional dependency is an interface/class with settings from a local settings file. Currently both the factory interface AND the settings interface are injected into something like 20 classes. Albeit, the number of classes would be less if the applications inheritance tree wasn't about 6-7 levels deep (meaning the IFactory and ISettings get passed through 6 constructors and finally arrive at a base class which assigns them to a field).

The deep inheritance isn't getting changed anytime soon, but I thought that I could lessen the load of these constructors by removing the ISettings from them and putting it in the IFactory implementation itself. The ISettings would be injected into the concrete factory through the constructor, and then instead of passing the ISettings as a part of the Create() method, it would just use the factory ISettings readonly property. These settings are guaranteed to be immutable once the program starts.

Is this frowned upon in any way, and are there better ways of going about this? Some coworkers think that Factories should almost never hold state...but they aren't exactly sure why.

These are similar questions that I found, that gave answers but I'm still looking for a more fleshed out answer if possible.

Adding Properties to Abstract Factory

Abstract Factory Pattern and Properties

Simple example:

interface IObjValidator
{
    bool Validate();
}

interface IObjValidatorFactory
{
    IObjValidator Create();
}

class ObjValidatorFactory : IObjValidatorFactory
{
    // Is it OK for the factory to hold this field?
    private readonly ISettings settings;    

    public IObjValidator(ISettings settings)
    {
        this.settings = settings;
    }    

    IObjValidator Create(ObjValidatorMode mode)
    {
        switch(mode)
        {
             case ObjValidatorMode.One:
                 return ObjValidator1();

             case ObjValidatorMode.Two:
                 return ObjValidator2(this.settings);
        }
    }
}

class ObjValidator1: IObjValidator
{
    public bool Validate()
    {
        // Do stuff.
    }
}

class ObjValidator2: IObjValidator
{
    private readonly ISettings settings;

    public ObjValidator2(ISettings settings)
    {
        this.settings = settings;
    }

    public bool Validate()
    {
        // Use ISettings field data here.            
    }
}