Best approach to extend this C++ factory implementation?

I would like to extend an existing piece of code, and I'm not sure about the cleanest design approach to do so. I'm wondering if the existing design really supports the kind of extension that I'm thinking of.

There is a factory which looks like this:

class XYZFactory
{
public:
  static XYZFactory& getDefaultInstance() // so this is a singleton!
  // ... some create methods
  // std::unique_ptr<ABC> createABC();
private:
  std::unique_ptr<XYZFactoryImpl> m_impl;
}
---
XYZFactory::XYZFactory() : m_impl(std::make_unique<XYZImpl>;

Now the problem is that I would like to extend the functionality of XYZImpl by deriving from it. I would like to avoid to expose that implementation detail however in the factory class, like adding a separate XYZFactory constructor with a ExtendedXYZImpl as an argument to inject that extension.

Would killing the singleton design and subclassing from XYZFactory help?

Any ideas?

Thank you!

Mark