Factory pattern for child classes with changing parameter/ return types in virtual functions

I would like to write a factory for classes which essentially only differ in the type of message they process, but otherwise are completely the same. I thought about writing a child class and let them inherit from it to have a common ancestor. Since the message types the children have to handle differ, I thought about using a template class as common ancestor, pretty much like in the following example.

template <typename T>
class Parent 
{
public:
  virtual void Set(T message); 
  virtual T Get();
}

class ChildA : public Parent<MessageType_A>
{
public:
  void Set( MessageType_A message);
  MessageType_A Get(); 
private:
  MessageType_A mMessage;
}

class ChildB : public Parent<MessageType_B>
{
public:
  void Set( MessageType_B message);
  MessageType_B Get(); 
private:
  MessageType_B mMessage;
}

However, due to the template in the parent, I don't have a common base class to use for the factory and can't think of a way to solve the problem. I was thinking of using std::any, unions or variants but these introduce only more obstacles to my problem instead of solving them. Would anyone have an idea of how I could create a factory with my described requirements?