Design a tree-like hierarchy of ownership classes

I have a class, call it App, that will instantiate and own a set of classes of type Agents, each of which will instantiate and own a set of Listeners. Ownership is unique and non-transferable, hence obviously, std::unique_ptr. And ownees can vary in number, although they won't usually be to many, hence, a std::vector<std::unique_ptr> works good.

At compile-time, depending on some directive, I can tell which Agents is App going to need, and which Listeners is each Agent going to need. Agents to the App, or Listeners to each Agent, can vary in the future, but always at compile-time though, once the program is set to run nothing will change until a new version of the program is deployed.

So the question is, how can I implement all this tree-like hierarchy in an organised and future-proof way?

I tried implementing a Factory Method pattern, that will give the correct Agents to the App (just throw in a different factory depending on the compile-time flag), but then I'm a bit stuck at how to tell the Agents which Listeners they need in a way that is also clean. I though of again a factory of Listeners that take the Agent as a parameter, and then has a horrible switch statement querying the type of the Agent and giving him his Listeners, but that seems to get messy very quick.

PD: using VC2010, which is just partially compliant to C++11 and fully compliant to TR1.