How to use abstract factory on interacting widgets

Context

In many GUI frameworks, it is very common that widgets are placed in a container to arrange them visually in a window. For example, in Gtkmm, a window with a label and a button (side by side) will look like this:

int main(int argc, char *argv[])
{
    auto app = Gtk::Application::create(argc, argv, "no.abstract.factory");

    // Create widgets to appear in the window:
    Gtk::Label label{"A label"};
    Gtk::Button button{"A button"};

    // Create a container to hold widgets and
    // add widgets inside the container at
    // specific locations. This gives control
    // on where widgets will be located in the
    // window when displayed to users.
    Gtk::Grid container;
    container.attach(label, 0, 0, 1, 1);
    container.attach(button, 1, 0, 1, 1);

    // Create the window and add the container
    // to it:
    Gtk::Window window;
    window.add(container);

    window.show_all();

    return app->run(window);
}

To avoid depending too much on the GUI framework, the abstract factory pattern could be used, as described in the book Design Patterns, by Gamma and al. Using this pattern, interfaces (or abstract classes) are created to represent the widgets and constructor calls are hidden behind factory:

IGUIFactory* factory = new GtkmmGUIFactory();  // Or QtGUIFactory() ...

// Later in the code (we now have no idea what is the
// underlying GUI framework) ...

ILabel* label = factory->CreateLabel();
IButton* button = factory->CreateButton();

// ...

With this approach, the application code does not depend on the GUI framework, except for the factory, which means there is minimal work to be done if another GUI framework is to later be adopted.

Question

In the Gtkmm example above not only are widgets created, but they are also added to a container for arrangement in a window. The container is then itself added to the window.

With the abstract factory patten approach, however, with interfaces hiding away all of the GUI framework's details, how can I actually arrange my widgets in the container? If I create an interface to abstract the container, The call to Gtk::Grid::attach will never be accessible because the widgets it takes as its first argument will be of the interface type, not the Gtkmm type. I could also use casting, but that would mean the framework's widget interfaces is derived from publicly and I want to avoid that.

So in this context, how can GUI elements interact with each other once they are created and the GUI handles are lost? For example, how could we re-write the Gtkmm example above but in an abstract way, so as to not depend too much on the framework?

Note: my example is in C++ but I don't mind if the answer is not, as long as the answer is clear.