How to avoid downcasting in this specific class hierarchy design?

I've got an assignment to create a sort of a multi-platform C++ GUI library. It wraps different GUI frameworks on different platforms. The library itself provides an interface via which the user communicates uniformly regardless of the platform he's using.

I need to design this interface and underlying communication with the framework properly. What I've tried is:

1. Pimpl idiom - this solution was chosen at first because of its advantages - binary compatibility, cutting the dependency tree to increase build times...

class Base {
public:
    void show();
    // other common methods
private:
    class impl;
    impl* pimpl_;
};

//Framework A
class Base::impl : public FrameWorkABase{ /* underlying platform A code */ };

//Framework B
class Base::impl : public FrameWorkBBase { /* underlying platform B code */ };

However, to my understanding, this pattern wasn't designed for such a complicated hierarchy where you can easily extend both interface object and its implementation (e.g. if I wanted to create Button : Base, where Button::impl would inherit from Base::impl). Also if the user wanted to subclass button from the library UserButton : Button, the problem would be similar and also he would need to know the specifics of the pimpl idiom pattern to properly initialize the implementation.

2. Simple implementation pointer - the user doesn't need to know the underlying design of the library - if he wants to create a custom control, he simply subclasses library control and the rest is taken care of by the library

//Framework A
class impl : public FrameWorkABase { /* underlying platform A code */ };

//Framework B
class impl : public FrameWorkBBase { /* underlying platform B code */ };

class Base {
public:
    void show();
protected:
    impl* pimpl_;
};

Since the implementation is protected, the same impl object will be used in Button - this is possible because both FrameWorkAButton and FrameWorkBButton inherit from FrameWorkABBase and FrameWorkABase respectively. The problem with this solution is that every time i need to call e.g. in Button class something like impl->click(), I need to downcast the impl, because click() method is not in FrameWorkABase but in FrameWorkAButton, so it would look like this (static_cast<FrameWorkAButton>(impl))->click(). And excessive downcasting is a sign of bad design. Visitor pattern is unacceptable in this case since there would need to be a visit method for all the methods supported by the Button class and a whole bunch of other classes.

Can somebody please tell me, how to modify these solutions or maybe suggest some other, that would make more sense in this context? Thanks in advance.