Inheriting a struct privately

Suppose we have this:

class Father
{
struct features
    {
        string hairColor = blonde;
        int age = 1;
        bool dumb = true;
    } m_Features; 

public:
    Father();
    ~Father();

    bool isDumb() { return m_Features.dumb; }
};

Then we have another class:

class FatherHandler : private Father
{
public:
    // constructors here

    void handle(Father *fa);
};

And now in the implementation:

void FatherHandler::handle(Father *fa)
{
    if (fa->isDumb)
    {
        m_Features.dumb = true; 
    }
}

The g++ compiler does not allow the above code because m_Features is private; however I'm inheriting from Father privately; therefore it should be allowed. How can the above code be modified such that FatherHandler can modify Father without triggering this compile-time error? Note that I inherit privately to denote the 'uses' criteria, rather than inheriting publicly which uses the 'is-a' criteria.