Inheriting nested class doesnt allow accessing the nested class' owner (class) non-public data

The following sample demonstrates my problem. I'm trying to deserialize an [complex] object from a file and that object (class) should provide only const getters to the final user. All the deserialization process should be done by the Parser interface, which can be a JSONParser and/or XMLParser (using their respective 3rd-party parsing libraries).

#include <string>

namespace tmx
{
    class File
    {
    // The getters
    public:
        static File& from(std::string path);

        const int getData1();
        const int getData2();
        // ...

    // The "hidden" (non-public) setters, used in the parsing
    protected:
        void setData1(int v);
        void setData2(int v);

    // The hidden constructors. File class must be created through the static function
    private:
        File();
        File(std::string path);
        File(const File& copy);

    // The public Parser, so that it can be inherited
    public:
        class Parser
        {
        public:
            virtual ~Parser() { };
            virtual void parse(File& file) = 0;

            void test(File& file)
            {
                file.protectedData1 = 123; // Access protected data normally! :)
            }
        };

    // The "hidden" (non-public) file data
    protected:
        int protectedData1;
        int protectedData2;
        // ...
    };

    class XMLFilePaser : public File::Parser
    {
    public:
        void File::parse(File& file) override
        {
            file.protectedData1 = 123; // Cant access protected data :(
        }
    };
}

As we can see, the nested Parser class can freely access the File's data, no matter if its private or protected, but the inherited parsers can't. What is the reason and how could I accomplish this WITHOUT USING BOOST?

As far as I'm concerned, I kinda could use frienship (make Parser friends of File), but still, I couldnt inherit the Parser's friendship (only its methods/attributes).