Design alternative for objects with mutually exclusive data members

It was tough to summarize the problem in the title; so allow me to clarify the situation here.

I have a class that I'm designing that represents a BER TLV structure. In this specification, the "data" portion of the TLV can contain raw bytes of data OR other nested TLVs. To support both forms, I use the same structure but with two vectors (only one will actually contain something, depending on what we find as we parse the TLV data):

class BerTlv
{
public:

    void Parse(std::vector<std::uint8_t> const& bytes_to_parse);

    // Assume relevant accessors are provided

private:

    // Will be m_data or m_nestedTlvs, but never both
    std::vector<std::uint8_t> m_data;
    std::vector<BerTlv> m_nestedTlvs;
};

From the outside, after this object is fully constructed (all TLV data parsed), the user will need to detect what kind of data they are dealing with. Basically, they'd have to check m_data.empty(), and if so, use m_nestedTlvs. I'm not really happy with this approach; it smells like it lacks a better design.

I thought of some form of a union, although I do not think a real union would be appropriate here since vector data is heap allocated. So I thought of std::variant:

std::vector<std::variant<BerTlv, std::uint8_t>> m_data;

However, I'm worried this negatively impacts the std::uint8_t case since that's literally just byte data. It will now become non-continuous as well. The variant only benefits the nested TLV case and not by much.

Next I considered using the visitor pattern here, but I can't quite visualize what the interface would look like or how this would improve usability in both cases (raw data vs nested TLVs). Is visitor the right solution here?

Nothing I've thought of so far feels right, so I'm hoping for feedback on a better design approach to this problem. The general problem here is having data members that are sometimes unused or are mutually exclusive. It's a problem I run into in other contexts as well, so it would be great to have a general design approach to such a problem.

Note that I have access to C++14 features and below.