Solve the virtual pure method and templates common issue

(For simplicity sake, serializer will be called write and deserializer will be called read)

I'm writing a C++ game serializer from scratch with no library allowed.

The main concern i have is to keep the read and write in sync (the read values must be the same as the written ones). So the Packer handles both tasks and is specified with an enum.

What i have

enum PackerType {
    WRITE,
    READ
}

template <PackerType PType>
class Packer {
    char *buffer; // Packer will write here
    uint32_t index;

    template <typename T>
    void Pack(T & value); // Calls appropriate functions depending on PType
}

What i need

void WriteAndRead(Packer & p) {
    p.Pack(32);
    p.Pack("Hello World");
}

Packer<WRITE> wpacker;
Packer<READ> rpacker;
rpacker.buffer = wpacker.buffer;

WriteAndRead(wpacker); // Will write everything in wpacker.buffer
WriteAndRead(rpacker); // Will read wpacker.buffer

So i know this is not possible in C++, but what i'm looking for is an elegant way of dealing with this issue. I'm already aware of type-erasure, but i'm not a fan of the solution.