std::vector with type known at runtime

I am writing a small C++ library to process audio files. The file header tells me the number of bits per sample (8 bits, 16 bits, etc.). Ideally, I'd like to store that in a std::vector<T>, where T=std::uint8_t for 8 bits, std::uint16_t for 16 bits, etc.

1. One approach would be to just store everything as a bunch of bytes and use type punning all over the place. I am not a huge fan of this method. I'd rather do it once when reading the data and then proceed with a nicely typed vector.

2. I could also use the largest possible type and call it a day, but this seems rather wasteful and performance would likely take a hit.

3. Else, I could template my entire codebase on the bits per sample type, explicitly instantiate all the valid widths and have some factory function pick the appropriate implementation. Is that a reasonable thing to do in audio engineering?