How to design a loader for different types of images?

I'm working on a side project that does matrix calculations over a large number of images. To achieve this, I'm using several different libraries (including libpng, libjpg, libtiff and easybmp) to load, maybe preprocess (i.e. grayscale or resize), and store the images. Each of these libraries stores the images differently, and this is where the issue occurs. I want to use strategy to have a base class named Image and multiple derived classes (i.e. ImagePNG, ImageBMP, etc.), and factory to instantiate an object depending of the type of the image I want to load.

One way I thought of solving this was by using void* or std::any inside the base class and store the object there. However, I'd prefer if the base class wouldn't have any objects inside it, only pure virtual functions and I'm not a fan of casting in C++.

Another issue is that I want the code to be faster, and using strategy and factory seems like it's going to slow it down drastically, which is why I thought of dropping them and use templates only. However, that would provide other issues, since templates are runtime and I'm not sure about the design since it would ask for a lot of template specialization.

The good thing is that I need to return all images as a pointer to uint8_t array to be able to process them, which means that only the loading part could be different for each type of the image.

The bad thing is that I still need to use some preprocessing that is already implemented in the libraries that I use. I could write it myself, but the libraries are out there for a long time, and I doubt that I could achieve a better performance by writing it myself. Also, the preprocessing isn't my final goal, which is why I'd rather not implement it myself if I don't have to.

Does anyone have any advice on how to design this? Or some feedback on the ideas that I presented?

Any help is very much appreciated!