Architecture of C++ matrix classes: inheritance or specialisation?

I would like to define a class : template <class T, std::size_t M, std::size_t N> class Matrix with simple and classic methods applied to any matrix (std::array<std::array<T, N>, M> m_data;).

I then want to have specialised classes such as: template <class T> class Matrix<T, 2, 2>. Ideally I want to be able to use the generic methods of the first class in the specific class. To do this I was thinking of creating a template class <class T> class Matrix2x2 : public Matrix<T, 2, 2> to have heritage. The problem is that to use for example the method class Matrix<T, M, N>::transpose() from Matrix2x2 I have to cast the value returned in Matrix2x2, something I'd rather avoid. Matrix2x2<T> and Matrix<T, 2, 2> are completely equivalent in practice, so using class specialisation seems appropriate, but then you have to rewrite all the methods and either duplicate some code or create an instance of the non-specialised class and I want some specific methods for specific dimension (e.g. rotation) in each corresponding method.

I've tried using inheritance, it works but I have a lot of code duplication, I'd like to refactor all the code. What's the best design? Is there a better solution?

Thank you for your help.