Removing jQuery for performance reasons justified?

I am on a new project and our job is to rewrite an e-commerce website that is having performance problems on mobile devices.

We are re-writing the javascript based on a more object-oriented/modular architecture which I think is great! However my team lead said that we should remove all the jQuery calls and replace with javascript like so domElem.querySelectorAll(query) , which has better performance. I understand jQuery does some kind of caching in the background which can create memory issues.

I am a little sceptical of this, firstly because it seems like a case of 'premature optimization', that is, we should find the bottle-necks first before we re-write anything. And secondly I haven't found anything on the internet that says that jQuery has significant performance problems.

The current website does have a lot of overlapping dom branch queries which I think creates a lot of redundancy. That is there is too much querying happening, and on our new architectual approach we are restricting our object/module to fewer dom queries and more targeted dom queries which is great. This does need to be re-written.

But whether or not we use domElem.querySelector(query) or $(domElem).find(query), I can't see there as being much of a difference. Is my thinking right?