Handle network requests/responses in a UI independent manner (and cancellable)

I have an app that is composed of 2 list fragments and 1 simple fragment with a layout view.
First list fragment shows a list of ~20 items.
Each item when clicked results in a network call that brings from a server ~70 items (max) that are displayed in another list in the second list fragment.
Once the user clicks in the list item data specific to that item are fetched from network and displayed in the last fragment.
So basically the first list is "broken" down to data. But the items in the second list are not more than ~70.

Design:
I have used volley queue for the network calls and functionality-wise it works fine and I display the data I need fine.
I have added a listener to the volley request and from my listener I update my adapter(s)

Problem:
Due to the fact that on click of the first fragment as I move to the second I start these backend calls and as I go to the third these calls are queued (I can see it in the logs), if I go to the third fragment the calls for the rest of the items are kept on firing and same if I go back to first.
Due to the fact that I get the responses for these requests from the Volley within the adapters and the actual fragment might not be "relevant" anymore, how can I design this better?