Should the caller or callee access object fields? (Java)

I have a Spring web service. When building the Response, our ResponseBuilder class uses a Context object. Every field on Response has its own associated business logic and [Name]FieldBuilder Singleton class, which operates on some information stored in Context. We want to avoid RequestScope and storing state in these classes in order to prevent Java from instantiating too many objects (we are working with >500tps at this time).

The Context object has Media, Campaigns, ClientInfo, etc. Inside each of these internal objects are further fields. Each Response field only needs some part of the information stored in Context.

From a design perspective, should I pass Context as-is to every FieldBuilder, and have them unpack it? Or should the unpacking be done in ResponseBuilder and the method signatures of the FieldBuilder be more specific?

Please let me know if more context is needed, thanks.