Destroying created objects from factory (dangling pointer issue)

Suppose I have a factory class with a creation method that creates objects and also has a shutdown option that invalidates all the created objects (so they cannot be used anymore).

something like that (C#):

public class MyFactory {
    public MyObject CreateObject() { ... }
    public void DestroyAllObjects() { ... }
}

The clients of this code create objects, cache the reference and use it. Objects can be either manually destroyed or will be destroyed by calling 'DestroyAllObjects' in the factory class.

The problem here is a 'dangling pointer' issue. How can I notify the client code that the cached object was invalidated and shouldn't be used anymore? I was thinking of either using callbacks (con: complicates client code) or check on each call that the object is still valid (con: complicates MyObject code + code dup).

I understand that in an ideal world the objects will destroy themselves, but the system has a requirement for a centralized shutdown method.

Would love to hear your thoughts if someone encountered a similar issue in the past.