Self removable resource from flyweight

I'm working on application with different resources which I need to close, while working with reactive streams.

I've got factory based on flyweight pattern which keeps references to objects, and they implements AutoCloseable interface. Problem is i'm using close() inside Autocloseable class, and here is my question: what is best solution to remove reference to closed resource inside factory? Can I throw some kind of event and catch it in factory, or after every action that can close resource should I iterate through references map and remove closed resources?

For better context: I'm using reactivex Observable which emits directory event (create, remove file/directory), and after every subscriber unsubscribed to it i'm closing WatchService which i'm using.