Map with a multiple types of generic items for value

I have 3 different Set act as a cache and each serves for different reason. Additionally the keys are of different types e.g. Integer, String etc
I was thinking to create a wrapper class around this, but then I thought of just having them all as part of 1 hashmap and based on the key I can pick the proper Set
But I am not sure what is the proper way to do that.
I'd like to avoid something like:

private final Map<Integer, Set<Object>> cache = new HashMap<>();

public boolean exists(Integer type, Object key) {
    return cache.get(type).contains(key);
}

public void addKey(Integer type, Object key) {
    if(type == CACHE_1) {
        Set<Object> set = cache.get(type);
        if(set == null) {
            set = new HashSet<>();
            cache.put(type, set);
        }
        set.add(key);
    }
}  

Is there a way to make it more type specific?

Update
These can be called as:

addKey(CACHE_1, "foo");  
addKey(CACHE_2, 123);  

or

if(exists(CACHE_1, "foo")  
if(exists(CACHE_2, 123)