How to avoid casting from superclass?

I am developing a game with different game entities. It works quiet well, but i would like to get rid of some casting operations in my code. For example, when i am checking if a Bullet hits an Enemy, i need to cast both objects to able to reduce the health (property of Enemy) based on the damage (property of Bullet).

I store the entities in a map together with their corresponding classes. It looks like this:

Map<Class<? extends Entity>, List<Entity>> entities;

Here are my methods to put, receive and remove entities from the map:

void add(Entity entity) {
    Class<? extends Entity> type = entity.getClass();
            
    if (!entities.containsKey(type)) {
        entities.put(type, new CopyOnWriteArrayList<>());
    }
    
    entities.get(type).add(entity);
}

List<Entity> getAll(Class<? extends Entity> type) {
    return entities.getOrDefault(type, Collections.emptyList());
}

void remove(Entity entity) {
    getAll(entity.getClass()).remove(entity);
}

Finally here is my code (which runs in a game loop) to check if a Bullet hits an Enemy:

for (Entity bullet : data.getAll(Bullet.class)) {
        for (Entity enemy : data.getAll(Enemy.class)) {
            if (bullet.box.overlaps(enemy.box)) {
                // Bullet hits Enemy
                Bullet bulletHit = (Bullet) bullet;
                Enemy enemyHit = (Enemy) enemy;
                
                enemyHit.health -= bulletHit.damage;
                if (enemyHit.health <= 0) {
                    data.remove(enemyHit);
                }
                data.remove(bulletHit);
                
                break;
            }
        }
    }

Is there any way to avoid these casting operations for the Bullet and the Enemy? One solution i was thinking about, is to get rid of the the map and just use many list of those specific entity types, but this would infalte my code.