Is it possible to avoid casting in this scenario?

I'm trying to implement a Status Effect system on my game. I'll be calling Status Effect S.E to keep it short.

The primary focus on the game is on the magic system, and of course there shall be S.E's being applied to enemies you hit with certain spells. Common things like Slow, Stun, Snare, etc are present, but there shall be special S.E's with unique interactions(like Freezing the enemy when he's slowed bellow a threshold percent, triggering explosions when you hit an enemy with a fire spell and other crazy things you can imagine).

Since some S.E's types(like slow) require knowing all other S.E's of that type for the effect on the Actor movement to be applied correctly, the S.E's are applied to a IManager. Concrete examples on the usage of IManager: S.E's apply Decorators to a certaing actor component, and when one effect ends, things need to be restored and reapplied to avoid bugs. Increases and reductions to movement speed are applied like this: Flat changes first, Percent last, and percent are accumulative(10% slow now + 10% slow later result in 20% total slow if both S.E's are active)

In code, the interfaces looks like this:

public interface IReceiveStatusEffect
{
    Dictionary<string, IStatusManager> StatusManagers { get; set; } //ID of the effect and the Manager for that Effect
    void ReceiveEffect(IStatusEffect effect);
}

public interface IStatusEffect
{
    IStatusManager Manager { get; set; }
    GameObject Target { get; set; }
    float Duration { get; set; }
    string ID { get; } //The ID is Used on concrete implementations to get the SpellManager from IReceiveStatusEffect
    void ApplyEffect();
    void RemoveEffect();
}

public interface IStatusManager
{
    List<IStatusEffect> AllEffects { get; set; }
    void ProccessEffects(); // Called when an Effect is removed and some things needs to be recalculated. Sometimes call Apply again from the remaining AllEffects
}

An Actor that can receive an S.E need to have a IReceiveStatusEffect, so the flow is like this:

ConcreteStatusEffect status = new ConcreteStatusEffect(target, duration, IReceiveStatusEffect.StatusManagers);
// Constructor gets the correct manager from StatusManagers or create it if there's none
IReceiveStatusEffect.ReceiveEffect(status)

With everything (over)explained, here's the problem i'm facing: Since S.E's are quite diverse, some will require knowing concrete implementations of something(like concrete implementations of IStatusManager), like the Slow S.E example i gave earlier. I tried simply casting, and it worked, but pretty much everywhere i look everyone says that casting shall be avoided at all costs; That if you need to cast, there's probably a bad design there; That casting is an anti-pattern, etc

But I just have no idea on how i can get the information i need right now without casting. I thought and thought, but this flow seems to be the best(if not for the casting) for what i currently need, and can't think of another design pattern for the problem, so I'm trying to find a solution that does not use castings since it seems frawned upon.