Strongly-typed Message Handlers in Mediator in C#

I'm currently having a problem implementing a solution for inter-object communication in my current project. I decided to try out an object like the mediator pattern, where objects communicate with each other using messages via a broadcast to the mediator. The mediator then sends the message to objects that have specifically listened to the message being broadcast.

Previous mediator objects I've used relied on holding containers to handlers only handling a base message, forcing listeners to cast the messages before they could handle them. I thought I could get by this by having collections of handlers themselves in a keyed collection according to some arbitrary type. By doing this, I'm hoping to avoid any kind of casting a listener has to do to respond to a message.

The trouble is, I can't seem to follow how to get an object that allows others to Register for a strongly-typed message to maintain a single collection of different kinds of strongly-typed message handlers.

Ideally, I'd like the public-facing interface to look as follows:

class Mediator
{
    private Dictionary<Type, ???> handlers; // how to I retain the strongly-typed handler


    public void RegisterForBroadcast<T>(Action<T> handler) where T : IMessage
    {
        // how can I turn the strongly-typed handler into something I can use?
    }

    public void UnregisterFromBroadcast<T>(Action<T> handler) where T : IMessage
    {
    }

    public void Broadcast<T>(T message) where T : IMessage
    {
        // how do I get the collection of handlers from where it's stored and use it?
    }
}

Is it possible with this design to have the strongly-typed message handling that I want? If so, how do I do it?