What java design pattern to avoid casting data entity

I'm currently implementing asynchronous-like event queue inside my application - it is designed to work like this: one component reacts to some user input and putting event to the queue, and another "listener" is checking whether there is event of specific type inside the queue and run it's own business logic

There can be various event types (like USER_MOUSE_CLICK, USER_KEYBOARD_CLICK etc) and every implementation has it's own "event-object" type

It looks like this (I'm ommiting constructors - setting all fields and getters/setters - they are just normal default ones):

public abstract MyEvent<T> {
    private EventType eventType;
    private T eventData;
}

public MouseClickEvent extends MyEvent<ClickPoint> { // ClickPoint class contains x,y of mouse click
    public MouseClick(ClickPoint point) {
        super(EventType.USER_MOUSE_CLICK, point);
    }
}

public KeyboardClickEvent extends MyEvent<String> { // character that has been clicked on keyboard
    public MouseClick(String key) {
        super(EventType.USER_KEYBOARD_CLICK, key);
    }
}

I have also a service with queue of MyEvent instances and the method to retrieve first event of provided EventType if exists - it's look like

...
private List<MyEvent> queue;
...
public MyEvent fetchMyEvent(EventType eventType) {
    for(MyEvent event : queue) {
        if(event.getEventType().equals(eventType) {
            return event;
        }
    }
    return null;
}
...

The problem is that when I'm trying to retrieve the event I need to cast it to specific implementation like

// some listener logic
MouseClickEvent event = (MouseClickEvent ) eventService.fetchMyEvent(EventType.USER_MOUSE_CLICK);
log("The X point of mouse click was: " + event.getEventData().x);

I don't like this casting - I feel like I have no control of a types when I'm fetching events and I see this situation like a 'weak point' and bug generator. Is there any design pattern to avoid this, or should I redesign whole system? Or maybe this is the only way and I should not care