Rust generic "component registration" pattern

I'm learning Rust, and am running up against a wall to do with managing ownership.

I have some different structs implementing trait Device (which actually simulate real hardware attached on a physical bus, with a generic interface). Each is assigned a unique id : u16. These Devices should all be a registered by their ID in a struct Controller. Sometimes the Controller is sent generic messages with an attached ID and generic command which the Device should perform (as accepted by the trait). I'd like to be able to be able to register different components on the fly, and all of the communication/mutation of the components should go through the Controller, so it seems natural to me that the Controller should own via Box<> es all of the registered components.

The problem is that there is a particular special device Pic implementing Device which should always be on the bus, i.e. registered in the Controller (in the real hardware, it is soldered on, not plugged into an expansion slot). The rest of the simulated hardware needs to be able to read and mutate the Pic directly by calling special functions only it has. At least this is implemented on the real hardware this way (wires go directly from these extra parts of the hardware directly to the Pic, bypassing the IO controller). But, once the Controller generically registers the Pic like any other object, it forgets its type and just knows it implements trait Device, so the other parts of the system can't access it anymore. I don't want my controller to have to know about the Pic specifically as a device, keeping a special space for one, and have special cases e.g. for looking it up as an ordinary device.

I think the natural thing to do is to keep the owner of the Pic somewhere else, and make the Controller not own any devices. Then problem is that the controller still needs to mutate the Devices sometimes, so would presumably need to register mutable references. Then the borrow-checker would prevent anyone else from obtaining a mutable reference to the Pic by any other means anyway.

(Obviously I need some kind of redesign here, but) my question is: How can I avoid this problem? I think I can store all my Devicees in RefCells just for the Pic's benefit but this seems like a bit of a hack (and incurs a runtime penalty).