Decorator pattern: enhanced design?

I'm reading and example provided in "Head first design patterns book" about decoration pattern.

I have noticed 2 things:

1. if you will need to remove a decorator from the stack of the warped decorators, you will have to iterate one by one through the component reference, which is O(n) complexity.
2. Conceptually I find it wrong to wrap(encapsulate) the base component in to the decorator object, it should be reversed, the component object should encapsulate the decorating objects.

I'm new to design patterns, and there is high probability that I'm wrong, please explain to me what is specifically wrong with the way I think so I can learn.

I have created a different design , which solves the problems that I have mentioned maybe they add new problems, please feel free to point the issues.

here is the UML Diagram of the suggestion:

basically what I did, is that I have created a dictionary in the Component class which saves which decorators have been added, and made the Decorator abstract class not inherit from the component yet from Interface(so the component abstract class).

in this way, we can remove any decoration we want with O(1) complexity, and it is more logically constructed in the way that the component wraps the decorator, not the vice versa.

I understand that maybe I didn't noticed some advantage of the original Decorator pattern design, please advice me.