Why is Observer pattern usually portrayed as one to many?

While researching Observer pattern I have noticed, that the connection between the Subject and the Observer is usually one to many. Why so? Is there any particular reason why many to many relationship could cause any problems?