Should serialization logic be in the entity or other class

Where should object serialization logic (the mapping of fields into XML or JSON names and values) be placed? Inside each entity object OR into a different set of classes only concerned with serialization? Any other best practices out there that relate to this question?

For example:

class Person {
    String name;
}

Some people go about it this way:

class Person {
    String name;
    public String toJson () {
      // build JSON, use 'name' field
    }
}

But if we also needed toXML(), toCSV(), toXYZ() keeping that direction would create horribly polluted code and break the Single Responsibility Principle which is already broken even with a single toJson method, IMHO.

Another option and this is what I typically do:

interface Serializer {  public String toJson (); }

class PersonJsonSerializer implements Serializer {
    private Person p;
    public PersonJsonSerializer (Person p) { this.person = p; }
    public String toJson () {
      // build JSON, use p.name
    }
}

then a factory hands out Serializers depending on entity types:

class JsonSerializerFactory {
    public Serializer getSerializer (Object o) {
        if (o instanceof Person) {
            return new PersonJsonSerializer ((Person)o);
        }
        else if (o instanceof Account) {
            return new AccountJsonSerializer ((Account)o);
        }
        // ... etc
    }
}

There would also be XMLSerializerFactory, CSVSerializerFactory, and so forth.

However, most of the times people want to have full control over the serialization and would not buy into it and prefer to have toJson methods inside each class. They would claim is way simpler and less error prone.

What is the preferred way to go, and are there better alternatives to implement a solution to this problem?