Design pattern for building a data structure in Java (merging a bunch of sources)?

I have a data structure, looks something like this:

@JsonInclude(JsonInclude.Include.NON_NULL)
@JsonPropertyOrder({
    "details",
    "interests",
    "goals"
})
public class Person {

    @JsonProperty("basics")
    @Valid
    private Basics basics;

    @JsonProperty("interests")
    @JsonDeserialize(as = java.util.LinkedHashSet.class)
    @Valid

    @JsonProperty("interests")
    @JsonDeserialize(as = java.util.LinkedHashSet.class)
    @Valid
    private Set<Interest> interests = new LinkedHashSet<Interest>();

    @JsonProperty("goals")
    @JsonDeserialize(as = java.util.LinkedHashSet.class)
    @Valid
    private Set<Goal> goals = new LinkedHashSet<Goal>();

    public Person(Basics basics, Set<Interest> interests, Set<Goal> goal) {
        this.basics = basics;
        this.interests = interests;
        this.goals = goals;
    }

    /**
     * 
     * @return
     *     The basics
     */
    @JsonProperty("basics")
    public Basics getBasics() {
        return basics;
    }

    /**
     * 
     * @param basics
     *     The basics
     */
    @JsonProperty("basics")
    public void setBasics(Basics basics) {
        this.basics = basics;
    }

    public Person withBasics(Basics basics) {
        this.basics = basics;
        return this;
    }

    // All Setters, Getters, Etc

    @Override
    public String toString() {
        return ToStringBuilder.reflectionToString(this);
    }
}

I created this really weird noargs PersonBuilder class to take a bunch of different sources of Person data and build the data structure. In my mind, I started modeling it after StringBuilder, but looking at it now, it's a mess. It makes sense to me, but I know I would hate to use this if I didn't develop it myself. Here it is (and an example app that uses it):

public class PersonBuilder {
    private Person person = null;

    public PersonBuilder() {
    }

    public void addFaceBook(Person p) {
        if (person == null) {
            person = p;
        } else {
            // Bunch of logic to merge the FaceBook Person in
            // E.g., only add interests from Facebook if they are not defined.
            if (person.getInterests() == null) {
                person.setInterests(p.getInterests);
            }
        }
    }

    public void addTwitter(Person p) {
        if (person == null) {
            person = p;
        }
    }

    public Person getPerson() {
        return person;
    }
}

public class App {

    public static void main(String[] args) {
        PersonBuilder personBuilder = new Person();

        // Get a result from Facebook let's say
        personBuilder.addFacebook(result);

        // Get a result from Twitter
        personBuilder.addTwitter(result);

        Person person = PersonBuilder.getPerson();
    } 
}

Basically, the idea is, I want to build my data structure. Each source of the data structure has logic of what it takes (e.g., if I got the bio from Twitter, don't take the bio from Facebook, but if it's null, then take it). In the end, after I've taken all the sources in, I want a Frankenstein monster of mixed parts from all the sources, mashed together into my single data structure. I'm accomplishing it, but my code is hard even for me to use. In essence, I'm not asking how to do this, but rather how to do it well.

I understand the Builder pattern (from Effective Java), and the AbstractFactory Pattern, but this is a bit different (or at least it seems so in my mind) due to the fact that I'm essentially creating this single behemoth of an object.