Design Pattern for Loading Config File into Complex Singleton

I am writing a service that maintains the state of a graph (nodes, edges, occupants on nodes, etc.).

There are various configs for the nodes, edges, occupants, etc. in 2 XML files.

I'm wondering what the best way is to structure this "singleton" for the graph representation is, especially in the context of using Dependency Injection.

Originally, I was thinking having something like this:

public class XmlClass1 {
    [XmlElement("asdf")]
    public int asdf;
    ...

    public static XmlClass1 GetXmlClass1FromFile(string xmlPath1) {
        // do some serialization into an instance of XmlClass1 and return it
    }
}

public class XmlClass2 {
    [XmlElement("things")]
    public List<Node> Things;
    ...

    public static XmlClass2 GetXmlClass1FromFile(string xmlPath2) {
        // do some serialization into an instance of XmlClass2 and return it
    }
}

public class Node {
    public int Id;
    public List<Edge> Edges;
}

public class Edge {
    public Node Source;
    public Node Destination;
}

public class Graph {
    private List<Node> _nodes;
    private List<Edge> _edges;
    private List<Occupant> _occupants;
    ...

    public Graph(string xmlPath1, string xmlPath2) {
        _xmlPath1 = xmlPath1;
        _xmlPath2 = xmlPath2;
    }

    public void Init() {
        // read in configs from _xmlPath1, _xmlPath2 and set all the private fields step by step, e.g.
        // 1. Read in Node Ids and create Node objects.
        // 2. Read in Edge Ids with sourceNodeId and destNodeId. Look up these Node ids from the list created in step 1. Create Edge objects.
        // 3. Assign Edges to each Node.
        // ...
        var xmlClass1 = XmlClass1.ReadFromFile(_xmlPath1);
        var xmlClass2 = XmlClass2.ReadFromFile(_xmlPath2);
        ...
    }

So I'd do something like var graph = new Graph(); graph.Init(); and then inject this instance of Graph everywhere. I could then define public methods on Graph to do fancy lookups.

Pro: - XML (de)serialization logic is in XMLClass1 and XMLClass2. All the XML annotations and definitions are there so I don't have to pollute Garage with it. Con: - When I pass around Graph as a dependency, it's going to have all these random initialization functions that aren't relevant in the application lifecycle after the initial XML loading is done.

So now I'm thinking of making some GraphFactory which has an Init function which reads files through XMLClass1 and XMLClass2, does all the setting up, and then returns a new Graph where everything about the graph is just passed through the constructor, and it's nice a clean to start defining getters/setters/methods.

What is the best way to go about this? Keep in mind I'd like to make this as dependency-injection-friendly as possible.