When using a Builder Pattern why shouldn't I reuse the builder-object to access the object configuration?

when using a Builder Pattern why shouldn't I reuse the builder-object to access the object configuration? For example:

Normal way:

ObjectA(ObjectBuilder b) {
    this.a = b.getA();
}
public Object getA(){
    return this.a;
}

But why can't I just use this:

ObjectA(ObjectBuilder b) {
    this.builder = b;
}
public Object getA(){
    return this.builder.getA();
}

Thanks :)