C# Design Pattern for updating child private fields

I have an unmanaged block of memory controlled by a Heap class. The class is responsible for tracking and moving elements, resizing the allocated memory, and disposing of the memory.

public class Heap
{
  private IntPtr data_; // Pointer to the unmanaged data
  private int capacity_; 
  private int size_;
  private LinkedList<Reference> elements_;

  public Reference<T> Add<T>() where T : unmanaged 
  { /* Creates new reference and returns it */ }

  public bool Remove<T>(Reference<T> element ) where T : unmanaged 
  { /* Removes reference, moves all subsequent elements and updates their pointers */ }

  private void Resize( int capacity )
  { /* Allocates new unmanaged memory, copies data, frees old memory, updates all element pointers */ }
}

Each element in the Heap is denoted by a Reference<T> object, which contains type info and a pointer to the element in the heap. The pointer is stored in a private field, but the value of the element is accessible by a property that uses the private pointer to get/set the value.

public abstract class Reference
{
  protected void* pointer_;
  protected int sizeInBytes_;
  protected Type type_;
}

public class Reference<T> : Reference
  where T : unmanaged
{

  public T Value
  {
    get { return *(T*) pointer_; }
    set { *(T*) pointer_ = value; }
  }

}

Since the Heap can resize and remove elements, I need a way to update the pointer_ field of each affected reference. If the memory is resized, all of the elements will need to be updated, as new memory is allocated, the contents copied over, and the old memory freed. If just one or a few objects are moved, then only those need to be updated.

I want to be able to update the pointers, but I don't want to expose the ability to do so to anywhere outside of the parent object, which in this case is Heap. I do intend to add other unmanaged data structures that can re-use the Reference<T> class to do the same.

How can I make it so that the parent class (in this case Heap) is the only class able to update Reference<T>.pointer_?