Go properly implementing constructor pattern

Lets take this very small example, having a function that modify value inside the struct:

package learn

type Point struct {
    x int
    y int
}

func (p *Point) Set(x int, y int) {
    p.x = x
    p.y = y
}

this works properly, used like this for instance:

package main

import (
    "NewPattern/learn"
    "fmt"
)

func main() {
    p := learn.Point{}
    p.Set(5, 6)
    fmt.Print(p)
}

it outputs the expected value: {5,6}

Now let's say I don't want the user having a constructor, I can change the code by adding this function:

func NewPoint(x int, y int) Point {
    return Point{x, y}
}

then I can use in main like this:

func main() {
    p := learn.NewPoint(3, 8)
    fmt.Print(p)
    p.Set(5, 6)
    fmt.Print(p)
}

and it works as expected returning {3 8}{5 6}.

Well now we want to prevent creating point without calling the constructor - not really the case here, but can make sense for complex classes - so we avoid exporting Point and we create an interface instead, so I refactored the code like this: (this is not working!)

package learn

type point struct {
    x int
    y int
}

type Point interface {
    Set(x int, y int)
}

func (p *point) Set(x int, y int) {
    p.x = x
    p.y = y
}

func NewPoint(x int, y int) Point {
    return point{x, y} //error here
}

This says:

cannot use point literal (type point) as type Point in return argument:
    point does not implement Point (Set method has pointer receiver)

I can "fix" this by modifying the method in:

func NewPoint(x int, y int) point {
    return point{x, y}
}

but this just move the error in main, that is refactored like:

func main() {
    var p learn.Point
    p = learn.NewPoint(3, 8) //error here!
    fmt.Print(p)
    p.Set(5, 6)
    fmt.Print(p)
}

and the error is:

cannot use learn.NewPoint(3, 8) (type learn.point) as type learn.Point in assignment:
    learn.point does not implement learn.Point (Set method has pointer receiver)

by googling I managed to solve in this way:

func NewPoint(x int, y int) *point {
    return &point{x, y}
}

but as a result in the main we are obtaining: &{3 8}&{5 6} as a print, ad as well I don't get actually what is happening behind the scenes.

I guess this is somehow related by having things passed and maybe "returned" by value, is this the case? But I don't know how the first examples without interface worked without effort. Can please someone clarify these details that I think are essential to Go understanding.