Factory Pattern, open closed principle, interfaces and generics

I've tried to refactor some code based on the Open-Closed Principle, but I just don't seem to be able to get the following classes right when it comes to applying design patterns. (I apologize for the many classes outlined below - I have reduced them as much as possible, but the rest is needed to show you my design).

The set-up consists of the following classes:

public interface IPortFactory
{
    IPort CreatePort(int id, PortDetails details);
}

public class PtpPortFactory : IPortFactory
{
    public IPort CreatePort(int id, PortDetails details)
    {
        var ptpPortDetails = details as PtpPortDetails;
        if (ptpPortDetails == null)
        {
            throw new ArgumentException("Port details does not match ptp ports", "details");
        }

        return new PtpPort(id, FiberCapability.FromValue(ptpPortDetails.Capability));
    }
}

public interface IPort
{
    int Id { get; }
}

public interface IInternetPort : IPort
{
    bool OfCapability(FiberCapability capability);
}

public class PtpPort : IInternetPort
{
    private readonly FiberCapability _capability;

    public PtpPort(int id, FiberCapability capability)
    {
        _capability = capability;
        Id = id;
    }

    public int Id { get; private set; }

    public bool OfCapability(FiberCapability capability)
    {
        return capability.Equals(_capability);
    }
}

Besides PtpPort, I have PonPort, which also implements IInternetPort, whereas CatvPort just implements IPort.

Already in this code, I think there's sign of code smell. In CreatePort in PtpPortFactory, the pretty thing would be for it to accept PtpPortDetails (which inherits from PortDetails) instead of having to cast. However, if I do so, I cannot create a PonPortFactory, which also implements IPortFactory, as these ports would need PonPortDetails. Or a CatvPortFactory for that matter.

Another code smell appears when I use the port factory:

PortType portType = command.PortType;
IPortFactory portFactory = portType.GetPortFactory();
var portsToSelectFrom = ports.Select(port => (IInternetPort) portFactory.CreatePort(port.Id, port.PortDetails)).ToList();

I would really like not having to do the downcast from IPort to IInternetPort, but simply have CreatePort return IInternetPort.

The last bit of information needed to understand the above is probably the following class (based on Jimmy Bogards Enumeration class):

public abstract class PortType : Enumeration<PortType, int>
{
    public static readonly PortType Ptp = new PtpPortType();
    public static readonly PortType Pon = new PonPortType();
    public static readonly PortType Catv = new CatvPortType();

    protected PortType(int value, string description)
        : base(value, description) { }

    public abstract IPortFactory GetPortFactory();

    private class CatvPortType : PortType
    {
        public CatvPortType() : base(2, "catv") { }

        public override IPortFactory GetPortFactory()
        {
            return new CatvPortFactory();
        }
    }

    private class PonPortType : PortType
    {
        public PonPortType() : base(1, "pon") { }

        public override IPortFactory GetPortFactory()
        {
            throw new NotImplementedException("Pon ports are not supported");
        }
    }

    private class PtpPortType : PortType
    {
        public PtpPortType() : base(0, "ptp") { }

        public override IPortFactory GetPortFactory()
        {
            return new PtpPortFactory();
        }
    }
}

I really hope that someone can help me along the way (I've tried introducing generics, but seem to always hit the barrier of C# not supporting return type covariance).

Moreover, any other tips 'n tricks to help me along my path to write better code would be greatly appreciated.