FlyWeight Design pattern shouldn't look works

I have a small problem with FlyWeight Design Pattern implementation in Java 11.

I have example to drawing colored rectangles where Intrinsic property is color and Extrinsic properties are dimensions of rectangle.

So, that means I will a new instance only if I have different colored rectangle.

You can look whole my implementation here:

https://github.com/zrebec/DesignPatterns/tree/master/DesignPatterns/src/sk/zrebec/learn/java/designpatterns/flyweight

I have there the "classic" method when I creating a new rectangle every single time (commented constructor in MyRectangle method and 2 lines in Program constructor and uncommented FlyWeight method using RectFactory

I have created Factory

public class RectFactory {

    private static final HashMap<Color, MyRectangle> rectsByColor = new HashMap<Color, MyRectangle>();

    public static MyRectangle getRect(Color color) {

        MyRectangle rect = (MyRectangle)rectsByColor.get(color);

        if (rect == null) { 

            rect = new MyRectangle(color);
            rectsByColor.put(color, rect);
        }

        return rect;

    }

}

Then my constructor looks like this:

public MyRectangle(Color color) {
        this.color = color;
    }

    public void draw(Graphics g, int upperX, int upperY, int lowerX, int lowerY) {
        g.setColor(color);
        g.fillRect(upperX, upperY, lowerX, lowerY);
    }
}

and implementation in Program.java looks like this:

MyRectangle rect = RectFactory.getRect(getRandomColor());
rect.draw(g, getRandX(), getRandY(), getRandX(), getRandY());

But I have very similar result with both methods.

On my computer draw 4000000 rectangles takes around 7 seconds and with classic method (Without using Factory) too

FlyWeight method

First result:

Used memory in byes: 171 853 6 This took 7392 miliseconds

Second result:

Used memory in byes: 172 005 6 This took 7047 miliseconds

Classic method when rectangle instance is created every single time

First result:

Used memory in byes: 171 664 8 This took 7489 miliseconds

Second result:

Used memory in byes: 171 760 8 This took 7178 miliseconds

**

FlyWeight design pattern should be around 200 to 300% faster, shouldn't?

Please, where I have problem in implementation? Of course, you can download code and try that.

Thanks for any advice