W5500 connection problem

I have several versions of a board with the w5500 as eth. chip. and At 1284p as MCU.
With some switches (at least two different types/brands) the link won’t come up. Other switches and computers work fine. All tested (faulty and non faulty) switches are gigabit switches that support auto-mdi and auto negotiation.

In case of the non working switches I see the link led turn on for about a second, shortly after the link led turned off the act led flashes briefly. After about a second or two this led blinking repeats. Connecting the powered up device to another (known to be working switch) results in an immediate link.

Any idea what this means and what the problem could be??
I followed the Wiz55io schematic and w5500 ref. schematic.
PMODE 111 shows above behaviour.
PMODE 011 (100T, auto neg. off) → no led activity at all

No problems with the switches that work (no link problems / no instability etc)