Several died ethernet interfaces on Arty A7 boards

[TomF]

Hello,

we are using Arty A7 for prototyping our product which has ethernet connection. After some time, the ethernet connection is not possible anymore and it seems that the PHY is defective.

However, it is not clear what could be the cause for this problem. It seems to be independent of the temperature of the environment. The time until it fails is also very random, from weeks to months.

Is there a way to find the reason for this and identify the failing component? Are there similar reports concerning this problem?

 

[JColvin]

Hi @TomF,

As far as I know, you are the first person to bring up a repeating problem with the Ethernet or PHY.

Could you let us know a little bit more about your setup and general environment to help narrow down some possibilities?

Thanks,
JColvin

[ghelbig]

Perhaps a related issue:

I just received an Arty-A7-100, and am trying to run the Avnet LwIP example working.

First issue is that their software repository (UG-AES-A7MB-7A35T-G-Arty_lwIP_EthernetLite_VIV2015_v1.pdf - page 32) does not work.

• If I try to create a project as described (page 34) SDK aborts.
• If I try to build a BSP using LwIP from that repository, the build fails.

Second issue is that if I use the Xilinx LwIP the initialization routine hangs.

Something is weird, perhaps marginal, that is stopping the Ethernet port from working.  Or it could be that the IP needs to be configured specially for this board?

I'm using 2018.3 - per my client's specification.

Regards,

Gary.

 

 

 

Edited October 13, 2021 by ghelbig

[TomF]

There is nothing really special about our setup or environment. As the boards also die when the temperature is low, there must be something else that is not working correctly. One thing I can tell for sure: the ethernet does not have a general problem, as it is working before it fails.

The periphery we are using is only some I2C, SPI and the XADC as well as the PHY. The power is supplied by the 9V port of the Arty-A7. 

[ghelbig]

19 hours ago, ghelbig said:

Perhaps a related issue:

I just received an Arty-A7-100, and am trying to run the Avnet LwIP example working.

I'm using 2018.3 - per my client's specification.

Regards,

Gary.

An update:  I made it work using Vivado 2015.2.  It sometimes reports an 'invalid temac configuration', but it usually works.

There is something definitely wonky with later SDK (and perhaps IPI) versions.  I am not looking forward to the task of porting it to later version. 

TomF:  What versions of software are you using?  There is a comment in the Avnet documentation saying that they found an issue with provided software, and had to patch it to make it work on the Arty card.  (And BTW, what is your "low temperature" ?)

G.

[TomF]

I am using Vivado 2019. To be clear, the boards work for some time, weeks or even months but suddenly die in terms of ethernet connection.

[zygot]

There's no question that Xilinx ( and other vendors have changed their Ethernet related IP of for logic designs... and not in a way that users would prefer.

One likely issue is licensing of the Temac core. You can get a free, limited time, evaluation license for free, but once it expires you may not be able to build designs without getting a new license. I have no idea if Xiinx allow you to perpetually obtain free evaluation licenses for the same core. This is one reason to avoid free stuff.

It's not likely that your Ethernet PHY has gone bad... though the one on my Zedboard ( a PS connected interface ) hasn't been working for some time. There's no license for PS peripherals. I haven't had the time to resolve this issue but suspect the RJ45 connector. I don't have a suspicion as to why this would fail.

There are some obvious tests to perform here. The first one would be to use an old bitstream that was know to work. If the Ethernet interface sill doesn't work then perhaps the PHY or magnetics in the RJ45 connector is worth looking into.

In the old days, Altera used to have time limited evaluation for some of its IP. But that scheme actually had a timer buried in the logic that required a JTAG connection to operate and would stop after n minutes. Xilinx has never had such a thing to my knowledge. Regardless, it's hard to imagine that there would be logic that magically knows the date in order to terminate an evaluation session.. especially after reconfiguration and over many power cycles.

Another thing to check are the Vivado messages related to your Ethernet block.

 

Edited December 2, 2021 by zygot

[asmi]

On 12/2/2021 at 1:40 PM, zygot said:

One likely issue is licensing of the Temac core. You can get a free, limited time, evaluation license for free, but once it expires you may not be able to build designs without getting a new license. I have no idea if Xiinx allow you to perpetually obtain free evaluation licenses for the same core. This is one reason to avoid free stuff.

Yes they do - I've extended it many times over years.

But this is beside the point because Arty only has 10/100 MBit PHY and so you can use the EthernetLite IP which is free and permanent (no ticking bomb like in evaluation version of TEMAC).

Edited December 3, 2021 by asmi

[TomF]

The problem that I have is concerning devices where the bitstream is stored in the flash memory and nothing is reprogrammed. So the IP license does not apply here. 

[zygot]

Well, in that case you need to start looking at the hardware. Since your PHY only supports 10/100 Mbps you have a chance.

You have a few things to look at:
Broken cables
Broken magnetics in the RJ45 connector
Broken PHY
Broken FPGA

The 1 GbE Ethernet PHYs on Digilent boards always come up out of hardware reset in auto-negotiation mode. So, I can power up two boards, connect a cable between the RJ45 connectors and see that this had taken place via the LED indicators. I don't know how the PHY on your board is set up to work post power-on.

PHYs that I've used have automatic crossover capability, so you never need a crossover cable. Ethernet has the same problem that RS-232 has... uni-directional signalling connected to the same connector pins; Ethernet just addresses the solution differently.

I make use of the Ethernet PHYs on many FPGA platforms all the time. I use some really old boards and have only had an issue once, with the Zedboard Ethernet. The physical layer of Ethernet is pretty robust and I am not aware of any common issues with it failing. I've never had one fail on a PC.

An unlikely situation is that your FPGA is damaged, but still somewhat functional. Are all of the other pins on the IO bank that connect the PHY to the FPGA still functioning normally?

Personally, I wouldn't rely on FLASH configuration as a test of the bitstream, as that's just one more potential problem; I'd configure via JTAG using the old, known good, bitstream. Debugging is easier when you reduce the number of variables to the minimum and take advantage of known good vectors that are available.

Have you inspected the RJ45 connector for physical damage?

Edited December 6, 2021 by zygot

[elodg]

I second zygot's idea. The Arty A7 has auto-negotiation enabled in the PHY. An unprogrammed but powered board should establish link to any Ethernet switch or auto-crossover supporting computer and light up the link LED. That would confirm that the MDI interface is working. If it does, the next step is seeing whether the MDIO interface towards the FPGA is working and if the PHY registers can be read and written. If that works too, data traffic on the MII interface should be analyzed, perhaps with ILA.

[zygot]

In working on a recent project that I used the Zedboard for I resolved a mystery that has bothered me for a while.

The Ethernet port stopped working on the board years ago. I don't use it much so I've just not bothered to figure out what's wrong.

I was using the unused GEM via EMIO. The Ethernet example designs are notoriously obscure and make targeting either GEM difficult. Sometimes, this can be good. When I ran the DMA Interupt example for ETH0 ( the one connected to the board RJ45 connector ) , it worked without error. This tells me that the PHY and MIO pins are fully functional. This example SDK project, like many, assume PHY loopback.

So, it you are unsure how to test the RJ45 connector separately from the rest of the board, this is one way to do it. Run send and receive a frame with the PHY in loopback mode. Since the SDK GEM examples support MicroBlaze, and it's likely that your Ethernet design uses a MicroBlaze, you should be able to try this out. You could create an HDL design to program the PHY via MDI and do the same thing.

So, the RJ45 connector can fail, mechanically or electrically, as is the case for my elderly Zedboard.

Anyway, for some reason this thread popped into my brain...