Why the Arty server data sending suddenly slow down?

[LSC]

Hi,

I am following this tutorial "https://reference.digilentinc.com/learn/programmable-logic/tutorials/arty-getting-started-with-microblaze-servers/start" and made some modification to the "echo.c" to send bytes of data to client every second. The server will repetitively send 240012 bytes every second. I experimented and found that there is a maximum allowable bytes to be sent in every "tcp_write" function called. So, I split the "tcp_write" function to be called multiple times (each time will send about 30000 bytes). So, it takes about 8-9 "tcp_write" function called every second. On my client side, I received all these 240012 bytes within 550ms. Initially, it is running normally and after a while, the server start sending all the 240012 bytes data very slowly. It takes >10 seconds to send all the 240012 bytes. It is inconsistent on when this happen. Sometimes, it can happen after 50 cycles and sometimes it can happen after 500 cycles.

 

I also tested this by storing the SDK to the SPI, but it still happens. I reconnect my client and the problem exist in the first cycle itself. I rerun the server and the problem disappear and only appear after sometime. So, it means the problem exists in Arty server. I dont find any memory leakage in my coding in "echo.c". The only settings I changed in "system.mss" are "tcp_snd_buffer" = 32000 and "tcp_mss" = 2000. My main question is:

1) How do I troubleshoot and solve this? Has anyone use the server this way without issue?

 

Other questions:

2) Can I send more bytes in 1 "tcp_write" function called? I noticed the limit it can send is around 31000 bytes after playing with the  "system.mss" settings, "tcp_snd_buffer" = 32000 and "tcp_mss" = 2000.

3) I noticed the client side receive the data in batches of 1460 bytes. So it means, the client may receive all the 240012 bytes in 240012/1460 = 165 times. How do I enable the server to send more than 1460 bytes each batch? FYI, I suspect the 1460 bytes is limited by the server and not the client because I used the same client for other application and I never encounter that it was limited to 1460 bytes.

[D@n]

I'm not sure I can answer your entire question, but I can answer a piece of it: the ethernet standard has a maximum packet size of near 1500 bytes.  If you figure that the TCP header size is 20 bytes, the IP header size is 20 bytes, and the ethernet header size is 14 bytes (not including the preamble) and 4 bytes of CRC, 1460 sounds about right.  While there are extensions to the ethernet protocol that allow larger packets to be sent, that's at least my best guess as to why the 1460 byte/batch limit.

As for problem #1, have you considered a packet collision being the problem?  The standard/traditional approach to collision detection is to slow down the link.  Your TCP link has several packets flying across it, any of which could result in a collision: TCP data from the server to the client, TCP acknowledgements from the client to the server, address resolution protocol requests and responses, as well as data link speed negotiation messages.  This would explain why restarting the server fixes the problem--the reloaded server has no information about collisions on the network that it needs to avoid.  You might be able to turn off the data link speed negotiation messages via the MDIO port of the network PHY on the Arty.  You might also be able to adjust how long an address persists in an ARP table.  Together these might help you debug the problem.

Another reason why the network might suddenly slow down would have to do with the data link speed negotiation.  Should the PHY renegotiate speeds, it could drop your rate from 100Mbps down to 10Mbps.  Again, the MDIO port on the Arty PHY gives you access to querying the PHY and asking what it is doing--whether auto-negotiation is turned on or not, and what speed it thinks it is running at.  (I also seem to recall that the LEDs on the ethernet connector may reflect the negotiated speed as well ...)

Hope this helps,

Dan

[LSC]

Thanks. I am newbie here. So a lot of things are still unclear to me. I actually dont know how to go about with your suggestion above because a lot of technical thing that I yet to learn. I thought the TCP protocol should handle this collision? How am I going to handle it in the final c programming level? When this happen, I stop the client from sending message. So, I suppose the only possible collision is the acknowledgement from client which I dont have any control of, but do I still need to handle this? I tested the same client program communicating with server on linux platform before (server is on beaglebone), without this issue.

[jpeyron]

Hi LCS,

If you are still having issues trouble shooting your server, here is a freeRTOS web server that might work for your situation. Here is an example of the Arty broadcasting udp packets from the Ethernet port. Hope this helps!

cheers,

Jon

[LSC]

thanks. the example is very interesting. we switched to microzed following the same tutorial for ethernet. it seems work for this. however, the client software is different, so it could be due to client or the arty.