Slow DDR3 RAM down

[robfinch]

It has been challenging getting the DDR3 RAM  on the NexysVideo board to work reliably. For a while it appeared that the RAM might be bad. I tried running the traffic generator example and it reported no errors. However, my own circuit apparently causes issues. I spent a couple of days trying to identify what was creating an issue. The interface to the MIG component appears to be straight-forward. Writing to the RAM only updated the first byte in a burst. Then I tried lowering the clock rate to the DDR RAM and now it almost works. It is still not 100% but much better. Bytes other than the first one can be updated. I was wondering if there was an alternate slower part that could be selected in the MIG generator? I would like to experiment with lower clock rate. The minimum clock rate for the part still appears to be too high. The clock rate only goes down to 600MHz and I would like to try 533 to see if things work. My gut tells me the RAM has gone bad.

[asmi]

1 hour ago, robfinch said:

Almost but not quite working, just by lowering the DDR3 clock frequency. At a higher frequency nothing gets written. So I would like to be able to try a lower frequency yet.

This is absolutely wrong way to do it. Leave DDR3 at nominal frequency, and use AXI Interconnect/crossbar or a FIFO to move the data from your application clock domain to the memory clock domain. Also check that you've properly constrained your design - there might be undiscovered timing issues. Whatever the issue is, it's definitely got nothing to do with DDR3 controller, so look for issues in your own code.

[asmi]

Nexys Video houses Artix FPGA and so can only drive DDR3 up to 400 MHz (for DDR3) or 333 MHz (for DDR3L). I would love to know how have you managed to run it at 600 MHz.

[robfinch]

I was thinking of the DDR3 rate. The clock I was using was only 300MHz (which is used for double data rate transfers). But that is about the minimum accepted. I am too lazy to look up all the specs and try and define a custom slower part.

[zygot]

Have you enough energy to look at this: https://forum.digilentinc.com/topic/22197-a-guide-to-using-ddr-in-the-all-hdl-design-flow/

[robfinch]

Yes, thanks for the reference.

Almost but not quite working, just by lowering the DDR3 clock frequency. At a higher frequency nothing gets written. So I would like to be able to try a lower frequency yet.

$$F 10000 20000 333
$$D 10000 20000
:010000 333 333 010 014 018 01C 020 333  )*+3-3/3
:010008 333 333 333 014 018 01C 020 333  )*3333/3
:010010 333 00C 333 333 333 01C 020 024  33+3-3/0
:010018 333 00C 010 014 018 333 020 333  )*3,-3/3
:010020 333 00C 010 333 018 333 333 333  )*3,-3/3
:010028 333 333 333 014 333 333 020 024  )*3,3./3
:010030 333 00C 010 333 018 01C 020 333  333,-3/3
:010038 333 333 010 333 333 01C 020 024  33+,3330
:010040 333 333 010 014 333 333 333 024  33+,3.30
:010048 333 00C 010 333 018 01C 333 333  )*33-3/3
:010050 333 00C 333 333 018 01C 333 333  )333-3/3
:010058 333 333 010 333 333 333 020 333  3333-3/3
:010060 333 333 010 014 333 333 333 024  33+,3.30
:010068 333 00C 010 333 333 01C 020 333  )*+3-.33

 

[robfinch]

I am not saying there is anything wrong with the DDR controller. I only posted because I already spent two or three days looking for issues in my own code and cannot find any. It is modelled after the DDR controller code provided by Digilent. I have created a multi-port memory controller which allows multiple devices to access the RAM, each has its own read cache. I am sure there is a bug somewhere, but for now, I would just like to be able to try a lower frequency, in part if it works, I can postpone looking for the issue with my code and work on other parts of the project. I am using a WISHBONE interconnect in my project. AXI is a great way to go, but it has a little bit more overhead to it.

[asmi]

1 hour ago, robfinch said:

I can postpone looking for the issue with my code

Lol What makes you think that this issue won't affect some other parts of your project? Did you run simulations to see what's going on - like does your data actually reach memory controller intact, and all byte enables are set properly? The latter is very common mistake, I myself screwed it up more times than I'm willing to admit, even though I know of this trap all too well.

Lower frequencies won't help you because whatever issue you've got is going to manifest itself anyways. If anything, it's good that you have a clear way to reproduce that issue. Now run some sims to see where exactly your data is lost on it's way to the memory controller (or on the way back from it - the issue just as well might be on a reading side of things).

 

1 hour ago, robfinch said:

AXI is a great way to go, but it has a little bit more overhead to it.

AXI can't possibly have any overhead - it's just a bus, which is simply a bunch of wires.

Edited January 20, 2022 by asmi

[zygot]

4 hours ago, robfinch said:

I have created a multi-port memory controller which allows multiple devices to access the RAM, each has its own read cache. I am sure there is a bug somewhere, but for now, I would just like to be able to try a lower frequency, in part if it works, I can postpone looking for the issue with my code and work on other parts of the project. I am using a WISHBONE interconnect in my project.

He more you post about this project the more I'm thinking that I'm reading about the tribulations of someone trying to create a complicated design with absolutely no formal structured methodology to the process.. just code and see what happens if the tools are willing to provide a bitstream. The first thing that got blamed was perhaps the hardware, then the DDR PHY clock. 

My advise is to step away form this project for a while until you are ready to deal with it realistically. Work on something else for a couple of days and then come back and read what you've posted here.

This complicated project, as you describe it requires at least thinking about a formal approach to the design and verification, if not actually writing down a list of objectives for constructing a plan of attack. You say that you're boot-strapping a known good project. Nothing wrong with that, so implement it and verify that the hardware works as expected. Once you have a single channel DDR controller working it might be time to start planning a multi-channel controller version.. that is if the 'simple' version meets timing. How is your multi-channel controller going to work? Round robin service? You can certainly write HDL and testbenches to exercise this part of your design independently of actual DDR activity. Start simple and build up to complex. Verify low level functionality in your design hierarchy before throwing it all together and hoping for the best with no plan on how you are going to do debugging.  Xilinx seems to have abandoned hard DDR controllers as well as soft multi-channel DDR controllers and pushed this into the AXI IP. You don't have to use AXI. I certainly wouldn't unless my design was connected to ZYNQ interconnect. But if you do use Xilinx AXI IP then you will have to learn how to use it. Think about WISHBONE once a basic multi-channel controller is working. If you are using WISHBONE only because you have third party design elements that you want to use I'd suggest ditching unnecessary bus structures.

Simulating Series 7 DDR3 is a pain but it has to be done. You can save yourself some aggravation by partitioning your design simulation. When you have done enough simulation verification and timing closure to attempt implementing your design i hardware start with getting single burst read and write commands working before attempting to maximize data rates.

Most importantly, spending time planning out a complex design isn't a waste of time. It's a necessity and will save time as well as the embarrassment of posting comments [most of your thoughts in this post].

Consider this thought: if you don't provide the tools with reasonable timing constraints if is quite possible unrealistically good timing reports that can make you think that your design could work on hardware when in fact it can't. The tools are only as good as the input you give them. What you are describing is not a trivial design. 

Edited January 21, 2022 by zygot

[robfinch]

I built and ran the DDR Test project and captured some output from the serial port.

Found two issues with my code. 1) Multiple read commands were being sent back-to-back. It looked to me looking at the user guide 586 that this was needed to be able to read multiple data packets back. I now understand that the controller simply reads more-and-more data once a single read command is issued. 2) The address was being incremented for each read command submitted. Fixing these two issues did not give better results. There must still be another issue somewhere.

I got it working! I switched from dumping the read data into a reg to dumping it into a block RAM cache. It seems to work perfectly now.

Now I am wondering about the power consumption of the controller if just keeps reading data even if the data is not needed. How does the read get shut off?

What happens if the write fifo is full? Does a write command need to be issued to empty it?

[zygot]

9 minutes ago, robfinch said:

I built and ran the DDR Test project and captured some output from the serial port.

Is this a reference to the tutorial that I posted? If you have questions about that please post to that location. I'm hoping that you actually read the material provided rather than just re-created the Nexys Video projects.

11 minutes ago, robfinch said:

I got it working!

Not sure what that means in the context of the ensuing questions that follow. 

Trying to use DDR for single byte or word random read or write operations is terribly inefficient; especially for reads that have a significant latency per command. DDR and more advanced memories are designed to fill cached buffers for processors running at GHz speeds; a few unneeded bytes here or there is a small price for overall throughput. The DDR controller PHY shouldn't be executing commands continuously when your design doesn't have any need to read or write data.

[robfinch]

 

Quote

The DDR controller PHY shouldn't be executing commands continuously when your design doesn't have any need to read or write data.

I would agree with that. That is why I was wondering if there was a shutoff command? Suppose a burst of 4 packets of 128 bits is to be read. How does the controller get shutdown after reading the 4 packets? There is a READ command (001) and WRITE  command (000), are there others? As far as I can tell, it will just keep reading, likely until it fills its fifo at least.

Edited January 21, 2022 by robfinch
accidently pasted code