Jump to content

Thoughts on Simulation


Recommended Posts

Even some of us who think of ourselves as fairly competent FPGA developers can get confused as to what's really taking place when we write our source HDL, create a bitstream, and simulate ( when we need to debug the hardware ) what we think ( the 'what we think' as opposed to 'what is' being the crux of the matter ) our FPGA and connected IO are doing. The case for making simulation a core part of the design process has been made elsewhere so this thread will deal with the large gap between reality and our limited ability to model reality effectively.

Simulation is not an Oracle that magically understands all physical behaviors and identifies and reveals problems or deficiencies in our HDL code for solving every aspect of a particular interface. It's easy, while focused on unwanted behaviors in our hardware, to forget about all of the variables involved in generating a configuration file. For now, let's forget about errors in timing, errors in RTL design and errors in grasping what might be happening in the physical realm of our logic and IO interfaces.

Depending on the setting of our tools ( synthesis, placer and route, etc ) what gets expressed in the configuration of the FPGA might not resemble our source code. Hopefully it provides the desired behavior for every condition that we need it to. Having this is mind during simulation is foundational.

Fortunately, FPGA vendors offer support for simulators in helping close the gap between what we think our source code is trying to express and what the FPGA has been configured to do given the state of various inputs. This is where timing simulation comes into play. You can tell ISE or Vivado to create an SDF netlist of the optimized, synthesized, placed and routed logic as it will exist in the FPGA. As your designs get more complex timing simulation gets to be more of a requirement.

For a long time ModelSim was the only real simulator option around. Of course there's never been one version of ModelSim for FPGA development. Each vendor worked with ModelSim to allow simulating the details of it's device architectures. So, you might have found the 'free' version of ModelSim provided by Xilinx or Altera to be adequate for basic RTL simulation. But if you produced FPGA based products then you soon learned that you had to spring for the very expensive paid  'full' version of Altera ModelSim or Xilinx ModelSim. You could also buy ModelSim from Mentor for a particular FPGA device family. Altera and a few other FPGA vendors still rely on ModelSim to provide the basic simulator framework. Xilinx, for a long time now has gone its own way with ISIM or XSIM. I know one guy who swears by free third party simulation tools as a better choice for all of those very expensive simulator company or vendor simulator options. I've been using, off and on the very expensive simulators and 'free' vendor simulators for over 3 decades now and I'll need some compelling proof that open source or free tools are an adequate replacement for them. I've worked for a lot of companies making FPGA based products and none of them rely on such tools. In fact I;ve never come across any that use them as optional development aids.

Simulation is such a complicated topic that I do hope that this thread generates some interesting observations, thoughts, and opinions from a wide range of practitioners. So far, I haven't even alluded to most of the interesting aspects of FPGA simulation.

VHDL and Verilog are languages designed to do much more that logic simulation but let's keep this thread to that sphere.

Link to comment
Share on other sites

@zygot,

I must be missing something.  You've described a post-place-and-route simulation, but you haven't quite said why it was required.  Shouldn't this simulation provide identical results to the pre-place-and-route simulation if the design meets timing?

Can you share an example from your own experience of a time when a design met timing, but not post-place-and-route simulation?

That's question #1.  Question #2: Running a full simulation of any design can be quite costly.  For a CPU this might mean starting the CPU, going through the bootloader, running whatever program that follows.  For a video system, it might mean working through several frames of video.  When you use this post-place-and-route simulation, do you go through this much effort in simulation, or can you cut corners anywhere?

Still skeptical,

Dan

Link to comment
Share on other sites

So what do you get when two skeptics have a discussion? Hopefully, two better informed skeptics.

1 hour ago, D@n said:

Shouldn't this simulation provide identical results to the pre-place-and-route simulation if the design meets timing?

Nope. As I mentioned, what gets put into the LUTs and CLBs may not resemble your input HDL source. A lot of this depends on your settings for the synthesis and place and route tools. Most of the time I start off with the default balanced timing settings. If you look around at what kinds of things you can change you will get a sense of how different the post-route design might be from the RTL behavioral design. There's all kinds of things like preserving the heirarchy or not, register duplication or not, etc. Then there are other considerations like whether or not the synthesis tool decided to infer latches. I guarantee that inferred latches can alter the behavior of a design, and not consistently over many builds. You might think that finding these is easy because there will be a message. Well, maybe. But if you are using third party netlist components, encrypted or not, you will never find them without doing a post-route timing simulation. I know this from experience. Don't forget that timing reports are based on post-route analysis. If that design, as elaborated by all of the tools processes, differs from what you intended in your source then the timing score won't be very useful. As I've said, the timing simulation helps identify these differences.Also, once you've been at this for a while you learn that just because you provide really good timing constraints ( usually for IO not internal delays ) that doesn't mean that the tools will give you what you are asking for. The tool always get to completion and says Done! But, for difficult designs I've often looked at the timing report to see a lot of failing paths, even for the constrained ones. If you are doing team design and don't have access to hardware then timing simulation is all you have. Your contribution might operate at a particular clock by itself but when integrated into a larger design bring down the whole design. This doesn't just apply to team projects... the same thing can happen when re-using previous code in a very high performance or dense design using the same part. Port your code to another device and there are all kinds of potential obstacles standing in your way.

Currently, I'm working on a knotty Ethernet project. Simulation ( many evolutions of it ) work great. Timing score is always a perfect 0. Behavior is, well let's say 'arbitrary' and not consistent. Plus each build has a slightly different behavior somewhere in the design. This happens with a minor change to unrelated logic or not change in source code at all.This situation is not uncommon for designs with complicated logic operating at high clock rates. Most of the time poor or inadequate timing constraints are the first remedy to such problems ( figuring out how to constrain a design might require a bit of great simulation to understand what yo9u need to do). The difference between RTL behavioral simulation and timing simulation is that timing simulation accounts for the delays inside your device ( post-route). RTL simulation pretty much deals with ideal signal paths. This can change from build to build with or without any source code changes. You can, of course, add some crude delay elements to an RTL simulation on the IO to a testbench but you can't ever account for logic placement and internal delays at a useful level without running a timing simulation

I've learned not to get too over-joyed with perfect timing scores in my final reports. I've learned not to get too complacent with what appears to be 'working' designs. I've learned that FPGA development results are a lot more dynamic than I wished they were.

[edited] I forgot to mention that, depending on your settings, the synthesis tool will extract lines of your code and implement them as a known structure; like counter, state machine etc. That doesn't mean that it is understanding exactly what it is that you want to do....

1 hour ago, D@n said:

When you use this post-place-and-route simulation, do you go through this much effort in simulation, or can you cut corners anywhere?

The problem with timing simulation is that the timescale has to be in picoseconds and even if your RTL testbench and timing testbench are looking for the same faults the timing simulation will take a LOT longer. It should take longer because now it has to check the relationship of signal transitions to a clock edge. Of course the problem with any simulation is that if you are looking for the wrong things you are unlikely to have a simulation identify issues.

We'd all like simulation to be a 10 second affair that's peripheral to the design process. If you are using the simulator to find syntax errors then mission accomplished. ( I mentioned that I've had ISIM refuse to compile a simulation with code that the implementation tools were happy with enough to produce a bitstream... ). If you want to find those corner situation problems then you will find that, except for the simplest designs, a decent testbench will take a while.

The simulator in not the only tool that you should have in your toolbox. It is just a dumb tool like a hammer. It won't toe the nail into the frame by itself. The burden of understanding what issues to expect and simulate for are on the testbench writer. And there are a lot of places where a design might be deficient.

Link to comment
Share on other sites

@zygot,

I've only ever seen a couple of bugs where one placed solution would work and another one that passes the same timing requirements would not.  One bug was fixed by registering the outputs of the FPGA in the I/O elements.    The same can be applied to the inputs: register them as soon as they enter the chip, in the I/O element.  I've also been counseled to output clocks via ODDRs or OSERDES's.  Together, these approaches have kept roblems from cropping up in a PNR dependent fashion.  Are you unable to use these solutions?  Or do these approaches not apply for your designs?

Dan

Link to comment
Share on other sites

18 minutes ago, D@n said:

Together, these approaches have kept roblems from cropping up in a PNR dependent fashion

I appreciate your point. This raises the little issue of the 'elephant' in the room. Is our development process focused on fixing problems or gaining insight and a deeper understanding of how real world devices and interfaces work? There are, sometimes, certain easy to apply 'fixes' that seem to resolve a host of deficient design flaws. No doubt that this is the quickest path to a 'working' design. Then there is having the insight and knowledge to design for overcoming specific issues when and where it is required. Developing this knowledge and 'gut instincts' about designing FPGA systems takes quite a but longer, asking more questions when things appear to be 'working', spending the time to characterize particular scenarios, and of course simulating, simulating and simulating ( hopefully with better testbenches for each iteration. The things that you mention are IO timing related. A design can meet IO timing and still fail in internal portions of a complicated design.

No design is a good design if it doesn't address all of the issues that the real world hardware will face or all of the potential conditions that the logic will be exposed to. Gaining the insight to account for all of this in a design is an evolutionary, and usually time-consuming process.

As to my personal experience. I take the short route when I have to meet a deadline. I take the longer route when I want to learn something.

Give me a 'working' design and I can find a situation where it doesn't work as the designer intended.

Link to comment
Share on other sites

@zygot,

I'm still listening to your preaching, but not getting it.  I just don't believe in Voodoo logic design.

As background, I've helped the SymbioticEDA team build a PNR algorithm for iCE40 FPGA's, and I'm also peripherally aware of Google sponsoring similar work for the Xilinx chips as well.  (Both are part of the nextpnr project)  This was how I knew to comment about registered I/O's.  There are also metastability problems--problems that simulation won't necessarily reveal, post-PNR or not.  (Well, you might get lucky ...)  I trust you've been around long enough to avoid these.

My point is, having looked into the internals of the various FPGA's, and having written and debugged PNR algorithms, I'm still looking for an example of a design that passes a logic simulation, a timing check, and yet fails a post-PNR simulation.  My interest is to know if there's something that needs to be done within PNR to keep this from happening.

Do you have such an example that you can share?  (Other than latches--you have shared about those, but we already know that latches are bad.)  Even better if so, can you explain the underlying phenomenology that caused the problem?

Dan

Link to comment
Share on other sites

4 hours ago, D@n said:

I trust you've been around long enough to avoid these

There's no avoiding meta-stability issues, just minimizing the effects when they occur and occasionally finding a different way to implement you logic. It's a complex topic. You will find a number of 'solutions'. Pick your poison.

4 hours ago, D@n said:

I just don't believe in Voodoo logic design.

Neither do I. It would be nice if one could find a fool-proof way to create or implement perfect designs. FPGA devices aren't perfect. FPGA tools aren't perfect. OUr understanding of the problem is not perfect. I think that it's possible to become competent enough in say, C programming, to be able to adapt to almost any environment and be useful. It's not possible to be so good at it that you can develop fault-proof designs for real world hardware in real world environments. If you do all PC work you will have some challenges moving on to embedded work. FPGA development is a lot more complex than developing software for a particular CPU.

4 hours ago, D@n said:

I'm still looking for an example of a design that passes a logic simulation, a timing check, and yet fails a post-PNR simulation.

[edited] No problem, just keep doing FPGA development until someone has a very difficult project with specific validation requirements. Th's that short reply....

I've seen this on many occasions in many workplaces. The project that I'm working on now fits the bill. I'm not a believer in testimonials. You just never get all of the information that you need to make a good assessment. I've worked with brilliant people who just plain got an analysis wrong so I don't tend to follow testimonials blindly. By the way, to turn your argument around a bit; you're saying that you've done 100's of timing simulations and know  beyond a reasonable doubt that they are worthless? I know of quite a few companies who'd be shocked to hear this.

Every once in a while I run into someone who claims that everyone else, all the textbooks, all the professors have gotten something wrong and that they have the answer to a problem that no one else has found. This has happened to me at least 5 times as I can recall at the moment. One was a Phd with a small company selling power meters to utility companies. I can't think of a one of them who were able to prove themselves to be correct ( you might tend to think that if a utility is buying product that's proof enough but I think that if a utility sees a way to make more money they might not be too curious about how it's made ). Rarely, someone does come along and re-write the textbooks and alter a curriculum; but not that often. I am only conveying information that is a best practice in industry in my experience. There's usually, but not always, a good reason why most experienced places do something a certain way. Most companies are not into wasting time and money in development.

I really have no need nor interest in convincing anyone that the have to incorporate timing analysis into their skill set. ( and it is a skill set as using a tool is the only way to become adept at using it ) When people are ready to understand that they need to use a particular tool then all I need to do is inform them of its existence. [edited] In general, having to develop a new skill while under a lot of pressure to meet a deadline is not an ideal way to work. Better to have the honed skill in the toolbox ready for when it's needed.

My reaction to a lot of what you've been saying is that you haven't been exposed to very complicated designs, you haven't been exposed to rigorous validation requirements, or you aren't quite getting what's involved in the whole process. iCE40 FPGA's are one small niche in the FPGA world and represent small devices with limited capabilities ( that doesn't make them easier to develop for ). Again, you seem to think that IOs are a design. IOBs in Xilinx devices have features enhancing the part's usability for particular applications. I'm sure that you've created designs that don't fit into IOBs.

My experience with significantly large and or complex and or high speed designs is that it's very hard to nail down logic placement and routing over many design cycles and preserve performance and operational reliability. Once you arrive at such a situation timing simulation becomes more than an intellectual concept.

Link to comment
Share on other sites

2 hours ago, D@n said:

Did you see the recent presentation showing the Verilator can beat the big vendor simulators in speed by 5x or more?

Worked for me... it's a while ago now but I had my drivers working when silicon arrived and knew there was a saturation bug in the datapath that needed a gain staging workaround.

Success story for the marketing department, if there were any.

Link to comment
Share on other sites

5 hours ago, D@n said:

Did you see the recent presentation showing the Verilator can beat the big vendor simulators in speed by 5x or more?

No, but I have seen ads telling me that I can eat anything that I want, do no exercise, and take a totally harmless pill that will return my body and appeal to my 20's self ( OK, disclaimer, I was very thin but not all that appealing back then either... ). I can eat my steak dinner 5x faster by shoving the contents of the plate into a blender and chugging it right from the jar... might even be healthier.

I do get the concept that having tools that provide answers very quickly can improve productivity. I've submitted card decks to a mainframe and waited a day or two to get back the results. I see simulation as a bit more than a tool though. The verification process is about more than moving on to another design challenge. The whole design process is an intellectual exercise. A simulator is ( can be ) to the intellect like a piece of gym equipment is to the body. There's more to design than 'I came, I saw, I conquered'. Understanding the nuanced details is, I think, better than blasting through projects.

Still, not feeling it. I guess that the argument that "never tried it, don't know what is, don't know what it does, don't care, don't need it" just isn't that compelling to me.

Again, I'm not trying to challenge anyone's religious views. When you get to a spot where the you start thinking that the must be something else to try then perhaps timing simulation will come to mind. If you never get to that spot then don't worry about it. If the pinnacle of your FPGA career is making the hex led displays look like a clock you don't have to worry about timing constraints or much else... certainly not timing simulations.

 

Link to comment
Share on other sites

  • 10 months later...

It'a been too long since anyone posted to this thread so...

I've often mentioned that simulation is an art form. It's also educational. The reason why we simulate is not only to figure out what the chances that our HDL will perform properly is before trying it out on hardware, where bad implementations can actually damage your hardware, but also because simulation is the best instructor available if you want to get better at understanding the complexities of the behavior of the hardware that your code is trying to interact with, and hence better at anticipating what needs to be done for future similar projects. This means that simulation is often an iterative process; try cut one, find the deficiencies, try cut two, find the deficiencies, etc., etc.... This is particularly true when writing code to interact with hardware external to your FPGA or FPGA board. A naive perspective on simulation is that it's an extra step in the design process or that looking at a few signal states at the end of the simulation tells you if your code is pretty good or not.

As an example, let's say that you want to hook up an FT232H USB Multi-purpose interface end-point device. This is a good example because the datasheet and application notes don't offer all of the information that you might need in order to use the device in a particular mode. To further complicate matters you might need to use a driver API with your PC application that is similarly lacking in necessary information.

I'm posting this because every once in a while I'm in a hurry to get things done and choose to 'cut corners". I almost always end up doing what I suspected should have been done in the first place; just after wasting time trying to figure out why the hardware doesn't jive with the simulation. The truth is that sometimes Vivado doesn't simulate all of its FIFO and BRAM behavior accurately. Sometimes it warns you about those deficiencies and sometimes not. Vivado certainly doesn't offer help simulating external devices.

The 'proper' way to simulate your HDL interacting with a device like the FT232H is by first writing a model of how you understand the device works. This model then becomes part of your simulation as opposed to just trying to manipulate signals manually. Your understanding of what the model needs to simulate starts with information like timing information for the device pins like data bus and flags from the datasheet, and from supplemental information like application notes that, helpfully, fill in missing information. You might think that you can write a testbench without the model but until you are running applications, using the drive API, you don't realize that static timing information isn't always enough. So just write the model first, spending more time up front and eventually saving overall project time. Expect there to be revisions to your model as debugging progresses, and your understanding of the task at hand gets better and your simulation gets more representative of the actual hardware. Again the process of good simulation is also the process of learning how to simulate.

Just some friendly advice.

[edit] I forgot to mention that there are two basic ways to write your device model. One is to use the same, simulation only features of your HDL, just as you would write any testbench, and the other is to write your model as if it were being synthesized as part of your FPGA code. Which way you want to do this depends on what you are trying to accomplish. Since your HDL capable of being synthesized can run on an arbitrary clock you can actually do behavioral or timing simulation sufficiently. Just don't try and synthesize it.

Link to comment
Share on other sites

13 minutes ago, zygot said:

...

The 'proper' way to simulate your HDL interacting with a device like the FT232H is by writing a model of how you understand the device works. Your understanding is based on timing information for the device pins like data bus and flags, and from supplemental information like application notes that, helpfully, fill in missing information. You might think that you can write a testbench without the model but until you are running applications, using the drive API, you don't realize that static timing information isn't always enough. So just write the model first, spending more time up front and eventually saving overall project time. Expect there to be revisions to your model as debugging progresses, and your understanding of the task at hand gets better and your simulation gets more representative of the actual hardware.

Just some friendly advice.

Ditto.  Sometimes I  feel like I spend all my time building simulation models, but they are important.

One of the things I like about building simulation models using Verilator is that you can also simulate the interaction with the outside world as well.  Because Verilator uses C++, you can get access to every O/S feature you have.  You can make VGA or even HDMI simulators that draw the "screen" on a window of your development work station, you can make UART simulators that you can interact with over TCP/IP, you can make SD Card simulators for 4GB SD cards where every byte is accounted for and actually works, you can make ethernet simulators that actually send and receive packets, and more.

The disadvantage is that the simulation is all clock based.  You lose the fractional clock resolution.  For me, this has been good enough.

Recent language developments have created standard interfaces to interact with your local O/S environment as well, I just don't have experience with them (yet).

IMHO, simulation models like this let you go from component testing to integrated system testing.  They're quite valuable--as you've pointed out.

Dan

Link to comment
Share on other sites

As you were posting your reply I added a few lines to mine that were forgotten.

Yes, sometimes it would be nice to be able to simulate a complete system including your code, your board, external devices and boards, and software. This is why VHDL was created. In theory you can simulate a jet plane including all of the electrical, mechanical and software components as they might perform during an environmental scenario. People have tried.

There are a lot of options for getting as close to simulating what you need to simulate, in order to shorten hardware testing and verification, as accurately as needed. All of them have limitations for some projects that need to be dealt with.

I have to admit that your approach of using a language like C++ (Ugh, but that's another topic...) to facilitate the whole design flow is irresistible. I find that my C finger brains get confused with my VHDL finger brains when doing a project involving hardware and OS software.

Link to comment
Share on other sites

3 hours ago, zygot said:

I find that my C finger brains get confused with my VHDL finger brains when doing a project involving hardware and OS software.

This was me the other day:

badcast.png.29fef1d854a5c9c9cce14f9085992e5d.png

Is there anything more that needs to be said here?  :D

Dan

Link to comment
Share on other sites

4 hours ago, D@n said:

Verilator ... The disadvantage is that the simulation is all clock based.  You lose the fractional clock resolution.  For me, this has been good enough.

enter VHDL-AMS and spend coffee breaks reminiscing about the days when simulation "was" all clock based... Adaptive time step control... works for differential equations so it'll sure work for digital systems, too... strictly monotonic time was yesterday... umm, which 'yesterday', the one we just had or the one that hasn't happened yet...

Oh well, I digress...

Link to comment
Share on other sites

Vivado and similar logic simulators like ModelSim can have an arbitrary step period. Want 1 picosecond  step time? They will do that. Want to simulate a synthesized and routed design in a system. They do that too. There are a number of tools in the simulation toolbox. Why would you want to drive nails with a really large pipe wrench when there's a hammer is available?

Cycle accurate simulation is fine; just not the only tool available. Behavioral simulation of HDL source code is akin to evaluating wellness from a picture. Timing simulation working with placed and routed elements with known delays is a better way to see what's going inside a design near ready for hardware. Simulator interpretations of HDL don't always agree with how the synthesis tool understands your code. In the end it's good to know how to use them all and understand what their limitations are to be effective in the simulation verification portion of your design.

Link to comment
Share on other sites

Archived

This topic is now archived and is closed to further replies.

×
×
  • Create New...