Sign in to follow this  
zygot

Thoughts on Simulation

Recommended Posts

Even some of us who think of ourselves as fairly competent FPGA developers can get confused as to what's really taking place when we write our source HDL, create a bitstream, and simulate ( when we need to debug the hardware ) what we think ( the 'what we think' as opposed to 'what is' being the crux of the matter ) our FPGA and connected IO are doing. The case for making simulation a core part of the design process has been made elsewhere so this thread will deal with the large gap between reality and our limited ability to model reality effectively.

Simulation is not an Oracle that magically understands all physical behaviors and identifies and reveals problems or deficiencies in our HDL code for solving every aspect of a particular interface. It's easy, while focused on unwanted behaviors in our hardware, to forget about all of the variables involved in generating a configuration file. For now, let's forget about errors in timing, errors in RTL design and errors in grasping what might be happening in the physical realm of our logic and IO interfaces.

Depending on the setting of our tools ( synthesis, placer and route, etc ) what gets expressed in the configuration of the FPGA might not resemble our source code. Hopefully it provides the desired behavior for every condition that we need it to. Having this is mind during simulation is foundational.

Fortunately, FPGA vendors offer support for simulators in helping close the gap between what we think our source code is trying to express and what the FPGA has been configured to do given the state of various inputs. This is where timing simulation comes into play. You can tell ISE or Vivado to create an SDF netlist of the optimized, synthesized, placed and routed logic as it will exist in the FPGA. As your designs get more complex timing simulation gets to be more of a requirement.

For a long time ModelSim was the only real simulator option around. Of course there's never been one version of ModelSim for FPGA development. Each vendor worked with ModelSim to allow simulating the details of it's device architectures. So, you might have found the 'free' version of ModelSim provided by Xilinx or Altera to be adequate for basic RTL simulation. But if you produced FPGA based products then you soon learned that you had to spring for the very expensive paid  'full' version of Altera ModelSim or Xilinx ModelSim. You could also buy ModelSim from Mentor for a particular FPGA device family. Altera and a few other FPGA vendors still rely on ModelSim to provide the basic simulator framework. Xilinx, for a long time now has gone its own way with ISIM or XSIM. I know one guy who swears by free third party simulation tools as a better choice for all of those very expensive simulator company or vendor simulator options. I've been using, off and on the very expensive simulators and 'free' vendor simulators for over 3 decades now and I'll need some compelling proof that open source or free tools are an adequate replacement for them. I've worked for a lot of companies making FPGA based products and none of them rely on such tools. In fact I;ve never come across any that use them as optional development aids.

Simulation is such a complicated topic that I do hope that this thread generates some interesting observations, thoughts, and opinions from a wide range of practitioners. So far, I haven't even alluded to most of the interesting aspects of FPGA simulation.

VHDL and Verilog are languages designed to do much more that logic simulation but let's keep this thread to that sphere.

Edited by zygot

Share this post


Link to post
Share on other sites

@zygot,

I must be missing something.  You've described a post-place-and-route simulation, but you haven't quite said why it was required.  Shouldn't this simulation provide identical results to the pre-place-and-route simulation if the design meets timing?

Can you share an example from your own experience of a time when a design met timing, but not post-place-and-route simulation?

That's question #1.  Question #2: Running a full simulation of any design can be quite costly.  For a CPU this might mean starting the CPU, going through the bootloader, running whatever program that follows.  For a video system, it might mean working through several frames of video.  When you use this post-place-and-route simulation, do you go through this much effort in simulation, or can you cut corners anywhere?

Still skeptical,

Dan

Share this post


Link to post
Share on other sites

So what do you get when two skeptics have a discussion? Hopefully, two better informed skeptics.

1 hour ago, D@n said:

Shouldn't this simulation provide identical results to the pre-place-and-route simulation if the design meets timing?

Nope. As I mentioned, what gets put into the LUTs and CLBs may not resemble your input HDL source. A lot of this depends on your settings for the synthesis and place and route tools. Most of the time I start off with the default balanced timing settings. If you look around at what kinds of things you can change you will get a sense of how different the post-route design might be from the RTL behavioral design. There's all kinds of things like preserving the heirarchy or not, register duplication or not, etc. Then there are other considerations like whether or not the synthesis tool decided to infer latches. I guarantee that inferred latches can alter the behavior of a design, and not consistently over many builds. You might think that finding these is easy because there will be a message. Well, maybe. But if you are using third party netlist components, encrypted or not, you will never find them without doing a post-route timing simulation. I know this from experience. Don't forget that timing reports are based on post-route analysis. If that design, as elaborated by all of the tools processes, differs from what you intended in your source then the timing score won't be very useful. As I've said, the timing simulation helps identify these differences.Also, once you've been at this for a while you learn that just because you provide really good timing constraints ( usually for IO not internal delays ) that doesn't mean that the tools will give you what you are asking for. The tool always get to completion and says Done! But, for difficult designs I've often looked at the timing report to see a lot of failing paths, even for the constrained ones. If you are doing team design and don't have access to hardware then timing simulation is all you have. Your contribution might operate at a particular clock by itself but when integrated into a larger design bring down the whole design. This doesn't just apply to team projects... the same thing can happen when re-using previous code in a very high performance or dense design using the same part. Port your code to another device and there are all kinds of potential obstacles standing in your way.

Currently, I'm working on a knotty Ethernet project. Simulation ( many evolutions of it ) work great. Timing score is always a perfect 0. Behavior is, well let's say 'arbitrary' and not consistent. Plus each build has a slightly different behavior somewhere in the design. This happens with a minor change to unrelated logic or not change in source code at all.This situation is not uncommon for designs with complicated logic operating at high clock rates. Most of the time poor or inadequate timing constraints are the first remedy to such problems ( figuring out how to constrain a design might require a bit of great simulation to understand what yo9u need to do). The difference between RTL behavioral simulation and timing simulation is that timing simulation accounts for the delays inside your device ( post-route). RTL simulation pretty much deals with ideal signal paths. This can change from build to build with or without any source code changes. You can, of course, add some crude delay elements to an RTL simulation on the IO to a testbench but you can't ever account for logic placement and internal delays at a useful level without running a timing simulation

I've learned not to get too over-joyed with perfect timing scores in my final reports. I've learned not to get too complacent with what appears to be 'working' designs. I've learned that FPGA development results are a lot more dynamic than I wished they were.

[edited] I forgot to mention that, depending on your settings, the synthesis tool will extract lines of your code and implement them as a known structure; like counter, state machine etc. That doesn't mean that it is understanding exactly what it is that you want to do....

1 hour ago, D@n said:

When you use this post-place-and-route simulation, do you go through this much effort in simulation, or can you cut corners anywhere?

The problem with timing simulation is that the timescale has to be in picoseconds and even if your RTL testbench and timing testbench are looking for the same faults the timing simulation will take a LOT longer. It should take longer because now it has to check the relationship of signal transitions to a clock edge. Of course the problem with any simulation is that if you are looking for the wrong things you are unlikely to have a simulation identify issues.

We'd all like simulation to be a 10 second affair that's peripheral to the design process. If you are using the simulator to find syntax errors then mission accomplished. ( I mentioned that I've had ISIM refuse to compile a simulation with code that the implementation tools were happy with enough to produce a bitstream... ). If you want to find those corner situation problems then you will find that, except for the simplest designs, a decent testbench will take a while.

The simulator in not the only tool that you should have in your toolbox. It is just a dumb tool like a hammer. It won't toe the nail into the frame by itself. The burden of understanding what issues to expect and simulate for are on the testbench writer. And there are a lot of places where a design might be deficient.

Edited by zygot

Share this post


Link to post
Share on other sites

@zygot,

I've only ever seen a couple of bugs where one placed solution would work and another one that passes the same timing requirements would not.  One bug was fixed by registering the outputs of the FPGA in the I/O elements.    The same can be applied to the inputs: register them as soon as they enter the chip, in the I/O element.  I've also been counseled to output clocks via ODDRs or OSERDES's.  Together, these approaches have kept roblems from cropping up in a PNR dependent fashion.  Are you unable to use these solutions?  Or do these approaches not apply for your designs?

Dan

Share this post


Link to post
Share on other sites
18 minutes ago, D@n said:

Together, these approaches have kept roblems from cropping up in a PNR dependent fashion

I appreciate your point. This raises the little issue of the 'elephant' in the room. Is our development process focused on fixing problems or gaining insight and a deeper understanding of how real world devices and interfaces work? There are, sometimes, certain easy to apply 'fixes' that seem to resolve a host of deficient design flaws. No doubt that this is the quickest path to a 'working' design. Then there is having the insight and knowledge to design for overcoming specific issues when and where it is required. Developing this knowledge and 'gut instincts' about designing FPGA systems takes quite a but longer, asking more questions when things appear to be 'working', spending the time to characterize particular scenarios, and of course simulating, simulating and simulating ( hopefully with better testbenches for each iteration. The things that you mention are IO timing related. A design can meet IO timing and still fail in internal portions of a complicated design.

No design is a good design if it doesn't address all of the issues that the real world hardware will face or all of the potential conditions that the logic will be exposed to. Gaining the insight to account for all of this in a design is an evolutionary, and usually time-consuming process.

As to my personal experience. I take the short route when I have to meet a deadline. I take the longer route when I want to learn something.

Give me a 'working' design and I can find a situation where it doesn't work as the designer intended.

Edited by zygot

Share this post


Link to post
Share on other sites

@zygot,

I'm still listening to your preaching, but not getting it.  I just don't believe in Voodoo logic design.

As background, I've helped the SymbioticEDA team build a PNR algorithm for iCE40 FPGA's, and I'm also peripherally aware of Google sponsoring similar work for the Xilinx chips as well.  (Both are part of the nextpnr project)  This was how I knew to comment about registered I/O's.  There are also metastability problems--problems that simulation won't necessarily reveal, post-PNR or not.  (Well, you might get lucky ...)  I trust you've been around long enough to avoid these.

My point is, having looked into the internals of the various FPGA's, and having written and debugged PNR algorithms, I'm still looking for an example of a design that passes a logic simulation, a timing check, and yet fails a post-PNR simulation.  My interest is to know if there's something that needs to be done within PNR to keep this from happening.

Do you have such an example that you can share?  (Other than latches--you have shared about those, but we already know that latches are bad.)  Even better if so, can you explain the underlying phenomenology that caused the problem?

Dan

Edited by D@n

Share this post


Link to post
Share on other sites
4 hours ago, D@n said:

I trust you've been around long enough to avoid these

There's no avoiding meta-stability issues, just minimizing the effects when they occur and occasionally finding a different way to implement you logic. It's a complex topic. You will find a number of 'solutions'. Pick your poison.

4 hours ago, D@n said:

I just don't believe in Voodoo logic design.

Neither do I. It would be nice if one could find a fool-proof way to create or implement perfect designs. FPGA devices aren't perfect. FPGA tools aren't perfect. OUr understanding of the problem is not perfect. I think that it's possible to become competent enough in say, C programming, to be able to adapt to almost any environment and be useful. It's not possible to be so good at it that you can develop fault-proof designs for real world hardware in real world environments. If you do all PC work you will have some challenges moving on to embedded work. FPGA development is a lot more complex than developing software for a particular CPU.

4 hours ago, D@n said:

I'm still looking for an example of a design that passes a logic simulation, a timing check, and yet fails a post-PNR simulation.

[edited] No problem, just keep doing FPGA development until someone has a very difficult project with specific validation requirements. Th's that short reply....

I've seen this on many occasions in many workplaces. The project that I'm working on now fits the bill. I'm not a believer in testimonials. You just never get all of the information that you need to make a good assessment. I've worked with brilliant people who just plain got an analysis wrong so I don't tend to follow testimonials blindly. By the way, to turn your argument around a bit; you're saying that you've done 100's of timing simulations and know  beyond a reasonable doubt that they are worthless? I know of quite a few companies who'd be shocked to hear this.

Every once in a while I run into someone who claims that everyone else, all the textbooks, all the professors have gotten something wrong and that they have the answer to a problem that no one else has found. This has happened to me at least 5 times as I can recall at the moment. One was a Phd with a small company selling power meters to utility companies. I can't think of a one of them who were able to prove themselves to be correct ( you might tend to think that if a utility is buying product that's proof enough but I think that if a utility sees a way to make more money they might not be too curious about how it's made ). Rarely, someone does come along and re-write the textbooks and alter a curriculum; but not that often. I am only conveying information that is a best practice in industry in my experience. There's usually, but not always, a good reason why most experienced places do something a certain way. Most companies are not into wasting time and money in development.

I really have no need nor interest in convincing anyone that the have to incorporate timing analysis into their skill set. ( and it is a skill set as using a tool is the only way to become adept at using it ) When people are ready to understand that they need to use a particular tool then all I need to do is inform them of its existence. [edited] In general, having to develop a new skill while under a lot of pressure to meet a deadline is not an ideal way to work. Better to have the honed skill in the toolbox ready for when it's needed.

My reaction to a lot of what you've been saying is that you haven't been exposed to very complicated designs, you haven't been exposed to rigorous validation requirements, or you aren't quite getting what's involved in the whole process. iCE40 FPGA's are one small niche in the FPGA world and represent small devices with limited capabilities ( that doesn't make them easier to develop for ). Again, you seem to think that IOs are a design. IOBs in Xilinx devices have features enhancing the part's usability for particular applications. I'm sure that you've created designs that don't fit into IOBs.

My experience with significantly large and or complex and or high speed designs is that it's very hard to nail down logic placement and routing over many design cycles and preserve performance and operational reliability. Once you arrive at such a situation timing simulation becomes more than an intellectual concept.

Edited by zygot

Share this post


Link to post
Share on other sites
2 hours ago, D@n said:

Did you see the recent presentation showing the Verilator can beat the big vendor simulators in speed by 5x or more?

Worked for me... it's a while ago now but I had my drivers working when silicon arrived and knew there was a saturation bug in the datapath that needed a gain staging workaround.

Success story for the marketing department, if there were any.

Edited by xc6lx45

Share this post


Link to post
Share on other sites
5 hours ago, D@n said:

Did you see the recent presentation showing the Verilator can beat the big vendor simulators in speed by 5x or more?

No, but I have seen ads telling me that I can eat anything that I want, do no exercise, and take a totally harmless pill that will return my body and appeal to my 20's self ( OK, disclaimer, I was very thin but not all that appealing back then either... ). I can eat my steak dinner 5x faster by shoving the contents of the plate into a blender and chugging it right from the jar... might even be healthier.

I do get the concept that having tools that provide answers very quickly can improve productivity. I've submitted card decks to a mainframe and waited a day or two to get back the results. I see simulation as a bit more than a tool though. The verification process is about more than moving on to another design challenge. The whole design process is an intellectual exercise. A simulator is ( can be ) to the intellect like a piece of gym equipment is to the body. There's more to design than 'I came, I saw, I conquered'. Understanding the nuanced details is, I think, better than blasting through projects.

Still, not feeling it. I guess that the argument that "never tried it, don't know what is, don't know what it does, don't care, don't need it" just isn't that compelling to me.

Again, I'm not trying to challenge anyone's religious views. When you get to a spot where the you start thinking that the must be something else to try then perhaps timing simulation will come to mind. If you never get to that spot then don't worry about it. If the pinnacle of your FPGA career is making the hex led displays look like a clock you don't have to worry about timing constraints or much else... certainly not timing simulations.

 

Edited by zygot

Share this post


Link to post
Share on other sites

The presentation was comparing verilator 4.0 with the big simulation vendors.  4.0 is a brand new release, so if you tried it a while ago, it's probably 5x faster now.

Dan

Share this post


Link to post
Share on other sites

well, the important thing for me wasn't even speed, but having datapath and actual drivers in the same simulation.
Maybe it's just my corners of the woods, but often the software is more complex than the hardware.

 

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this