ZedBoard DDR Power Consumption Oddity

[MattW]

I have observed, what to me is, unexpected behavior in regards to the power consumption of the ZedBoard (Zynq-7000) when a processing core accesses the on board DDR memory in certain situations.  I have performed a number of tests were I monitor the power consumption of either the A9 core with DDR or a softcore Microblaze with DDR.  For both setups I have found cases where the power consumption of the board when actively running a program that frequently accesses DDR is lower than when the processor is sitting idle.  This does not make sense to me as I would expect the power when active to be the power of the core + the power to access DDR versus just the power of the core when idle.  I will describe a few of the cases I have tested and what I observed.  In all cases the “active” execution is running a version of GUPS (giga updates per second), a program that repeatedly accesses a random location in a memory array of a controllable size.

 

Microblaze connected to DDR through the AXI HP0 and HP1 ports of the Zynq A9. The local microblaze memory is only used for startup code.  All other code and data is in DDR.

Case 1: Microblaze with 32 kB I and D caches running GUPs with a 192MB working set.  When execution is complete the processor is put to sleep using the “sleep” assembly instruction

Observation 1a: With the I cache enabled and D cache disabled (meaning all accesses to the array to be updated should go to DDR) the board level power consumption is 7.2% *lower* when actively running than when put to sleep.

Observation 1b: When both caches are disabled (meaning all accesses should go to DDR), power consumption is 4.7% *lower* when actively running than when put to sleep.

 

Case 2: Microblaze with 32 kB I and D caches running GUPs with 4k working set (meaning it should fit within cache).  When execution is complete the processor is put in a loop that continuously accesses the same index of the data array and sums it into a dummy variable.

Observation 2a: With both caches disabled (meaning all accesses should go to DDR), power consumption is 10.6% *lower* in the busy wait than when actively running.

Observation 2b: With both caches enabled (meaning most/all accesses should hit in caches), power consumption is 1.1% higher when actively running than when put to sleep. 

 

Zynq-7000 A9 connected to DDR.

Case 3: A9 running GUPs with 384 MB working set (meaning most data accesses should go to DDR). When execution is complete the processor is put in idle loop similar to Case 2.

Observation 3a: With caches enabled power consumption is 0.1% *lower* during active execution than when in busy wait.

 

These observations are counter to a number of my expectations from traditional systems.

Expectation 1: A running core that misses in the cache (or has not cache) and must frequently go to DDR should have higher power consumption than a sleeping core.  The former has the power of both the core and DDR while the latter has just the power of a sleeping core (and basic DDR refresh).  Observations 1a and 1b are counter to this expectation.

Expectation 2: A core hitting in a local cache should consume less power than a core that has to frequently access DDR.  Observations 2a, 2b, and 3a are all counter to this expectation.

 

I am interested in two items related to the above observations:

1. A potential explanation for the above observations (I have a few possibilities, but none are easy to verify)

2. Is there some setup or configuration setting that would produce more expected behavior in terms of power consumption?

Thanks.

[elodg]

How and where are you measuring power consumption?

[MattW]

Across the current sense resistor on the ZedBoard that is designed for measuring current.  This provides the power consumption of the all components on the board.

[elodg]

I see. So your measurements refer to the board as a whole. My guess is that you are seeing higher power consumption peaks with cache hits, because the data is available and the processor can do the calculations without stalls. With cache misses, the processor stops until the cache line is fetched from DDR. Reading from DDR should takes a bit more power than from the cache, but power consumption should be evened out with less peaks. Total power integrated over time should be bigger, however.

As to sleep, I can only hypothesize that sleep is not working properly. Sleep power should be < idle power.

[MattW]

In regards to your first point, the power consumption I mention is average power consumption over a window.  The average power consumption for Case 2 is lower when accessing DDR (caches disabled) than when accessing cache.  As your answer says this would not be the expected case but is what I'm observing.

[elodg]

You could try disabling the memory controller in hardware and run the test program from OCM to make sure there is no DDR access happening. Compare that to hardware enabled, but expected to be running from cache.

[MattW]

I'm not sure if I 100% know what you intend by your statement so I tried cases: 1) an A9 with no DDR and 2) the original A9+DDR configuration where I call Xil_DisableMMU() at the start to hopefully disable the MMU.  If you where thinking something different than either of these let me know.  Since there is no DDR I can't do an apples-to-apples comparison with Case 3 because I can't have a 384 MB working set.  Instead I used a 4k working set.

Option 1 (no DDR)
-With caches enabled active power is high than sleep or busy wait states
-With caches disabled active power is lower than busy wait (which is a little odd, but might be explainable depending on the nature of the caches on OCM).  Active power is higher than when in sleep (wfi), which is what I'd expect.

Option 2 (disabled MMU)
-With or without caches active power is higher than sleep (wfi) but lower than busy wait.  The latter point seems a little odd, but it is hard to say for sure not knowing exactly what disable_MMU does.
Compared to the same setup run without the call to DisableMMU
-wfi idle power is unchanged
-active power with disabled MMU is lower than with MMU (which makes sense)
-busy wait power with disabled MMU is higher than with MMU with caches disabled (this makes less sense if disable MMU is truly disabling DDR)

On initial inspection I'm not sure if this sheds any light on the situation.