Jump to content
  • 0

WaveForms SDK: Silent Crash (Multiple Analog Discovery II's)


maling

Question

Hello all!

I've been working with the WaveForms SDK for over a year now to control a collection of nine Digilent Analog Discovery II's connected to a powered USB hub. For the most part, it's been great - but recently I've encountered a problem that I can't seem to debug.

This is primarily because on encountering most errors within the SDK, the SDK appears to abort the main Python script's execution silently, without raising an exception in Python. In this case, I've attempted to use FDwfGetLastErrorMsg, but no error is recognized by this method either.

Question 1: Is there a more powerful way to access error messages from the WaveForms SDK when using Python?

As for the issue itself, it's hard to pin down without those messages.

Something appears to happen within the WaveForms SDK, a fixed time after several functions from the WaveForms SDK are called by my Python process. These functions are called repeatedly, in a measurement loop that is meant to run for hours or days. Everything works correctly for some number of iterations, but eventually the execution always aborts silently. The number of successful iterations varies semi-randomly, but it appears to fail earlier if I have more Digilents connected. (With five, it runs for quite some time; with seven, it fails in a few minutes; with nine, it fails within one minute.)

I have verified that the timing of the silent failure does not coincide with any line of Python code that I am executing; it occurs a fixed amount of time after I have called the most recent SDK function.

Overall, it sounds very much like some kind of communication error between the Digilents and the WaveForms process, with multiple devices stepping on each other's toes.

Question 2: Does this sound familiar? Is there a tutorial for best practices when communicating with multiple Digilent Analog Discovery II's? And/or do you have any tips for debugging in situations like these?

 

System details:

  • Python 3.8.3
  • Anaconda 4.8.3
  • Windows 10

Thank you!

Link to comment
Share on other sites

12 answers to this question

Recommended Posts

  • 0

Hi @maling

I run the following code with 10 device for more than 10minutes several times without any problem.
multi.py similar to your GetSpectra()

image.png.521b9d3610aba9e6d9e3b6c8a615222a.png

 

I only got error when disconnecting the USB cable:

image.png.2c50a6b7fac4ae75008dfbefd6b969aa.png


At which function it fails for you?
For the error message use print(szerr.value) or cast the C string.

from ctypes import *
import sys
import time
from dwfconstants import *

if sys.platform.startswith("win"):
    dwf = cdll.dwf
elif sys.platform.startswith("darwin"):
    dwf = cdll.LoadLibrary("/Library/Frameworks/dwf.framework/dwf")
else:
    dwf = cdll.LoadLibrary("libdwf.so")

szerr = create_string_buffer(512)

def FDwf(fun, args):
    if fun(*args) == 0:
        dwf.FDwfGetLastErrorMsg(szerr)
        print(szerr.value)
        raise

#dwf.FDwfParamSet(DwfParamOnClose, c_int(1)) # 0 = run, 1 = stop, 2 = shutdown

cDevice = c_int()
dwf.FDwfEnum(c_int(0), byref(cDevice))
print("Devices: "+str(cDevice.value))

hdwfs = []
for k in range(cDevice.value):
    hdwf = c_int()
    FDwf(dwf.FDwfDeviceOpen, [c_int(k), byref(hdwf)])
    hdwfs.append(hdwf)

for k in range(len(hdwfs)):
    FDwf(dwf.FDwfAnalogImpedanceReset, [hdwfs[k]])
    FDwf(dwf.FDwfAnalogImpedanceModeSet, [hdwfs[k], c_int(8)]) # 0 = W1-C1-DUT-C2-R-GND, 1 = W1-C1-R-C2-DUT-GND, 8 = AD IA adapter
    FDwf(dwf.FDwfAnalogImpedanceReferenceSet, [hdwfs[k], c_double(1e3)]) # reference resistor value in Ohms
    FDwf(dwf.FDwfAnalogImpedanceFrequencySet, [hdwfs[k], c_double(1e3)]) # frequency in Hertz
    FDwf(dwf.FDwfAnalogImpedanceAmplitudeSet, [hdwfs[k], c_double(1)]) # 1V amplitude = 2V peak2peak signal
    FDwf(dwf.FDwfAnalogImpedanceConfigure, [hdwfs[k], c_int(1)]) # start

tstart = time.time()
sts = c_byte()
for j in range(100): # repeat
    for i in range(100): # steps
        for k in range(len(hdwfs)): # device
            print("repeat: "+str(j)+"  step: "+str(i)+"  dev: "+str(k))
            FDwf(dwf.FDwfAnalogImpedanceFrequencySet, [hdwfs[k], c_double(1e3+99e3*i/99)]) # frequency in Hertz # Once you set a value, it will start its measurement routine automatically
        time.sleep(0.01)
        for k in range(len(hdwfs)):
            FDwf(dwf.FDwfAnalogImpedanceStatus, [hdwfs[k], None]) # ignore last capture since we changed the frequency

        # Read the result from each of the devices, once they complete.
        for k in range(len(hdwfs)):
            hdwf = hdwfs[k]
            while True:
                FDwf(dwf.FDwfAnalogImpedanceStatus, [hdwf, byref(sts)])
                if sts.value == 2:
                    break
            # Compute the resistance and reactance from the harvested values
            resistance = c_double()
            reactance = c_double()
            FDwf(dwf.FDwfAnalogImpedanceStatusMeasure, [hdwf, DwfAnalogImpedanceResistance, byref(resistance)])
            FDwf(dwf.FDwfAnalogImpedanceStatusMeasure, [hdwf, DwfAnalogImpedanceReactance, byref(reactance)])

print("Done with "+str(len(hdwfs))+" devices in "+str(time.time()-tstart)+"s")
dwf.FDwfDeviceCloseAll()

 

Link to comment
Share on other sites

  • 0

Hi @maling

Could it be a powering issue?
Depending on usage each AD2 could take 2-5W. With 9 units this could be 18-45W, 3.6-9A/5V Is this in the limits of the USB hub/supply.

Each FDwf functions returns true (1) on success or false (0) on failure.
It could be used like this:

bool myfunc(){
    if FDwf...(.) ==0 return false
    if FDwf...(..) ==0 return false
    ...
    return true
}

if !myfunc() {
    FDwfGetLastErrorMsg(sz)
    print(sz)
}

Link to comment
Share on other sites

  • 0

Hi @attila

Thanks for the reply!

I just checked on the power issue (there was a chance - my USB hub could only supply 36 W.) However, dividing the Discovery II's between two identical hubs at 36 W each, the error still occurs with exactly the same behavior.

I have tried to use FDwfGetLastErrorMsg(sz) as you suggested, but without the immediate return statements. Does the immediate return statement change the behavior of the function?

Example:

bool myfunc(){
    FDwf...(.)
    FDwf...(..)
    ...
}

myfunc()
FDwfGetLastErrorMsg(sz)
print(sz)

Link to comment
Share on other sites

  • 0

Hi @maling

The 36W should be typically suffice suffice for 9 devices.
The FDwfGetLastErrorMsg should be called immediately after the failing function. Calling another function will clear the error of previous function.
For the mentioned Python 3 support I've only updated the py examples.

Edited by attila
Link to comment
Share on other sites

  • 0

Hi @attila,

Ah, thank you, that explains a lot.

I've tried implementing FDwfGetLastErrorMsg using the following wrapper:

    def FDwf(self, fun, args):

        if fun(*args) == 0:
            # If the function returns 0, that means an error has occurred.

            print('')
            print('There has been an error while calling ' + str(fun) + ' with arguments' + str(args))
            print('')

            # We have to go find the error
            szerr = create_string_buffer(512)
            self.dwf.FDwfGetLastErrorMsg(szerr)

            # Then print it
            print(szerr)

            # And abort execution
            raise NameError('Digilent')

After converting every call to FDwf functions to use this wrapper, I still do not receive any error messages (although the functions execute just fine up until the typical failure). This seems to indicate that whatever failure is occurring is something that is not well-handled in the underlying libraries. I believe that is consistent with my earlier observation that execution does not abort immediately following any given line of Python code. Do you have any intuition as to what might be going on?

Link to comment
Share on other sites

  • 0

Hi maling,

 

You may also want to take a look at pydwf (see https://pypi.org/project/pydwf/), it is generally a lot easier to use as it takes care of all the ctypes shenanigans, and it also checks the return value of each API call and raises a Python exception when an error is encountered. As the author of the package, I can highly recommend it :-)

Edited by reddish
Link to comment
Share on other sites

  • 0

Hi @attila,

I still have it on my list to try @reddish's pydwf project, but I believe that the above implementation is sufficient to catch any error messages which are generated.

Given that no error messages are being produced, I was hoping you might have additional insight into possible causes of the failure. It feels very much like the failure is related to some manner of delayed communication with the Analog Discovery II. I don't have enough expertise to understand what might be happening under the hood, but I would be happy to provide example code if that would be useful.

Thank you!

Link to comment
Share on other sites

  • 0

Hi @attila,

See below for an illustration of what does, and does not, produce the failure. The full code base is a bit large - if you need a full working example, let me know and I can do some rearranging. Of the functions below, GetSpectrum() does not produce the failure, while GetSpectra() does.

Thank you!

 

All nine devices are first connected. They each have the following configuration applied:

    DLLs.FDwf(dwf.FDwfAnalogImpedanceReset, [hdwf])
    DLLs.FDwf(dwf.FDwfAnalogImpedanceModeSet, [hdwf, c_int(8)]) # 0 = W1-C1-DUT-C2-R-GND, 1 = W1-C1-R-C2-DUT-GND, 8 = AD IA adapter
    DLLs.FDwf(dwf.FDwfAnalogImpedanceReferenceSet, [hdwf, c_double(reference)]) # reference resistor value in Ohms
    DLLs.FDwf(dwf.FDwfAnalogImpedanceFrequencySet, [hdwf, c_double(start)]) # frequency in Hertz
    DLLs.FDwf(dwf.FDwfAnalogImpedanceAmplitudeSet, [hdwf, c_double(1)]) # 1V amplitude = 2V peak2peak signal
    DLLs.FDwf(dwf.FDwfAnalogImpedanceConfigure, [hdwf, c_int(1)]) # start

Then either GetSpectrum() is called for each device one by one (no failure), or GetSpectra() is called to measure all devices simultaneously (failure).

# Take an impedance spectrum measurement
def GetSpectrum(hdwf, settings, DLLs, verbose = False):
    # Just takes a single spectrum from a device that's already been enumerated and opened. That's all it does.

    # Get the Digilent WaveForm library
    dwf = DLLs.dwf
    dmgr = DLLs.dmgr
    ftd = DLLs.ftd
    osname = DLLs.osname

    # Get the settings from settings
    steps = settings.steps

    # Set up some C variables
    sts = c_byte()
    szerr = create_string_buffer(512)

    # Prepare containers for measurement results
    frequencies = settings.frequencies
    rgRs = [0.0]*steps
    rgXs = [0.0]*steps

    # Perform the measurement redundantly
    repeat = 1 # Integer number of times to repeat the measurement
    for j in range(repeat):
        for i in range(steps):

            DLLs.FDwf(dwf.FDwfAnalogImpedanceFrequencySet, [hdwf, c_double(frequencies[i])]) # frequency in Hertz # Once you set a value, it will start its measurement routine automatically
            time.sleep(0.01)
            DLLs.FDwf(dwf.FDwfAnalogImpedanceStatus, [hdwf, None]) # ignore last capture since we changed the frequency
            while True:
                if DLLs.FDwf(dwf.FDwfAnalogImpedanceStatus, [hdwf, byref(sts)]) == 0: # This just checks whether the measurement using this frequency value has completed
                    dwf.FDwfGetLastErrorMsg(szerr)
                    print("There's been an error reading impedance")
                    print(str(szerr.value))
                    quit()
                if sts.value == 2:
                    break
            resistance = c_double()
            reactance = c_double()
            time.sleep()
            DLLs.FDwf(dwf.FDwfAnalogImpedanceStatusMeasure, [hdwf, DwfAnalogImpedanceResistance, byref(resistance)])
            DLLs.FDwf(dwf.FDwfAnalogImpedanceStatusMeasure, [hdwf, DwfAnalogImpedanceReactance, byref(reactance)])
            rgRs[i] += abs(resistance.value) # absolute value for logarithmic plot
            rgXs[i] += abs(reactance.value)

    # Adjust the data considering the degree of repetition
    for i in range(steps):
        rgRs[i] /= repeat
        rgXs[i] /= repeat

    # Return the data
    return Spectrum(frequencies, rgRs, rgXs)



# Take an impedance spectrum measurement
    # Rearranges the for loops so that multiple measurements can occur simultaneously, even though the main computation is single threaded
    # Works from a list of devices that have already been enumerated and opened. Returns all of their spectra.
def GetSpectra(hdwfs, settings, DLLs, verbose = False):

    # Get the Digilent WaveForm library
    dwf = DLLs.dwf
    dmgr = DLLs.dmgr
    ftd = DLLs.ftd
    osname = DLLs.osname

    # Get the settings from settings
    steps = settings.steps # Number of steps

    # Set up some C variables
    sts = c_byte()
    szerr = create_string_buffer(512)

    # Prepare containers for measurement results
    frequencies = settings.frequencies
    resistances = [[0.0]*steps for x in range(len(hdwfs))]
    reactances = [[0.0]*steps for x in range(len(hdwfs))]

    # Permit averaging of multiple measurements
    repeat = 1 # Integer number of times to repeat the measurement

    for j in range(repeat):

        # For each frequency...
        for i in range(steps):

            # Set frequency for all devices
            for k in range(len(hdwfs)):
                DLLs.FDwf(dwf.FDwfAnalogImpedanceFrequencySet, [hdwfs[k], c_double(frequencies[i])]) # frequency in Hertz # Once you set a value, it will start its measurement routine automatically

            # Wait for settings to take effect
            time.sleep(0.01)

            # Clear out the most-recently-read value for all devices
            # Do this all at once, so that we can wait for all devices to finish the next measurement at once.
            for k in range(len(hdwfs)):
                DLLs.FDwf(dwf.FDwfAnalogImpedanceStatus, [hdwfs[k], None]) # ignore last capture since we changed the frequency

            # Read the result from each of the devices, once they complete.
            for k in range(len(hdwfs)):
                hdwf = hdwfs[k]

                # Wait for the measurement to complete for this device.
                while True:
                    if DLLs.FDwf(dwf.FDwfAnalogImpedanceStatus, [hdwf, byref(sts)]) == 0: # This just checks whether the measurement using this frequency value has completed
                        dwf.FDwfGetLastErrorMsg(szerr)
                        print("There's been an error reading impedance")
                        print(str(szerr.value))
                        quit()
                    if sts.value == 2:
                        break

                # Compute the resistance and reactance from the harvested values
                resistance = c_double()
                reactance = c_double()
                DLLs.FDwf(dwf.FDwfAnalogImpedanceStatusMeasure, [hdwf, DwfAnalogImpedanceResistance, byref(resistance)])
                DLLs.FDwf(dwf.FDwfAnalogImpedanceStatusMeasure, [hdwf, DwfAnalogImpedanceReactance, byref(reactance)])
                resistances[k][i] += abs(resistance.value) # absolute value for logarithmic plot
                reactances[k][i] += abs(reactance.value)


    # Adjust the data considering the degree of averaging
    # For each device...
    for k in range(len(hdwfs)):
        for i in range(steps):
            resistances[k][i] /= repeat
            reactances[k][i] /= repeat

    # Return the data
    return [Spectrum(frequencies, resistances[k], reactances[k]) for k in range(len(hdwfs))]


# The Digilent FDwf library requires you to explicitly check for errors after every execution.
# This is a wrapper for convenience.
def FDwf(self, fun, args):

    if fun(*args) == 0:
        # If the function returns 0, that means an error has occurred.

        print('')
        print('There has been an error while calling ' + str(fun) + ' with arguments' + str(args))
        print('')

        # We have to go find the error
        szerr = create_string_buffer(512)
        self.dwf.FDwfGetLastErrorMsg(szerr)

        # Then print it
        print(szerr)

        # And abort execution
        raise NameError('Digilent')

 

Edited by maling
Link to comment
Share on other sites

  • 0

Hi @attila,

(Re accepted answer)

That's solved it!

I made the szerr.value modification, and did not get any error messages. But your example code (which runs perfectly!) helped me realize why: I think the issue isn't from the dwf library.

My device connection step (FDwfEnum) was much more complicated than what you posted - I had originally based it on the Enumerate.py example, since I needed the device serial numbers and thought the other debug information might be useful.

When I modified my device connection step to use only the dwf library (cutting out most of the Enumerate.py example - anything related to dmgr, ftd, and dftd), suddenly my old code worked perfectly. It's still beyond me exactly what was causing the issue, but it looks like the revised connection procedure has fixed it.

Thank you for your patience in helping me sort through this. I hope this thread helps someone who finds themselves in similar shoes!

Edited by maling
Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...