Help with Serial Communication with OpenScopMZ

[herg]

I am working on a project with the hopes of interfacing with the OpenScopeMZ via serial communication, to send parameters to the oscilloscope function and read the data back. With the end goal being direct serial communication with the OpenScopeMZ (bypassing the FTDI serial-usb converter).

It's been difficult finding documentation on how I might be able to do this.

I have read through the OpenScope Protocol https://reference.digilentinc.com/reference/software/openscope/communication-protocol and see that it uses JSON to communicate, which I am not previously familiar with but have been doing my best to read up on it.

Currently I have been able to connect to the OpenScopeMZ via USB and Putty, and I accidentally managed to bring up the following dialogue:

Quote

 

Unsupported option, please try again

Enter the number of the operation you would like to do:
1. Enter JSON mode
2. Calibrate the instruments
3. Save the current calibration values
4. Manage your WiFi connections
5. View all files names on the SD card
6. View all files names in flash
7. Set the Oscilloscope input gain

{"mode":"JSON","statusCode":0,"wait":500}

 

The "Unsupported option, please try again" came up after I tried to paste the command to put the OpenScopeMZ into JSON Command mode into Putty with the following code    

Quote

{"mode":"JSON"}\r\n

Then I just pressed the '1' key and the following response came up:

Quote

{"mode":"JSON","statusCode":0,"wait":500}

 

I have the OpenScopeMZ working with WaveForms Live and am using a baudrate of 1250000 to communicate via Putty.

How can I send the JSON commands documented in https://reference.digilentinc.com/reference/software/openscope/communication-protocol to the OpenScopeMZ?

Also, any information that might help me to utilize the oscilloscope functions (set parameters, trigger, and read data) would be greatly appreciated.

[JColvin]

Hi @herg,

I apologize for the delay; I have reached out to another engineer, @AndrewHolzer, who has more experience with the JSON material than myself.

Thanks,
JColvin

[AndrewHolzer]

Hi @herg,

I think you may be sending a literal '\r\n' instead of the ascii representation for those characters. puTTY will send those line endings automatically, although this may depend on what OS you are running puTTY on. There is an option to send an implicit Line Feed (LF) character whenever a Carriage Return (CR) character is sent, underneath the Terminal  category to the left. I am running Win10 and haven't had to enable that option to get things to work, but you context may vary from mine.

Now, all you need to do to send a JSON command, for example the JSON mode command, is to send 

{"mode": "JSON"}

and puTTY will take care of the rest.

As to information about the OpenScope instruments and functionality, I can give you some information such as what commands WaveForms Live is sending when you make a certain UI action, and what order those commands need to be in for a serial application. If you want anything more specific however, I'm going to need to know exactly what you are looking for.

Regards,
AndrewHolzer

[herg]

Hi @AndrewHolzer,

Thank you for your reply. I have successfully established serial communication with the OpenScopeMZ using PuTTy, and have been able to send commands for parameter changes and then read the parameters that I changed back.

As a side note and for others trying to access the OpenScope in a similar way, I was able to find this file: 

https://forum.digilentinc.com/applications/core/interface/file/attachment.php?id=7554

Please note, when you click on this link it downloads an .htm file that contains a list of Statuscodes, some of which I came across when communicating with the OpenScope via JSON.

 

As for what I would like to use in the OpenScope instrumentation:

-setup the 2 oscilloscope channels

-setup the trigger and arm it for single or continuous run (depending)

-then read data back

 

Questions that I have:

-How is the oscilloscope data transmitted when I request to read it? Is there a trigger pin that is involved indicating the data is ready to be read?

I understand that the OpenScope only has so many samples it can store, and I was hoping to be able to pull that data out, then let the OpenScope continue sampling (free run). If this is possible, would there be any lapse in the data between the 2 "chunks" of that the on-board memory is capable of holding?

 

StatusCodes I have encountered so far:

2684354592 - InstrumentNotConfigured

2684354589 - InstrumentNotArmed

2147483673 - TRGUnableToSetTrigger

 

I am having trouble setting the trigger and reading data back. When I send the command to "read" oscilloscope data back i get the "TRGUnableToSetTrigger" error code.

 

Any documentation or help you can provide will be very much appreciated.

Thank you.

[AndrewHolzer]

Hey @herg,
 

I am writing you now to let you know that I am working on a response to your previous thread post, but that it will take me some time. You may expect it by the end of the day, but I ask for your patience while I put it together; I have other responsibilities and you deserve an in depth and informative response.

Regards,
AndrewHolzer

[herg]

Hey @AndrewHolzer,

Thank you for your help, I am very impressed with you and your company's speed and quality of responses thus far, and I completely understand.

I look forward to your response and being able to move farther ahead with my project.

 

Best Regards,

herg

[AndrewHolzer]

You can follow along with how WaveForms Live sends the commands to the OpenScope, using the browser tools to inspect the HTTP requests, since the commands are either sent to the agent if using serial communication, or directly to the OpenScope using these requests. For Chrome and Firefox, press Ctrl+Shft+i then navigate to the network tab to see these commands and the responses received to them. You can also get an idea as to what commands to send and in what order by looking at the WaveForms Live GitHub source, however, I am not expecting you to take a dive into that just to figure this out.

If you are following along in the browser tools, navigate to the instrument panel, configure the device how you want and then hit run. The first command that you should see sent, will be a multi-command that sets the parameters for channel 1, the "osc" portion below,  and then one that sets the trigger. These can be sent as separate commands, but the OpenScope also supports these multi-commands that combine several commands into one HTTP request. The relevant section in the code would be at line 729 in waveforms-live/src/pages/instrument-panel/instrument-panel.ts, where the command is sent to the device after being constructed. I only specified a single channel when running this, but for two channels the "osc" command will have both a "1" and "2" portion detailing the parameters for channel 1 and channel 2 respectively. 

{
    "osc": {
        "1": [
            {
                "command": "setParameters",
                "vOffset": 0,
                "gain": 0.25,
                "sampleFreq": 94000000,
                "bufferSize": 940,
                "triggerDelay": 0
            }
        ]
    },
    "trigger": {
        "1": [
            {
                "command": "setParameters",
                "source": {
                    "instrument": "osc",
                    "channel": 1,
                    "type": "risingEdge",
                    "lowerThreshold": 470,
                    "upperThreshold": 500
                },
                "targets": {
                    "osc": [
                        1
                    ]
                }
            },
            {
                "command": "single"
            }
        ]
    }
}

The next command you should see sent is one to read the oscilloscope channels. The response that you get back here depends on the state of the OpenLogger. If the trigger is armed, but hasn't fired, you should get something like

{
    "osc": {
        "1": [
            {
                "command": "read",
                "statusCode": 2684354571,
                "wait": 0
            }
        ]
    }
}

where StatusCode 2684354571 corresponds to am InstrumentArmed error. In this case, WaveForms Live re-issues the read command until something else is received. 

If there is data then you'll get a Chunked Response which looks something like this 

FF\r\n
{
    "osc": {
        "1": [
            {
                "command": "read",
                "statusCode": 0,
                "binaryOffset": 0,
                "binaryLength": 1880,
                "acqCount": 39,
                "actualSampleFreq": 93984962,
                "pointOfInterest": 470,
                "triggerIndex": 470,
                "triggerDelay": 0,
                "actualTriggerDelay": 0,
                "actualVOffset": -1,
                "actualGain": 0.25,
                "wait": 0
            }
        ]
    }
}\r\n
758\r\n
<binary of length 1880 bytes>\r\n
0\r\n
\r\n

In short, each chunk is prefixed with the byte length of that chunk in hexadecimal followed by a CRLF, then the data itself, also followed by a CRLF. In our case, the device first sends back the JSON response (I've made this 'pretty' for easier reading). From the above example we know that the JSON portion is 255 bytes long.

After the JSON portion comes the data portion.  Each sample in the binary data consists of two bytes multiplied by the number of enabled channels. For a run with multiple channels enabled the channels are interlaced into one sample in numeric order, so the first two bytes of the sample correspond to channel 1, and the second two to channel 2. After the binary data is another chunk length specifier, but this will be zero and is signal to you that the transfer has completed. Once you have this response, you need to extract the channel data from each sample by de-interlacing it. I can show you how it's done in WaveForms Live if you are interested.

Now, WaveForms Live sets the OpenScope trigger to Single, and resets it if doing a continuous run. There is the option to set the trigger to re-arm after acquisition with a Trigger Run command. I can't make a comment as to why WaveForms Live does it like this, since my time with the software was during OpenLogger development, and I don't know why this design decision was taken. In the case that you let it run continually, you'll just send a read command repeatedly and respond accordingly if the trigger is just armed, if there is data available, or some other state has been reached.

I believe this will answer your questions, however if you still need further information and details do ask and I will be happy to provide.

Regards,
AndrewHolzer

[herg]

Hey @AndrewHolzer,

Thanks for the information. I will try what you have suggested above and get back to you.

Regards,
herg

[herg]

Good Afternoon @AndrewHolzer,

Your last response has been very helpful. I was able to view the messages and data being sent through WaveformsLive on Google Chrome browser.

Just to clarify something in your last post, the only way to tell when the data buffer is full and ready to be read (for the osc instrument) I have to keep requesting a "read" and then check the response message and interpret the status? Would there be any way to assign one of the I/O pins to go high or low when the buffer is full and ready to be read?

Also, if you can explain to me how to de-interlace it (was not familiar with this term, but found it originates from TV signal processing) and extract the data points (time vs. voltage) that would be great, as that is very important for me.

 

Thanks again for all of your help.

Best Regards,
herg

[AndrewHolzer]

Hi @herg!

Good to hear from you again. I am happy to hear that you found my response helpful. I want to let you know that I'm working on a description of the de-interlace process and data-point extraction. I'm going to need some time to work on that, but you can be sure that it is in the pipeline. I do want to answer the questions in your second paragraph, however.

To start, there isn't currently any functionality the way you've described. The firmware would need some modification to allow the dynamic functionality you describe, or to wire that behavior into the system directly. Now, I came in after OpenScope, but looking at the schematic and the pinout on WaveForms Live, it looks like this functionality was at least planned for with the Trigger Out pin. I dug through the code firmware and found no use of this pin anywhere, unfortunately.

I'll continue on the de-interlace explanation, and I appreciate your patience throughout our correspondence.

Regards,
AndrewHolzer

[AndrewHolzer]

Hi @herg,

I apologize for the long delay, and want to say that I appreciate your patience. I took a look into the source code to reaffirm my knowledge and found that things are a bit easier than I have previously described to you. I had erroneously assumed that the oscilloscope data is sent to WaveForms Live with the samples interlaced into a binary blob. In short, each channel's databuffer is a contiguous portion of the binary data, whose lengths and positions are known from the JSON portion of the response. The read response is still sent in a chunked-transfer format, and I have included an example response for reference here. Note that I have added newlines in the JSON portion to make it easier to read and they aren't included in the response normally. Each chunk first begins with its length given in a hex format which is followed by a CRLF (\r\n in the example) and then the data portion also followed by a CRLF. The read response consists of the first chunk that contains a JSON data structure, followed by the binary data portion and finally followed by a zero-sized chunk.

In the JSON portion, each channel has a binaryOffset which is used to index the byte position in the binary data for that specific channel. The binaryLength details the byte count of that channels data buffer. Each sample consists of 2 bytes, so these binary values can be converted into a sample index and sample count by dividing the binary values by 2.

The time step for each channel is also found in the JSON portion, from the actualSampleFreq. This value is in mHz which is noted in the device enumeration as sampleFreqUnits. Also found in the enumeration is the voltageUnits, mV for the OpenScope, so proper scaling is required if you want straight Volt units. The triggerDelay is also needed from the JSON portion, as it is used to ultimately calculate the sample's time value as the trigger position. Trigger position can be found by the following formula:

triggerPosition = -1 * triggerDelay / 10^12 + dt * (# of samples in channel buffer)

Ideally, you'd use the binaryOffset and binaryLength to separate each channel buffer from the binary data. Then, for each channel buffer, iterate the collection sample by sample, or 2 bytes at a time. If we let idx represent the iteration index/count, the time value for that sample can then be found by the formula

t = idx * dt - triggerPosition

and the voltage value by dividing the value being indexed by 1000. At this point WaveForms Live packages the point for use with the charting system, but you are free to use it how you see fit. I understand that there is quite a bit being described here at a rather high level, and if there needs to be further clarification I'd be happy to help further.

I hope this info does you well and sees you to your goal.

Regards,
Andrew

Example Read Response

1FF\r\n
{
    "osc": {
        "1": [
            {
                "command": "read",
                "statusCode": 0,
                "binaryOffset": 0,
                "binaryLength": 4724,
                "acqCount": 36,
                "actualSampleFreq": 236127508,
                "pointOfInterest": 1181,
                "triggerIndex": 1181,
                "triggerDelay": 0,
                "actualTriggerDelay": 0,
                "actualVOffset": -1,
                "actualGain": 0.25,
                "wait": 0
            }
        ],
        "2": [
            {
                "command": "read",
                "statusCode": 0,
                "binaryOffset": 4724,
                "binaryLength": 4724,
                "acqCount": 36,
                "actualSampleFreq": 236127508,
                "pointOfInterest": 1181,
                "triggerIndex": 1181,
                "triggerDelay": 0,
                "actualTriggerDelay": 0,
                "actualVOffset": -50,
                "actualGain": 0.25,
                "wait": 0
            }
        ]
    }
}\r\n
24E8\r\n
<ch1-binary><ch2-binary>\r\n
0\r\n
\r\n