Live video processing on Zybo board?

[Shuvo Sarkar]

Dear expertise,

I have implemented the hybo_hdmi_in demo and it's perfectly working. Now, I want to show a binary mask in the region of interest at the VGA output. Now, my question, Is it possible to do it only by modifying video_demo.c file. Any kind of coding related idea will be helpful. 

Thanks in advance-

Shuvo

 

[artvvb]

@Shuvo Sarkar

What exactly needs to be done depends on what you mean by "region of interest" and "binary mask". I will assume that you are trying to replace some area of what is being displayed on the screen with a rectangular image. A good starting point would be to take the input stream and output it with modifications. The DemoScaleFrame function in video_demo.c does this. The resolution scaling being done by this function also may or may not be desirable for your project. The Bilinear interpolation function implemented on line 473 of the original source is the primary point of interest here. The three variables required to tell what is being written to in the destination frame are the index, i, which can be used to determine the color channel being written to, and the destination coordinate variables xcoDest and ycoDest. A good starting point to be able to see changes being made would be to add extra code to black out a rectangular area of the screen. This can be accomplished by wrapping the destFrame[iDest] statement within an if statement, that either writes a zero to destFrame[iDest] or runs the bilinear interpolation of the source frame, depending on the coordinates of the target pixel in the destination frame.

How you store, access, and process the binary mask (overlay image?) is a large topic that I would need more details to provide information on.

Let us know if you have more questions.

-Arthur

[Shuvo Sarkar]

Hi @Arthur

This is really a great help for me. Yes, you are correct, I am trying to replace some area of the output video with a binary mask (overlay image). 

Regards-

Shuvo

[Shuvo Sarkar]

Quote

 

Hi @artvvb,

Thanks again. Now, I can selected any area of a frame and apply masking on that area. So what I need to do, If I want to stream the output video with this masking?

 

Regards-

Shuvo

[artvvb]

@Shuvo Sarkar

Are you selecting the output frame using the serial interface to the demo? I assume that what is currently being displayed on stream is a single captured frame with the masking applied? If this is the case, it is likely possible to modify the demo further so that it runs the "copy, scale, and mask" algorithm repeatedly. By this I basically just mean placing the modified DemoScaleFrame function inside of a while loop.

Caveat: I not certain what the performance cost of doing this would look like, so it may be worth looking into how quickly the algorithm runs (perhaps a little outside of the scope of the current discussion).

Thanks,

Arthur

[artvvb]

To add to my comments on possible performance problems:

Running this masking algorithm in the Zynq PS will likely be inherently slower than running it in the PL. Adding a stage to the output pipeline in hardware is likely going to be a better approach, with the caveat that it would be a significant amount more work. This stage would likely need to be created as a custom IP core that either:

1. Takes in an AXI stream and outputs an AXI stream, probably placed near the AXI stream converter IP in the pipeline (if I am remembering things correctly, I don't have access to Vivado at the moment). There may be a xilinx provided IP that does something like this, but I am unsure.

2. Takes in VGA signals and outputs VGA signals, placed directly before the output port. This approach would require more work in detecting the pixel position and resolution of the data stream, but would avoid the complexity of AXI.

For the time being though, it is still worth trying to make the algorithm work in the PS, this is just a hypothetical for if the performance of the PS design is unacceptable.

Thanks,

Arthur

[Shuvo Sarkar]

Thanks a lot for your kind reply. By modifying video_demo.c code I mean function 8. I am now able to make any area back of a captured frame.

iDest = ycoDest * stride;

 

xcoSrc = 0.0;

for (xcoDest = 0; xcoDest < destWidth; xcoDest++)

{

ix1y1 = iy1 + ((int) xcoSrc) * 3;

ix2y1 = ix1y1 + 3;

ix1y2 = ix1y1 + stride;

ix2y2 = ix1y1 + stride + 3;

 

xDist = xcoSrc - ((float) ((int) xcoSrc));

 

/*

* For loop handles all three colors

*/

for (i = 0; i < 3; i++)

{

x1y1 = (float) srcFrame[ix1y1 + i];

x2y1 = (float) srcFrame[ix2y1 + i];

x1y2 = (float) srcFrame[ix1y2 + i];

x2y2 = (float) srcFrame[ix2y2 + i];

 

/*

* Bilinear interpolation function

*/

                if (xcoDest > 1710 &&  1800) &&(ycoDest > 50 && ycoDest < 165) {

                  destFrame[iDest] =   0;

                 }else {

                    destFrame[iDest] = (u8) ((1.0-yDist)*((1.0-xDist)*x1y1+xDist*x2y1) + yDist*((1.0-xDist)*x1y2+xDist*x2y2));       

                }

               iDest++;

        }

    xcoSrc += xInc;

  }

 ycoSrc += yInc;

}

/*

* Flush the framebuffer memory range to ensure changes are written to the

* actual memory, and therefore accessible by the VDMA.

*/

Xil_DCacheFlushRange((unsigned int) destFrame, DEMO_MAX_FRAME);

I also want to do pixel averaging to blur that selected area. But, I'm not sure about which variable represents exactly pixel on that location. Please give me some hints. And could you refer me about that Xilinx provided IP which takes AXI stream and does the modification? 

thanks again-

Shuvo

 

[artvvb]

Unfortunately it looks like Xilinx's Video Mixer IP requires an additional license to use (I have not looked into pricing). This means that (without buying the license for the mixer) modifying the hardware pipeline would probably require a custom IP, which would be quite difficult. I should also note that I am not aware of anyone at Digilent that has used the Video Mixer IP before, so any questions about that would probably have to go to Xilinx.

For the coordinate systems used in the DemoScaleFrame function, the comments at variable declaration are helpful. The most relevant here is at the declaration of xcoDest and ycoDest.

int xcoDest, ycoDest; // Location of the destination pixel being operated on in the destination coordinate system

The for loop that contains the Bilinear interpolation function "handles all three colors". This means that within the byte-buffer destFrame, each set of three consecutive bytes represents the R,G,B color values for that pixel with 8 bits of data apiece. The macro DEMO_STRIDE defines the line length of the video buffer - noting that the buffer is always sized to be able to contain 1920x1080x3 bytes of color data, regardless of what resolution is actually being used. What this means is that to index into an area of the destination frame outside of the for loop - which may be helpful if the target color of pixels that are indexed earlier in the for loop depends on the color of later pixels - you can set a destination index, iDest, to be 3 * 1920 * (the y coordinate of your pixel) + 3 * (the x coordinate of your pixel). The RGB values of this pixel will be placed at the memory addresses destFrame[iDest], destFrame[iDest+1], and destFrame[iDest+2].

EDIT: I just noticed the phrase pixel averaging in your post, the rest of my response is just describing this.

Could you explain what you mean by blurring the area of the input frame? There are a lot of different algorithms to do something like this...

As an example of what I mean, you could figure out what each of the pixels in the area to be blurred would be if they were bilinearly interpolated, take a weighted greyscale conversion of each of the pixels, then take the average of these greyscale pixels, then write that average to each of the color channels of each of the pixels in the target area. I don't believe that the result of this method would look particularly good (a grey box, where just how grey it is depends on the input frame), but might be another good step.

-Arthur