Adding a custom resolution to the Zybo Pcam 5C

[bitslip]

Hello,

I'm working with the Zybo Pcam 5C (18.2) reference design and want to add a custom resolution to the design: 2560 W x 2 H ( yes the height is 2 - it's not a mistake ).
With this resolution, I want to make the frame rate as high as possible.

I'm looking in the OV5640.h file to see how the existing resolutions have been set.
From example please look at the setting for 1920 * 1080 * 15fps below.

I know I'll have to change the fields I marked in Red. Do I need to change anything else to support  2560 W x 2 H ?

    config_word_t const cfg_1080p_15fps_[] =
    {//1920 x 1080 @ 15 fps, RAW10, MIPISCLK=210, SCLK=42MHz, PCLK=42M
        // PLL1 configuration
        // [7:4]=0100 System clock divider /4, [3:0]=0001 Scale divider for MIPI /1
        {0x3035, 0x41},
        // [7:0]=105 PLL multiplier
        {0x3036, 0x69},
        // [4]=0 PLL root divider /1, [3:0]=5 PLL pre-divider /1.5
        {0x3037, 0x05},
        // [5:4]=01 PCLK root divider /2, [3:2]=00 SCLK2x root divider /1, [1:0]=01 SCLK root divider /2
        {0x3108, 0x11},

        // [6:4]=001 PLL charge pump, [3:0]=1010 MIPI 10-bit mode
        {0x3034, 0x1A},

        // [3:0]=0 X address start high byte
        {0x3800, (336 >> 8 ) & 0x0F},
        // [7:0]=0 X address start low byte
        {0x3801, 336 & 0xFF},
        // [2:0]=0 Y address start high byte
        {0x3802, (426 >> 8 ) & 0x07},
        // [7:0]=0 Y address start low byte
        {0x3803, 426 & 0xFF},

        // [3:0] X address end high byte
        {0x3804, (2287 >> 8 ) & 0x0F},
        // [7:0] X address end low byte
        {0x3805, 2287 & 0xFF},
        // [2:0] Y address end high byte
        {0x3806, (1529 >> 8 ) & 0x07},
        // [7:0] Y address end low byte
        {0x3807, 1529 & 0xFF},

        // [3:0]=0 timing hoffset high byte
        {0x3810, (16 >> 8 ) & 0x0F},
        // [7:0]=0 timing hoffset low byte
        {0x3811, 16 & 0xFF},
        // [2:0]=0 timing voffset high byte
        {0x3812, (12 >> 8 ) & 0x07},
        // [7:0]=0 timing voffset low byte
        {0x3813, 12 & 0xFF},

        // [3:0] Output horizontal width high byte
        {0x3808, (1920 >> 8 ) & 0x0F},
        // [7:0] Output horizontal width low byte
        {0x3809, 1920 & 0xFF},
        // [2:0] Output vertical height high byte
        {0x380a, (1080 >> 8 ) & 0x7F},
        // [7:0] Output vertical height low byte
        {0x380b, 1080 & 0xFF},

        // HTS line exposure time in # of pixels Tline=HTS/sclk
        {0x380c, (2500 >> 8 ) & 0x1F},
        {0x380d, 2500 & 0xFF},
        // VTS frame exposure time in # lines
        {0x380e, (1120 >> 8 ) & 0xFF},
        {0x380f, 1120 & 0xFF},

        // [7:4]=0x1 horizontal odd subsample increment, [3:0]=0x1 horizontal even subsample increment
        {0x3814, 0x11},
        // [7:4]=0x1 vertical odd subsample increment, [3:0]=0x1 vertical even subsample increment
        {0x3815, 0x11},

        // [2]=0 ISP mirror, [1]=0 sensor mirror, [0]=0 no horizontal binning
        {0x3821, 0x00},

        // little MIPI <moderated>: global timing unit, period of PCLK in ns * 2(depends on # of lanes)
        {0x4837, 48}, // 1/42M*2

        // Undocumented anti-green settings
        {0x3618, 0x00}, // Removes vertical lines appearing under bright light
        {0x3612, 0x59},
        {0x3708, 0x64},
        {0x3709, 0x52},
        {0x370c, 0x03},

        // [7:4]=0x0 Formatter RAW, [3:0]=0x0 BGBG/GRGR
        {0x4300, 0x00},
        // [2:0]=0x3 Format select ISP RAW (DPC)
        {0x501f, 0x03}
    };

[BogdanVanca]

Hello @bitslip,

 

All the registers used for image windowing are between 0x3800 and 0x3814. You can find more info. by going to page 40 fro the fallowing datasheet: https://cdn.sparkfun.com/datasheets/Sensors/LightImaging/OV5640_datasheet.pdf

 

Best Regards,

Bogdan Vanca

[bitslip]

Hi Bogdan,

This doesn't answer my question.
I know about these registers...but as I said:

I don't want to JUST change the resolution - I want to make the frame rate (FPS) as high as possible.

Suppose we change the resolution to :  2560 W x 2 H using the windowing registers. What's next ?

1. What will be the highest FPS for that resolution ?

2. How do we set this new FPS ?

[BogdanVanca]

Hello @bitslip,

 

The next thing would be to make sure that the video clock generator outputs the right pixel clock for you desired resolution. For that you would have to configure the video clock generator. In the Zybo Project, this is done in VideoOutput.h (line 104 to 112), were the 742.5 MHz clock gets divided by three different factors for each resolution. 

The pixel clock represents the total number of pixels that need to be sent every second, or Pixel clock = Htotal × Vtotal × frame rate. In order to have space for the maximum frame rate, you will have to go with the highest possible frequency supported by the MIPI two-lane interface. In our case this frequency is equal with 192 MHz. So, theoretically you will have 192 MHz = 2560 x  2 x frame rate.  Unfortunately,  for this frequency the sensor only supports YUV format, so you would have to replace the AXIBayertoRGB IP with a another one that does the conversion to YUV, or you could simple put a Video Processing Subsystem after the AXIBayertoRGB.  

 

*I will forward your question to my colleagues but today all of them are out of office, so I will come back with a more definitive answer tomorrow. Hopefully that is ok for you. 

 

[bitslip]

Thanks,

What will be the highest frequency that will still be supported by the current Bayer Filter ?

[BogdanVanca]

Right now, the Stream clock that enters into the Bayer IP is the same aclk used trough the entire design. The IP doesn't have any internal constraint file for it, so I cannot put my finger on a value. In the current project, I think that it is at 150 MHz for covering the 1920x1080 resolution. Ideally, the highest supported frequency for this IP is equal with the video_aclk of the MIPI-CSI RX. For Zybo, this would be 175 MHz. You can achieve a  higher throughput  by increasing the Pixels per clock option to Dual or Quad. I presume that is already done in the project. 

 

[bitslip]

You mentioned the VideoOutput.h file as responsible for changing the pixel clock.

I'm a bit confused

Does it change the pixel clock for the sensor ?  Or only the IP components in the Block Design ( VDMA , rgb2dvi, etc...) ?

[BogdanVanca]

Yes, it can be a little bit confusing. Only the VDMA, rgb2dvi etc...

I was referring to this header file, because you have to make sure that you are providing the correct pixel clock for the resolution that you are expecting to see on you monitor.  So, for a more simple overview of the entire block design, you can devide the entire project in two: before the frame buffer and after the frame buffer. There is a pixel clock for the incoming sensor data, although the things are a little bit more complicated with the MIPI interface, and a pixel clock used for displaying the data at a chosen resolution.  

 

[bitslip]

 

Quote

Only the VDMA, rgb2dvi etc...

That's what I thought...I don't need these. I'll try to explain why.

You see...I don't send video to the monitor and don't use the VDMA.

My application requires to only receive images from the sensor and store them (in a special manner) in the ARM memory.

1.I removed all the data storage (VDMA) and Video Tx parts.

2.I designed a custom DMA to store the incoming data.

So now - the redesigned video chain looks like this:

MIPI_PHY (original) ------> MIPI_CSI (original) ------> Bayer (original) ------> DMA (my custom)

I tested it and it supports the existing resolutions very well. My custom DMA can support the new 2560 W x 2 H without a problem.
 

So now what I need is:

A. Make the sensor send the image at 2560 W x 2 H ( registers 0x3800 and 0x3814 ). This is straight forward. 

B. Increase the frame rate as much as possible. (I still need the red part of the video chain to function correctly because I don't want to redesign it). This is much more complex for me and I'm stuck at this stage.

Questions:

1. What is the highest frame rate that will still work with the red part of the video chain ( given a 2560 W x 2 H resolution ) ?  

2. How do I configure the OV5640 to this frame rate value ( what registers in the OV5640.h have to be updated ) ?

 

I hope it's clearer now,
Thanks a lot for your help Bogdan!

[BogdanVanca]

Yes, now everything is much clear for me. I will consult my colleagues tomorrow and I will try to give you a more detailed answer.

[BogdanVanca]

Hello @bitslip,

Sorry for my late response.

1. What is the highest frame rate that will still work with the red part of the video chain ( given a 2560 W x 2 H resolution ) ?  

Regarding the block design, the limitation will come from the maximum total bit rate that the MIPI-CSI Receiver has for your device speed grade. For Zynq-700  this is equal with 1260 Mb/s. So you could simple calculate:

 Total bit rate = 2560(w) * 2(h) * maximum frame rate(fps) * Nr. of bits per pixel(bit/pixel) = 1260 Mbp/s.

So, theoretically you could go as high as you want, as long as you are under the maximum total bit rate. I'm saying theoretically because I've never tested such a design. What I'm sure of, is that Zybo-Z7 can cover up the maximum data rate supported by the Pcam 5C.

I personally would not bother with the block design, because the real bottleneck of our project would be the video sensor. And that is because you cannot simply reduce the resolution and and increase the frame rate as much as you want. The frame rate has other dependencies such as the time of exposure and the integration interval. Exposure time is a variable that can at the longest be the inverse of the frame rate (e.g. 1/30s or faster for 30 fps). The larger the frame rate, the larger the time exposure would be. Unfortunately, I could'n find a maximum value into the Omnivision datasheet. But I'm more than sure that the sensor is not very versatile in that direction. 

For OV5640 the exposure time can be controlled by adjusting the time interval between recharging and sampling. It has an Auto Exposure Control (AEC).  Besides the automatic control, the exposure time can be set manually from external control. The maximum exposure values are in registers {0x380E, 0x380F} + {0x350C,0x350D}. For example, in our project you can check lines 473 to 478 were:

//HTS line exposure time in # of pixels Tline=HTS/sclk

{0x380c, (2500 >> 8 ) & 0x1F},

{0x380d, 2500 & 0xFF},

//VTS frame exposure time in # lines

{0x380e, (1120 >> 8 ) & 0xFF},

{0x380f, 1120 & 0xFF},

This configuration is for 30 fps. You have 84 MHz (sclk frecuency) / 2500 / 1120 = 30. You can change those values for going up with your frame rate. Again, I couldn't find a maximum value for those registers, so theoretically you can chose any values as long it fits your desired frame rate.

2. How do I configure the OV5640 to this frame rate value ( what registers in the OV5640.h have to be updated ) ?

As I presented before, the frame rate is not a variable that you could change by rewriting a register. It is more a result of what you are trying to do. Although, the closest registers to what you are asking, are those presented before who store the data for the total horizontal/vertical size. I would have wanted to give you a more clear response to your questions, bu I don't have a lot of insights regarding this sensor, only the datasheet given by OmniVision. I'm saying this because there is a probably good reason for which Omnivison chose a specific frame rate for each different resolution.  

 

[bitslip]

Thanks Bogdan!

So this is my plan:

1. First I'll set the resolution to 2560 W x 2 H ( registers 0x3800 and 0x3814 ) without changing any other register. The Pixel Clock and Exposure time will remain as in the default configuration of the reference design. 
If I understand you correctly, as long as my custom DMA supports this resolution - everything should work correctly and no changes will be required to the MIPI_PHY ------> MIPI_CSI ------> Bayer video chain.

2. Afterwards, I'll increase the FPS by increasing the Pixel Clock and decreasing the Exposure Time (lines 473 to 478 that you mentioned) and see how it impacts the image quality.

From what you said I understand that increasing the FPS (up-to a point of course) also won't require any changes in the MIPI_PHY ------> MIPI_CSI ------> Bayer video chain.

 

Let me know what you think,
Thanks again!

 

[Fred_HY]

Hello I don't know much about vdhl, I wanted to demonstrate Zybo-Z7-20-pcam-5c-2018.2-2, with vivado 2018.2, I tried everything with the different versions but I still don't have the images.I get a gray image on the monitor, I had to change the monitor but I have the same results. i need help please.

 

[BogdanVanca]

Hello @Fred_HY,

Please open a different forum thread. I don't want to mix two different discussions in a single forum thread. Thank You.