Issues encoding image captured using cv2.VideoCapture(). Goal is to display image in an iPywidget

asked 2020-10-30 17:27:41 -0600

iacisme
1 ●1

Hello;

In my use case, I'm using a Jetson Nano running Jupyter Lab. I ssh into the nano, and run Jupyter Lab on a web browser on the host machine, a laptop.

As such, I can't use cv2.imshow() to display the image. This only works when run locally.

Therefore, I've learned to use cv2.VideoCapture() to capture an image, but I'm having trouble encoding this image into the format required by ipywidgets.

Here is my code for capturing one camera frame:

import cv2

import ipywidgets
from IPython.display import display
from IPython.display import Image

gst_pipeline = "nvarguscamerasrc sensor-id=0 ! video/x-raw(memory:NVMM), width=(int)1280, height=(int)720, format=(string)NV12, framerate=(fraction)30/1 ! nvvidconv ! video/x-raw ! jpegenc ! image/jpeg ! appsink"

camera = cv2.VideoCapture(gst_pipeline, cv2.CAP_GSTREAMER)

capture_state, captured_image = camera.read()

print(capture_state)

print(captured_image)

The output of this code produces these results:

capture_state = True

captured_image = [[255 216 255 ... 179 255 217]]

However, when I run the next code sequence, I get the following error:

error: OpenCV(4.1.1) /home/nvidia/host/build_opencv/nv_opencv/modules/imgcodecs/src/grfmt_base.cpp:145: error: (-10:Unknown error code -10) Raw image encoder error: Maximum supported image dimension is 65500 pixels in function 'throwOnEror'

This is the code I run, that produces that error:

encode_parameters = [int(cv2.IMWRITE_JPEG_QUALITY),20]

encode_state, encoded_image = bytes(cv2.imencode('.jpg', captured_image, encode_parameters))

In looking around the internt, it seems there is a need to do some sort of transform. In this code example, they are using a numpy array as shown here:

def __init__(self, *args, **kwargs):
     super(Camera, self).__init__(*args, **kwargs)
     if self.format == 'bgr8':
         self.value = np.empty((self.height, self.width, 3), dtype=np.uint8)
     self._running = False

I would really appreciate if someone could help me figure out what I need to do to get my "capture_image" into the right format so that I can do the encoding.

Once this step is good, I'll be able to view this image in an iPywidget - accomplishing my overall goal.

Can someone please help me?

Cheers!

Comments

U can't declared variable encode_parameters = . See the answer cv2.imencode

supra56 ( 2020-10-31 07:30:52 -0600 )edit

Thanks for your comment, I checked out what you sent me but it didn't work in my case. However; I did find a solution. I'll post it on the main quesiton though.

iacisme ( 2020-11-02 15:23:30 -0600 )edit

add a comment

answered 2020-11-02 15:39:28 -0600

iacisme
1 ●1

updated 2020-11-02 15:40:17 -0600

I found the solution to my issue.

The problem is that iPywidgets is expecting a binary value, while OpenCV stores the image in an arrary format. You have to do a conversion of an integer array to a binary stream in order to use as in an iPywidget. Here is how you do this in Jupyter.

Start off by importing the correct libraries:

import cv2
import numpy as np

import ipywidgets
from IPython.display import display
from IPython.display import Image

Next build your image capture pipline and prove it works:

gst_pipeline = "nvarguscamerasrc sensor-id=0 ! video/x-raw(memory:NVMM), width=(int)1280, height=(int)720, format=(string)NV12, framerate=(fraction)30/1 ! nvvidconv ! video/x-raw, format=(string)NV12, width=(int)640, height=(int)480  ! videoconvert ! appsink"

camera = cv2.VideoCapture(gst_pipeline, cv2.CAP_GSTREAMER)

capture_state, captured_image = camera.read()

print(capture_state)

print(captured_image.shape)

print(type(captured_image.shape))

print(captured_image.size)

print(captured_image.dtype)

Here are the outputs I get from these print statements:

True
(480, 640, 3)
<class 'tuple'>
921600
uint8

To me this is proves an image was captured and that it's stored in an array.

Next encode the image:

encode_parameters = [int(cv2.IMWRITE_JPEG_QUALITY),75]

encode_state, encoded_image = cv2.imencode('.jpg', captured_image, encode_parameters)

print(encode_state)

print(encoded_image.shape)

print(type(encoded_image.shape))

print(encoded_image.size)

print(captured_image.dtype)

Here are the outputs I got:

True
(26978, 1)
<class 'tuple'>
26978
uint8

Next is the critical step. This is the step that converts the integer array into a binary stream:

binary_image = encoded_image.tobytes()

This will do the conversion for you.

Next, link the binary_image object to an Ipywidget, and viola! You get the displayed image from your CSI camera!

image_widget = ipywidgets.Image(value=binary_image, format='jpeg', width=224, height=224)

display_window = ipywidgets.HBox([image_widget])

display(display_window)

Following these steps produced and image widget in Jupyter Lab, and works either local or remote (using ssh into my device).

I still need to refine the steps for capturing a video stream, but it's going to be very similar to these steps.

Cheers!

edit flag offensive delete link

add a comment

Issues encoding image captured using cv2.VideoCapture(). Goal is to display image in an iPywidget

Comments

1 answer

Links

Question Tools

Stats

Related questions

Issues encoding image captured using cv2.VideoCapture(). Goal is to display image in an iPywidget edit

Comments

1 answer

Links

Question Tools

Stats

Related questions

Issues encoding image captured using cv2.VideoCapture(). Goal is to display image in an iPywidget