Revision history [back]

If found the solution to my issue.

The problem is that iPywidgets is expecting a binary value, while OpenCV stores the image in an arrary format. You have to do a conversion of an integer array to a binary stream in order to use as in an iPywidget. Here is how you do this in Jupyter.

Start off by importing the correct libraries:

import cv2
import numpy as np

import ipywidgets
from IPython.display import display
from IPython.display import Image

Next build your image capture pipline and prove it works:

gst_pipeline = "nvarguscamerasrc sensor-id=0 ! video/x-raw(memory:NVMM), width=(int)1280, height=(int)720, format=(string)NV12, framerate=(fraction)30/1 ! nvvidconv ! video/x-raw, format=(string)NV12, width=(int)640, height=(int)480  ! videoconvert ! appsink"

camera = cv2.VideoCapture(gst_pipeline, cv2.CAP_GSTREAMER)

capture_state, captured_image = camera.read()

print(capture_state)

print(captured_image.shape)

print(type(captured_image.shape))

print(captured_image.size)

print(captured_image.dtype)

Here are the outputs I get from these print statements:

True
(480, 640, 3)
<class 'tuple'>
921600
uint8

To me this is proves an image was captured and that it's stored in an array.

Next encode the image:

encode_parameters = [int(cv2.IMWRITE_JPEG_QUALITY),75]

encode_state, encoded_image = cv2.imencode('.jpg', captured_image, encode_parameters)

print(encode_state)

print(encoded_image.shape)

print(type(encoded_image.shape))

print(encoded_image.size)

print(captured_image.dtype)

Here are the outputs I got:

True
(26978, 1)
<class 'tuple'>
26978
uint8

Next is the critical step. This is the step that converts the integer array into a binary stream:

binary_image = encoded_image.tobytes()

This will do the conversion for you.

Next, link the binary_image object to an Ipywidget, and viola! You get the displayed image from your CSI camera!

image_widget = ipywidgets.Image(value=binary_image, format='jpeg', width=224, height=224)

display_window = ipywidgets.HBox([image_widget])

display(display_window)

Following these steps produced and image widget in Jupyter Lab, and works either local or remote (using ssh into my device).

I still need to refine the steps for capturing a video stream, but it's going to be very similar to these steps.

Cheers!

If I found the solution to my issue.