I'm writing around 1200 images to video with opencv's videowriter.
When running in pycharm my ram stays at around 6gb used.
When I run exactly the same code in jupyter my ram slowly increases and by the time we are nearly at 1200 frames the ram is maxed out and sometimes the computer completely hangs.
I've tired different codecs and they all have the same effect.
I've tried deleting the VideoWriter object after the rendering is complete and the RAM usage does not clear.
I've tried garbage collection and no effect on ram.
Restarting the jupyter kernel DOES clear the RAM.
Does anyone have any experience with this or can suggest any solutions?
EDIT:
Self hosted jupyter!
General configuration for OpenCV 4.2.0 =====================================
Version control: 4.2.0
Platform:
Timestamp: 2020-04-04T14:38:39Z
Host: Linux 4.15.0-1028-gcp x86_64
CMake: 3.9.0
CMake generator: Unix Makefiles
CMake build tool: /usr/bin/gmake
Configuration: Release
CPU/HW features:
Baseline: SSE SSE2 SSE3
requested: SSE3
Dispatched code generation: SSE4_1 SSE4_2 FP16 AVX
requested: SSE4_1 SSE4_2 AVX FP16 AVX2 AVX512_SKX
SSE4_1 (14 files): + SSSE3 SSE4_1
SSE4_2 (1 files): + SSSE3 SSE4_1 POPCNT SSE4_2
FP16 (0 files): + SSSE3 SSE4_1 POPCNT SSE4_2 FP16 AVX
AVX (4 files): + SSSE3 SSE4_1 POPCNT SSE4_2 AVX
C/C++:
Built as dynamic libs?: NO
C++ Compiler: /usr/lib/ccache/compilers/c++ (ver 4.8.2)
C++ flags (Release): -Wl,-strip-all -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wundef -Winit-self -Wpointer-arith -Wsign-promo -Wuninitialized -Winit-self -Wno-delete-non-virtual-dtor -Wno-comment -Wno-missing-field-initializers -fdiagnostics-show-option -Wno-long-long -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections -msse -msse2 -msse3 -fvisibility=hidden -fvisibility-inlines-hidden -O3 -DNDEBUG -DNDEBUG
C++ flags (Debug): -Wl,-strip-all -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wundef -Winit-self -Wpointer-arith -Wsign-promo -Wuninitialized -Winit-self -Wno-delete-non-virtual-dtor -Wno-comment -Wno-missing-field-initializers -fdiagnostics-show-option -Wno-long-long -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections -msse -msse2 -msse3 -fvisibility=hidden -fvisibility-inlines-hidden -g -O0 -DDEBUG -D_DEBUG
C Compiler: /usr/lib/ccache/compilers/cc
C flags (Release): -Wl,-strip-all -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wmissing-prototypes -Wstrict-prototypes -Wundef -Winit-self -Wpointer-arith -Wuninitialized -Winit-self -Wno-comment -Wno-missing-field-initializers -fdiagnostics-show-option -Wno-long-long -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections -msse -msse2 -msse3 -fvisibility=hidden -O3 -DNDEBUG -DNDEBUG
C flags (Debug): -Wl,-strip-all -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wmissing-prototypes -Wstrict-prototypes -Wundef -Winit-self -Wpointer-arith -Wuninitialized -Winit-self -Wno-comment -Wno-missing-field-initializers -fdiagnostics-show-option -Wno-long-long -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections -msse -msse2 -msse3 -fvisibility=hidden -g -O0 -DDEBUG -D_DEBUG
Linker flags (Release): -L/root/ffmpeg_build/lib -Wl,--gc-sections
Linker flags (Debug): -L/root/ffmpeg_build/lib -Wl,--gc-sections
ccache: YES
Precompiled headers: NO
Extra dependencies: ade /lib64/libgthread-2.0.so /lib64/libglib-2.0.so /lib64/libz.so dl m pthread rt
3rdparty dependencies: ittnotify libprotobuf libjpeg-turbo libwebp libpng libtiff libjasper IlmImf quirc
OpenCV modules:
To be built: calib3d core dnn features2d flann gapi highgui imgcodecs imgproc ml objdetect photo python3 stitching video videoio
Disabled: world
Disabled by dependency: -
Unavailable: java js python2 ts
Applications: -
Documentation: NO
Non-free algorithms: NO
GUI:
GTK+: NO
VTK support: NO
Media I/O:
ZLib: /lib64/libz.so (ver 1.2.3)
JPEG: libjpeg-turbo (ver 2.0.2-62)
WEBP: build (ver encoder: 0x020e)
PNG: build (ver 1.6.37)
TIFF: build (ver 42 - 4.0.10)
JPEG 2000: build (ver 1.900.1)
OpenEXR: build (ver 2.3.0)
HDR: YES
SUNRASTER: YES
PXM: YES
PFM: YES
Video I/O:
DC1394: NO
FFMPEG: YES
avcodec: YES (58.65.103)
avformat: YES (58.35.101)
avutil: YES (56.38.100)
swscale: YES (5.6.100)
avresample: NO
GStreamer: NO
v4l/v4l2: YES (linux/videodev2.h)
Parallel framework: pthreads
Trace: YES (with Intel ITT)
Other third-party libraries:
Lapack: NO
Eigen: NO
Custom HAL: NO
Protobuf: build (3.5.1)
OpenCL: YES (no extra features)
Include path: /io/opencv/3rdparty/include/opencl/1.2
Link libraries: Dynamic load
Python 3:
Interpreter: /opt/python/cp37-cp37m/bin/python (ver 3.7.6)
Libraries: libpython3.7m.a (ver 3.7.6)
numpy: /opt/python/cp37-cp37m/lib/python3.7/site-packages/numpy/core/include (ver 1.14.5)
install path: python
Python (for build): /opt/python/cp37-cp37m/bin/python
Java:
ant: NO
JNI: NO
Java wrappers: NO
Java tests: NO
Install to: /io/_skbuild/linux-x86_64-3.7/cmake-install
Code(there is a lot of it but the writing section):
for i, dte in enumerate(self._video_options["unique_dates"]):
df_date = self._get_date_df(i)
img = self.get_chart(df_date)
if i == 0:
# set writer
fourcc = cv2.VideoWriter_fourcc(*'XVID')
out = cv2.VideoWriter("./" + self._video_options["video_file_name"], fourcc=fourcc,
fps=self._video_options["fps"], frameSize=(img.shape[1], img.shape[0]))
out = self.write_extra_frames(i, out, img, df_date)
def write_extra_frames(self, i, out_writer, img, df_date):
times = self._video_options['frames_per_image']
last = False
for x in range(times):
# if first frame write the original data
if x == 0:
out_writer.write(img[:,:,::-1])
try:
df_date1 = self._get_date_df(i + 1, df_date[self.category_col])
temp_df = df_date.merge(df_date1.set_index(self.category_col)[self.value_col],
left_on=self.category_col,
right_index=True)
val_diff = (temp_df[self.value_col + "_y"] - temp_df[self.value_col + "_x"]) / (times - 1)
except IndexError:
last = True
# if not increment towards the next date evenly
else:
if not last:
temp_df[self.value_col] = temp_df[self.value_col + "_x"] + (val_diff * x)
if self._chart_options["sort"]:
temp_df = temp_df.sort_values([self.value_col, self.category_col])
img = self.get_chart(temp_df)
out_writer.write(img[:,:,::-1])
# increment loops
self._video_options['looptimes'] += 1
# LOGGING
self._write_log(self._video_options['starttime'],
self._video_options['looptimes'],
len(self._video_options['unique_dates']) * times)
return out_writer
