onCameraFrame callback must return Mat object, that will be show on the screen. 8UC4 RGBA Mat is expected. It looks like your code return some invalid or unexpected Mat.
You can set preview frame format to gray scale by setPreviewFormat method of your View object. So, color conversion to gray scale will not be needed.
