double free when cv::UMat from cv::Mat with custom UMatData

edit

It appears that cv::UMatData has two separate reference counts, one for mats, and one for UMats. I've recently run into a problem where I am trying to create a UMat from a Mat in order to get some OpenCL accelerated remapping, but the underlying UMatData is double destroyed. I've recreated a minimal example:

Edit 1: fixed no returns

#include <opencv/cv.h>
#include <iostream>

class TestAllocator : public cv::MatAllocator {
 public:
  TestAllocator() {}

  cv::UMatData* CreateUMatData(void* data, size_t size) {
    cv::UMatData* u = new cv::UMatData(this);
    u->data = u->origdata = static_cast<uchar*>(data);
    u->size = size;
    return u;
  }

  cv::UMatData* allocate(int dims, const int* sizes, int type, void* data, size_t* step, int flags,
                         cv::UMatUsageFlags usageFlags) const {
    std::cerr << "Unexpected allocation\n";
    return nullptr;
  }

  bool allocate(cv::UMatData* data, int accessflags, cv::UMatUsageFlags usageFlags) const {
    std::cerr << "Unexpected allcoation\n";
    return true;
  }

  void deallocate(cv::UMatData* u) const {
    std::cout << "Destroying object at 0x" << std::hex << (void*)u << std::endl;
  }
};

TestAllocator allocator_;

int main(int argc, char** argv) {
  unsigned char data[] = "abcdefghij";
  cv::Mat mat(cv::Size(2,5), CV_8UC1, (void*)data, 5);
  mat.u = allocator_.CreateUMatData(data, 10);
  mat.addref();
  mat.allocator = &allocator_;
  mat.u->userdata = nullptr;

  cv::UMat umat = mat.getUMat(cv::ACCESS_WRITE);
  umat.setTo(0);
}

I compile this example with

g++ --std=c++11 -o cv_umat_test cv_umat_test.cc -L ${CV_INSTALL_PATH}/lib/ -lopencv_core -I ${CV_INSTALL_PATH}/include/

where CV_INSTALL_PATH points to where I've installed opencv. When I run this program, the output is:

Destroying object at 0x0x163c020
Destroying object at 0x0x163c020

My question is, what is the correct way to avoid this double-destruction given that UMatData has two reference counts, and the data is destroyed when either goes to zero. It does not appear that the default allocator does anything special in this case. Am I supposed to use a custom allocator that checks both reference counts and only destroys data if both are zero? Am I using getUMat in an unintended way?