If you load image using imread() then the image will be in BGR format, OpenCV use BGR color space in default. See How the image matrix is stored in the memory.
So you need use the appropriate macro in cvtColor(),
means, CV_RGB2HSV should be CV_BGR2HSV on cvtColor()