Make a list of known raster format headers (eg. PNG files begin with \x89PNG\r\n\x1a\n) and a list of known vector format headers (eg. SVG is XML with an svg root element... it may or may not have and <?xml ...> declaration at the top) and check whether the file matches them.
The imread function doesn't support loading vector file formats.
If you're talking about distinguishing photographic/scanned/etc. content from losslessly-compressed (eg. PNG) files rendered from vector images (eg. inkscape --export-png=FILENAME.png FILENAME.svg) then that would be an appropriate OpenCV question.