A very basic question about image data

edit

I believe that RAW files are (with a few exceptions) as close as you can get to the pixel readouts, and usually has no filtering or image altering processing. There are some scientific cameras that just dump straight to their output, but that's typically due to very high frame rates. There are also a few cameras that do some processing before saving a RAW, but wikipedia says that they are rare and disliked for doing so.

As to the layout of any particular format, you would have to check the format specifications. With the exception of a bitmap, there is almost always many things going on. Some formats are lossy, and others are lossless. PNG is lossless, so whatever information you save is exactly what you get out. JPEG is lossy, so what you read is not necessarily what you saved.

So you have a PNG or JPEG image. What does the data look like? Who cares? What matters is what you get after it's read. When you call the imread function, it decodes the image into a pixel buffer. This has rows and columns and in those rows and columns it stores the digital intensity values (usually as BGR not RGB). If you print the memory as values you'll see 0xABCDEF. So that's good.

But, that is not what is in the RAW format for a given camera. Often RAW images store more bytes of data, say 12 bits per color per pixel. So the R would be 0x0000 to 0x1FFF, and the same for the B and G. If you have something that can read your raw format, it likely gets decoded into something like that. Except if it's a Bayer camera, which it probably is, you don't have RGB pixels at the same place, they're in a grid. So it's not useable directly. You have to process it before it can be used.

Typically after all the processing is done, you have compressed it down to an 8-bit image 0x00 to 0xFF with RGB for every pixel and then you save it as a PNG or JPEG. That is what is usable by a human, and what can be displayed on a monitor. A few monitors allow 10 or even 12 bits of display range, but they are typically for professionals only.

Webcams rarely offer the ability to get RAW images, and provided only the filtered and compressed images. VideoCapture then decodes them into the pixel buffer for you to use.

To get RAW video live, you typically need a scientific camera that outputs in Coax-Express, Camera Link, Gig-E, or now there are a few USB Vision.

I dont quite understand what you're trying to ask here but I hope you do realize that PNG, JPEG and RAW refer to the image compression methods (basically a turn on how you store images) and RGB values refer to the colorspace in which images are saved. An image could be saved in any format and it need not be RGB all the time (well but it technically is due to RGB being a common standard).

If you're asking about the differences between PNG, JPEG, and RAW. Then I would suggest reading up on image compression

When VideoCapture procures a frame from a video of a file then the answer to your question would be "what is format of the video file?" moreover "what is the compression method used for the video file?" I would suggest reading up on video compression formats / video codecs.

I would suggest framing the question with a clear understanding as to what it is you want to ask about would always be a good start.