mirror of
https://gitlab.freedesktop.org/gstreamer/gstreamer.git
synced 2024-07-04 21:55:55 +00:00
d47569cffc
Original commit message from CVS: Some more slight adjustments, plus more exact definitions for YUV formats supported. Please note that imo, we should support colorspace from/to *each* of these YUV formats, otherwise its whole existence has no use whatsoever apart from private formats.
296 lines
10 KiB
Plaintext
296 lines
10 KiB
Plaintext
Mimetypes in GStreamer
|
|
======================
|
|
|
|
1) What is a mimetype
|
|
---------------------
|
|
A mimetype is a combination of two (short) strings (words), the content
|
|
type and the content subtype, that make up a pair that describes a file
|
|
content type. In multimedia, mime types are used to describe the media
|
|
streamtype . In GStreamer, obsiously, we use mimetypes in the same way.
|
|
They are part of a GstCaps, that describes a media stream. Besides a
|
|
mimetype, a GstCaps also contains stream properties (GstProps), which
|
|
are combinations of key/value pairs, and a name.
|
|
|
|
An example of a mimetype is 'video/mpeg'. A corresponding GstCaps could
|
|
be created using:
|
|
GstCaps *caps = gst_caps_new("video_mpeg_type",
|
|
"video/mpeg",
|
|
gst_props_new("width", GST_PROPS_INT(384),
|
|
"height", GST_PROPS_INT(288),
|
|
NULL));
|
|
or using a macro:
|
|
GstCaps *caps = GST_CAPS_NEW("video_mpeg_type",
|
|
"video/mpeg",
|
|
"width", GST_PROPS_INT(384),
|
|
"height", GST_PROPS_INT(288)
|
|
);
|
|
|
|
Obviously, mimetypes and their corresponding properties are of major
|
|
importance in GStreamer for uniquely identifying media streams.
|
|
|
|
2) The problems
|
|
---------------
|
|
Some streams may have mimetypes or GstCaps that do not fully describe
|
|
the stream. In most cases, this is not a problem, though. For a stream
|
|
that contains Ogg/Vorbis data, we don't need to know the samplerate of
|
|
the raw audio stream, for example, since we can't play it back anyway.
|
|
The samplerate _is_ important for _raw_ audio, so a decoder would need
|
|
to retrieve the samplerate from the Ogg/Vorbis stream headers (that are
|
|
part of the bytestream) in order to pass it on in the GstCaps that
|
|
belongs to the decoded audio ('audio/raw').
|
|
However, other plugins *might* want to know such properties, even for
|
|
compressed streams. One such example is an AVI muxer, which does want
|
|
to know the samplerate of an audio stream, even when it is compressed.
|
|
|
|
Another problem is that many media types can be defined in multiple ways.
|
|
For example, MJPEG video can be defined as video/jpeg, video/mjpeg,
|
|
image/jpeg, video/avi with a compression of (fourcc) MJPG, etc. None of
|
|
these is really official, since there isn't an official mimetype for
|
|
encoded MJPEG video.
|
|
|
|
The main focus of this document is to propose a standardized set of
|
|
mimetypes and properties that will be used by the GStreamer plugins.
|
|
|
|
3) Different types of streams
|
|
-----------------------------
|
|
There are several types of media streams. The most important distinction
|
|
will be container formats, audio codecs and video codecs. Container
|
|
formats are bytestreams that contain one or more substreams inside it,
|
|
and don't provide any direct media data itself. Examples are Quicktime,
|
|
AVI or MPEG (bytestream). They mostly contain of a set of headers that
|
|
define the media stream(s) that is packed inside the container and the
|
|
media data itself.
|
|
Video codecs and audio codecs describe encoded audio or video data.
|
|
Examples are MPEG-1 video, DivX video, MPEG-1 layer 3 (MP3) audio or
|
|
Ogg/Vorbis audio. Actually, Ogg is a container format too (for Vorbis
|
|
audio), but these are usually used in conjunction with each other.
|
|
|
|
3a) Container formats
|
|
1 - AVI (Microsoft RIFF/AVI)
|
|
mimetype: video/avi
|
|
|
|
2 - Quicktime (Apple)
|
|
mimetype: video/quicktime
|
|
|
|
3 - MPEG (MPEG LA)
|
|
mimetype: video/mpeg
|
|
properties: 'systemstream' = TRUE (BOOLEAN)
|
|
|
|
4 - ASF (Microsoft)
|
|
mimetype: video/x-ms-asf
|
|
|
|
5 - WAV (PCM)
|
|
mimetype: audio/x-wav
|
|
|
|
6 - RealMedia (Real)
|
|
mimetype: audio/x-pn-realaudio
|
|
|
|
7 - DV (Digital Video)
|
|
mimetype: video/dv
|
|
properties: 'systemstream' = TRUE (BOOLEAN)
|
|
|
|
8 - Ogg (Xiph)
|
|
mimetype: application/x-ogg
|
|
|
|
9 - Matroska
|
|
mimetype: video/x-mkv
|
|
|
|
10 - Shockwave (Macromedia)
|
|
mimetype: application/x-shockwave-flash
|
|
|
|
Please note that we try to keep these mimetypes as similar as possible
|
|
to what's used as standard mimetypes in Gnome (Gnome-VFS/Nautilus) and
|
|
KDE (Konqueror).
|
|
|
|
3b) Video codecs
|
|
For convenience, the fourcc codes used in the AVI container format will be
|
|
listed along with the mimetype and optional properties.
|
|
|
|
Preface - (optional) properties for all video formats:
|
|
'width' = X (INT)
|
|
'height' = X (INT)
|
|
'pixel_width' and 'pixel_height' = X (2xINT, together aspect ratio)
|
|
|
|
1a - Raw Video (YUV/YCbCr)
|
|
mimetype: video/raw
|
|
properties: 'format' = 'XXXX' (fourcc)
|
|
known fourccs: YUY2, I420, Y41P, YVYU, UYVY, etc.
|
|
|
|
Note: some raw video formats have implicit alignment rules. We should
|
|
discuss this more.
|
|
Note: some formats have multiple fourccs (e.g. IYUV/I420 or YUY2/YUYV).
|
|
For each of these, we only use one (e.g. I420 and YUY2).
|
|
|
|
Currently recognized formats:
|
|
YUY2: packed, Y-U-Y-V order, U/V hor 2x subsampled (YUV-4:2:2, 16 bpp)
|
|
YVYU: packed, Y-V-Y-U order, U/V hor 2x subsampled (YUV-4:2:2, 16 bpp)
|
|
UYVY: packed, U-Y-V-Y order, U/V hor 2x subsampled (YUV-4:2:2, 16 bpp)
|
|
|
|
Y42B: planar, Y-U-V order, U/V hor 2x subsampled (YUV-4:2:2, 16 bpp)
|
|
YV12: planar, Y-V-U order, U/V hor+ver 2x subsampled (YUV-4:2:0, 12 bpp)
|
|
I420: planar, Y-U-V order, U/V hor+ver 2x subsampled (YUV-4:2:0, 12 bpp)
|
|
Y41B: planar, Y-U-V order, U/V hor 4x subsampled (YUV-4:1:1, 12bpp)
|
|
YUV9: planar, Y-U-V order, U/V hor+ver 4x subsampled (YUV-4:1:0, 9bpp)
|
|
YVU9: planar, Y-V-U order, U/V hor+ver 4x subsampled (YUV-4:1:0, 9bpp)
|
|
|
|
See http://www.fourcc.org/ for more information.
|
|
|
|
Note: YUV-4:4:4 (both planar and packed, in multiple orders) are missing.
|
|
|
|
1b - Raw Video (RGB)
|
|
mimetype: video/raw
|
|
properties: 'format' = 'RGB ' (fourcc)
|
|
'endianness' = 1234/4321 (INT) <- endianness
|
|
'depth' = 15/16/24 (INT) <- bits per pixel (depth)
|
|
'bpp' = 16/24/32 (INT) <- bits per pixel (in memory)
|
|
'red_mask' = bitmask (0x..) (INT) <- red pixel mask
|
|
'green_mask' = bitmask (0x..) (INT) <- green pixel mask
|
|
'blue_mask' = bitmask (0x..) (INT) <- blue pixel mask
|
|
|
|
'bpp' is the number of bits of memory used for each pixel. 'depth'
|
|
is the color depth.
|
|
|
|
24 and 32 bit RGB should always be specified as big endian, since
|
|
any little endian format can be transformed into big endian by
|
|
rearranging the color masks. 15 and 16 bit formats should generally
|
|
have the same byte order as the cpu.
|
|
|
|
Color masks are interpreted by loading 'bpp' number of bits using
|
|
'endianness' rule, and masking and shifting by each color mask.
|
|
Loading a 24-bit value cannot be done directly, but one can perform
|
|
an equivalent operation.
|
|
|
|
Examples:
|
|
msb .. lsb
|
|
- memory: RRRRRRRR GGGGGGGG BBBBBBBB RRRRRRRR GGGGGGGG ...
|
|
'bpp' = 24
|
|
'depth' = 24
|
|
'endianness' = 4321 (G_BIG_ENDIAN)
|
|
'red_mask' = 0xff0000
|
|
'green_mask' = 0x00ff00
|
|
'blue_mask' = 0x0000ff
|
|
|
|
- memory: xRRRRRGG GGGBBBBB xRRRRRGG GGGBBBBB xRRRRRGG ...
|
|
'bpp' = 16
|
|
'depth' = 15
|
|
'endianness' = 4321 (G_BIG_ENDIAN)
|
|
'red_mask' = 0x7c00
|
|
'green_mask' = 0x03e0
|
|
'blue_mask' = 0x003f
|
|
|
|
- memory: GGGBBBBB xRRRRRGG GGGBBBBB xRRRRRGG GGGBBBBB ...
|
|
'bpp' = 16
|
|
'depth' = 15
|
|
'endianness' = 1234 (G_LITTLE_ENDIAN)
|
|
'red_mask' = 0x7c00
|
|
'green_mask' = 0x03e0
|
|
'blue_mask' = 0x003f
|
|
|
|
2 - MPEG-1, -2 and -4 video (ISO/LA MPEG)
|
|
mimetype: video/mpeg
|
|
properties: 'systemstream' = FALSE (BOOLEAN)
|
|
'mpegversion' = 1/2/4 (INT)
|
|
known fourccs: MPEG, MPGI
|
|
|
|
3 - DivX 3.x, 4.x and 5.x video
|
|
mimetype: video/divx
|
|
optional properties: 'divxversion' = 3/4/5 (INT)
|
|
known fourccs: DIV3, DIV4, DIV5, DIVX, DX50, DIVX, divx
|
|
|
|
4 - Microsoft MPEG 4.1, 4.2 and 4.3
|
|
mimetype: video/x-msmpeg
|
|
optional properties: 'mpegversion' = 41/42/43 (INT)
|
|
known fourccs: MPG4, MP42, MP43
|
|
|
|
5 - Motion-JPEG (official and extended)
|
|
mimetype: video/jpeg
|
|
known fourccs: MJPG (YUY2 MJPEG), JPEG (any), PIXL (Pinnacle/Miro), VIXL
|
|
|
|
6 - Sorensen (Quicktime - SVQ1/SVQ3)
|
|
mimetypes: video/x-svq
|
|
properties: 'svqversion' = 1/3 (INT)
|
|
|
|
7 - H263 and related codecs
|
|
mimetype: video/h263
|
|
known fourccs: H263, i263, M263, x263, VDOW, VIVO
|
|
|
|
8 - RealVideo (Real)
|
|
mimetype: video/realvideo
|
|
|
|
9 - Digital Video (DV)
|
|
mimetype: video/dv
|
|
properties: 'systemstream' = FALSE (BOOLEAN)
|
|
known fourccs: DVSD, dvsd
|
|
|
|
10 - Windows Media Video 1 and 2 (WMV)
|
|
mimetype: video/wmv
|
|
properties: 'wmvversion' = 1/2 (INT)
|
|
|
|
11 - XviD
|
|
mimetype: video/xvid
|
|
known fourccs: xvid, XVID
|
|
|
|
12 - 3IVX
|
|
mimetype: video/3ivx
|
|
known fourccs: 3IV1, 3IV2
|
|
|
|
13 - Ogg/Tarkin (Xiph)
|
|
mimetype: video/x-tarkin
|
|
|
|
15 - VP3
|
|
mimetype: video/vp3
|
|
|
|
15 - Ogg/Theora (Xiph, VP3-like)
|
|
mimetype: video/x-theora
|
|
|
|
16 - Huffyuv
|
|
mimetype: video/x-huffyuv
|
|
known fourccs: HFYU
|
|
|
|
TODO: colorspace identifications for MJPEG? How? Move MPEG4-compatible codecs
|
|
over to video/mpeg with a subtype (xvid, divx3, divx4, divx5, iso, ...)?
|
|
interlacing?
|
|
|
|
3c) Audio Codecs
|
|
for convenience, the two-byte hexcodes (as are being used for identification
|
|
in AVI files) are also given
|
|
|
|
Preface - (optional) properties for all audio formats:
|
|
currently none
|
|
|
|
1 - Raw Audio
|
|
mimetype: audio/raw
|
|
properties: 'rate' = X (INT) <- samplerate
|
|
'width' = X (INT) <- memory bits per sample
|
|
'depth' = X (INT) <- used bits per sample
|
|
'law' = 0/1/2 (INT) <- no law (0), alaw (1) or mulaw (2)
|
|
'signedness' = X (BOOLEAN)
|
|
'channels' = X (INT) <- number of audio channels
|
|
'endianness' = 1234/4321 <- endianness of audio stream
|
|
|
|
2 - MPEG-1 layer 1/2/3 audio
|
|
mimetype: audio/x-mp3
|
|
properties: 'layer' = 1/2/3 (INT)
|
|
|
|
3 - Ogg/Vorbis
|
|
mimetype: audio/vorbis
|
|
|
|
4 - Windows Media Audio 1 and 2 (WMA)
|
|
mimetype: audio/wma
|
|
properties: 'wmaversion' = 1/2 (INT)
|
|
|
|
5 - AC3
|
|
mimetype: audio/a52
|
|
|
|
6 - FLAC (Free Lossless Audio Codec)
|
|
mimetype: audio/x-flac
|
|
|
|
4 - Status of this document
|
|
---------------------------
|
|
Not all plugins strictly follow these guidelines yet, but these are the
|
|
official types. Plugins not following these specs either use extensions
|
|
that should be documented, or are buggy (and should be fixed).
|
|
|
|
Blame Ronald Bultje <rbultje@ronald.bitfreak.net> aka BBB for any mistakes
|
|
in this document.
|