Types and Properties
There is a very large set of possible types that may be used to pass data
between elements. Indeed, each new element that is defined may use a new
data format (though unless at least one other element recognises that
format, it will be most likely be useless since nothing will be able to
link with it).
In order for types to be useful, and for systems like autopluggers to
work, it is necessary that all elements agree on the type definitions,
and which properties are required for each type. The &GStreamer; framework
itself simply provides the ability to define types and parameters, but
does not fix the meaning of types and parameters, and does not enforce
standards on the creation of new types. This is a matter for a policy to
decide, not technical systems to enforce.
For now, the policy is simple:
Do not create a new type if you could use one which already exists.
If creating a new type, discuss it first with the other &GStreamer;
developers, on at least one of: IRC, mailing lists.
Try to ensure that the name for a new format is as unlikely to
conflict with anything else created already, and is not a more
generalised name than it should be. For example: "audio/compressed"
would be too generalised a name to represent audio data compressed
with an mp3 codec. Instead "audio/mp3" might be an appropriate name,
or "audio/compressed" could exist and have a property indicating the
type of compression used.
Ensure that, when you do create a new type, you specify it clearly,
and get it added to the list of known types so that other developers
can use the type correctly when writing their elements.
Building a Simple Format for Testing
If you need a new format that has not yet been defined in our , you will want to have some general
guidelines on mimetype naming, properties and such. A mimetype would
ideally be one defined by IANA; else, it should be in the form
type/x-name, where type is the sort of data this mimetype handles (audio,
video, ...) and name should be something specific for this specific type.
Audio and video mimetypes should try to support the general audio/video
properties (see the list), and can use their own properties, too. To get
an idea of what properties we think are useful, see (again) the list.
Take your time to find the right set of properties for your type. There
is no reason to hurry. Also, experimenting with this is generally a good
idea. Experience learns that theoretically thought-out types are good,
but they still need practical use to assure that they serve their needs.
Make sure that your property names do not clash with similar properties
used in other types. If they match, make sure they mean the same thing;
properties with different types but the same names are
not allowed.
Typefind Functions and Autoplugging
With only defining the types, we're not yet there.
In order for a random data file to be recognized and played back as
such, we need a way of recognizing their type out of the blue. For this
purpose, typefinding
was introduced. Typefinding is the
process of detecting the type of a datastream. Typefinding consists of
two separate parts: first, there's an unlimited number of functions
that we call typefind functions, which are each
able to recognize one or more types from an input stream. Then,
secondly, there's a small engine which registers and calls each of
those functions. This is the typefind core. On top of this typefind
core, you would normally write an autoplugger, which is able to use
this type detection system to dynamically build a pipeline around an
input stream. Here, we will focus only on typefind functions.
A typefind function ususally lives in
gst-plugins/gst/typefind/gsttypefindfunctions.c,
unless there's a good reason (like library dependencies) to put it
elsewhere. The reason for this centralization is to decreate the
number of plugins that need to be loaded in order to detect a stream's
type. Below is an example that will recognize AVI files, which start
with a RIFF
tag, then the size of the file and then an
AVI
tag:
static void
gst_my_typefind_function (GstTypeFind *tf,
gpointer data)
{
guint8 *data = gst_type_find_peek (tf, 0, 12);
if (data &&
GUINT32_FROM_LE (&((guint32 *) data)[0]) == GST_MAKE_FOURCC ('R','I','F','F') &&
GUINT32_FROM_LE (&((guint32 *) data)[2]) == GST_MAKE_FOURCC ('A','V','I',' ')) {
gst_type_find_suggest (tf, GST_TYPE_FIND_MAXIMUM,
gst_caps_new_simple ("video/x-msvideo", NULL));
}
}
static gboolean
plugin_init (GstPlugin *plugin)
{
static gchar *exts[] = { "avi", NULL };
if (!gst_type_find_register (plugin, "", GST_RANK_PRIMARY,
gst_my_typefind_function, exts,
gst_caps_new_simple ("video/x-msvideo",
NULL), NULL))
return FALSE;
}
Note that
gst-plugins/gst/typefind/gsttypefindfunctions.c
has some simplification macros to decrease the amount of code. Make
good use of those if you want to submit typefinding patches with new
typefind functions.
Autoplugging has been discussed in great detail in the Application
Development Manual.
List of Defined Types
Below is a list of all the defined types in &GStreamer;. They are split
up in separate tables for audio, video, container, subtitle and other
types, for the sake of readability. Below each table might follow a
list of notes that apply to that table. In the definition of each type,
we try to follow the types and rules as defined by
IANA for as far as possible.
Jump directly to a specific table:
Note that many of the properties are not required,
but rather optional properties. This means that
most of these properties can be extracted from the container header,
but that - in case the container header does not provide these - they
can also be extracted by parsing the stream header or the stream
content. The policy is that your element should provide the data that
it knows about by only parsing its own content, not another element's
content. Example: the AVI header provides samplerate of the contained
audio stream in the header. MPEG system streams don't. This means that
an AVI stream demuxer would provide samplerate as a property for MPEG
audio streams, whereas an MPEG demuxer would not. A decoder needing
this data would require a stream parser in between two extract this
from the header or calculate it from the stream.
Table of Audio Types
Mime Type
Description
Property
Property Type
Property Values
Property Description
All audio types.
audio/*
All audio types
rate
integer
greater than 0
The sample rate of the data, in samples (per channel) per second.
channels
integer
greater than 0
The number of channels of audio data.
All raw audio types.
audio/x-raw-int
Unstructured and uncompressed raw fixed-integer audio data.
endianness
integer
G_BIG_ENDIAN (1234) or G_LITTLE_ENDIAN (4321)
The order of bytes in a sample. The value G_LITTLE_ENDIAN (4321)
means little-endian
(byte-order is least
significant byte first
). The value G_BIG_ENDIAN (1234)
means big-endian
(byte order is most
significant byte first
).
signed
boolean
TRUE or FALSE
Whether the values of the integer samples are signed or not.
Signed samples use one bit to indicate sign (negative or
positive) of the value. Unsigned samples are always positive.
width
integer
greater than 0
Number of bits allocated per sample.
depth
integer
greater than 0
The number of bits used per sample. This must be less than or
equal to the width: If the depth is less than the width, the
low bits are assumed to be the ones used. For example, a width
of 32 and a depth of 24 means that each sample is stored in a
32 bit word, but only the low 24 bits are actually used.
audio/x-raw-float
Unstructured and uncompressed raw floating-point audio data.
endianness
integer
G_BIG_ENDIAN (1234) or G_LITTLE_ENDIAN (4321)
The order of bytes in a sample. The value G_LITTLE_ENDIAN (4321)
means little-endian
(byte-order is least
significant byte first
). The value G_BIG_ENDIAN (1234)
means big-endian
(byte order is most
significant byte first
).
width
integer
greater than 0
The amount of bits used and allocated per sample.
buffer-frames
integer
Any
The number of frames per buffer. The reason for this property
is that the element does not need to reuse buffers or use data
spanned over multiple buffers, so this property - when used
rightly - will decrease latency.
Note that some people think that this property is very ugly,
whereas others think it is vital for the use of &GStreamer; in
professional audio applications. The special value zero is
reserved and implies that size is variable between buffers.
All encoded audio types.
audio/x-ac3
AC-3 or A52 audio streams.
There are currently no specific properties defined or needed for
this type.
audio/x-adpcm
ADPCM Audio streams.
layout
string
quicktime
, dvi
,
microsoft
or 4xm
.
The layout defines the packing of the samples in the stream. In
ADPCM, most formats store multiple samples per channel together.
This number of samples differs per format, hence the different
layouts. On the long term, we probably want this variable to die
and use something more descriptive, but this will do for now.
block_align
integer
Any
Chunk buffer size.
audio/x-cinepak
Audio as provided in a Cinepak (Quicktime) stream.
There are currently no specific properties defined or needed for
this type.
audio/x-dv
Audio as provided in a Digital Video stream.
There are currently no specific properties defined or needed for
this type.
audio/x-flac
Free Lossless Audio codec (FLAC).
There are currently no specific properties defined or needed for
this type.
audio/x-gsm
Data encoded by the GSM codec.
There are currently no specific properties defined or needed for
this type.
audio/x-alaw
A-Law Audio.
There are currently no specific properties defined or needed for
this type.
audio/x-mulaw
Mu-Law Audio.
There are currently no specific properties defined or needed for
this type.
audio/x-mace
MACE Audio (used in Quicktime).
maceversion
integer
3 or 6
The version of the MACE audio codec used to encode the stream.
audio/mpeg
Audio data compressed using the MPEG audio encoding scehem.
mpegversion
integer
1, 2 or 4
The MPEG-version used for encoding the data. The value 1 refers
to MPEG-1, -2 and -2.5 layer 1, 2 or 3. The values 2 and 4 refer
to the MPEG-AAC audio encoding schemes.
framed
boolean
0 or 1
A true value indicates that each buffer contains exactly one
frame. A false value indicates that frames and buffers do not
necessarily match up.
layer
integer
1, 2, or 3
The compression scheme layer used to compress the data
(only if mpegversion=1).
bitrate
integer
greater than 0
The bitrate, in bits per second. For VBR (variable bitrate)
MPEG data, this is the average bitrate.
audio/x-qdm2
Data encoded by the QDM version 2 codec.
There are currently no specific properties defined or needed for
this type.
audio/x-pn-realaudio
Realmedia Audio data.
raversion
integer
1 or 2
The version of the Real Audio codec used to encode the stream.
1 stands for a 14k4 stream, 2 stands for a 28k8 stream.
audio/x-speex
Data encoded by the Speex audio codec
There are currently no specific properties defined or needed for
this type.
audio/x-vorbis
Vorbis audio data
There are currently no specific properties defined or needed for
this type.
audio/x-wma
Windows Media Audio
wmaversion
integer
1,2 or 3
The version of the WMA codec used to encode the stream.
audio/x-paris
Ensoniq PARIS audio
There are currently no specific properties defined or needed for
this type.
audio/x-svx
Amiga IFF / SVX8 / SV16 audio
There are currently no specific properties defined or needed for
this type.
audio/x-nist
Sphere NIST audio
There are currently no specific properties defined or needed for
this type.
audio/x-voc
Sound Blaster VOC audio
There are currently no specific properties defined or needed for
this type.
audio/x-ircam
Berkeley/IRCAM/CARL audio
There are currently no specific properties defined or needed for
this type.
audio/x-w64
Sonic Foundry's 64 bit RIFF/WAV
There are currently no specific properties defined or needed for
this type.
Table of Video Types
Mime Type
Description
Property
Property Type
Property Values
Property Description
All video types.
video/*
All video types
width
integer
greater than 0
The width of the video image
height
integer
greater than 0
The height of the video image
framerate
double
greater than 0
The (average) framerate in frames per second. Note that this
property does not guarantee in any way that
it will actually come close to this value. If you need a fixed
framerate, please use an element that provides that (such as
videodrop
).
All raw video types.
video/x-raw-yuv
YUV (or Y'Cb'Cr) video format.
format
fourcc
YUY2, YVYU, UYVY, Y41P, IYU2, Y42B, YV12, I420, Y41B, YUV9, YVU9,
Y800
The layout of the video. See FourCC definition site
for references and definitions. YUY2, YVYU and UYVY are 4:2:2
packed-pixel, Y41P is 4:1:1 packed-pixel and IYU2 is 4:4:4
packed-pixel. Y42B is 4:2:2 planar, YV12 and I420 are 4:2:0
planar, Y41B is 4:1:1 planar and YUV9 and YVU9 are 4:1:0 planar.
Y800 contains Y-samples only (black/white).
video/x-raw-rgb
Red-Green-Blue (RBG) video.
bpp
integer
greater than 0
The number of bits allocated per pixel. This is usually 16, 24
or 32.
depth
integer
greater than 0
The number of bits used per pixel by the R/G/B components. This
is usually 15, 16 or 24.
endianness
integer
G_BIG_ENDIAN (1234) or G_LITTLE_ENDIAN (4321)
The order of bytes in a sample. The value G_LITTLE_ENDIAN (4321)
means little-endian
(byte-order is least
significant byte first
). The value G_BIG_ENDIAN (1234)
means big-endian
(byte order is most
significant byte first
). For 24/32bpp, this should always
be big endian because the byte order can be given in both.
red_mask, green_mask and blue_mask
integer
any
The masks that cover all the bits used by each of the samples.
The mask should be given in the endianness specified above. This
means that for 24/32bpp, the masks might be opposite to host byte
order (if you are working on little-endian computers).
All encoded video types.
video/x-3ivx
3ivx video.
There are currently no specific properties defined or needed for
this type.
video/x-divx
DivX video.
divxversion
integer
3, 4 or 5
Version of the DivX codec used to encode the stream.
video/x-dx
Digital Video.
systemstream
boolean
FALSE
Indicates that this stream is not a system
container stream.
video/x-ffv
FFMpeg video.
ffvversion
integer
1
Version of the FFMpeg video codec used to encode the stream.
video/x-h263
H-263 video.
There are currently no specific properties defined or needed for
this type.
video/x-h264
H-264 video.
There are currently no specific properties defined or needed for
this type.
video/x-huffyuv
Huffyuv video.
There are currently no specific properties defined or needed for
this type.
video/x-indeo
Indeo video.
indeoversion
integer
3
Version of the Indeo codec used to encode this stream.
video/x-jpeg
Motion-JPEG video.
There are currently no specific properties defined or needed for
this type. Note that video/x-jpeg only applies to Motion-JPEG
pictures (YUY2 colourspace). RGB colourspace JPEG images are
referred to as image/jpeg (JPEG image).
video/mpeg
MPEG video.
mpegversion
integer
1, 2 or 4
Version of the MPEG codec that this stream was encoded with.
Note that we have different mimetypes for 3ivx, XviD, DivX and
"standard" ISO MPEG-4. This is not a good
thing and we're fully aware of this. However, we do not have a
solution yet.
systemstream
boolean
FALSE
Indicates that this stream is not a system
container stream.
video/x-msmpeg
Microsoft MPEG-4 video deviations.
msmpegversion
integer
41, 42 or 43
Version of the MS-MPEG-4-like codec that was used to encode this
version. A value of 41 refers to MS MPEG 4.1, 42 to 4.2 and 43
to version 4.3.
video/x-msvideocodec
Microsoft Video 1 (oldish codec).
msvideoversion
integer
1
Version of the codec - always 1.
video/x-pn-realvideo
Realmedia video.
rmversion
integer
1, 2 or 3
Version of the Real Video codec that this stream was encoded
with.
video/x-rle
RLE animation format.
layout
string
"microsoft" or "quicktime"
The RLE format inside the Microsoft AVI container has a
different byte layout than the RLE format inside Apple's
Quicktime container; this property keeps track of the
layout.
depth
integer
1 to 64
Bitdepth of the used palette. This means that the palette
that belongs to this format defines 2^depth colors.
palette_data
GstBuffer
Buffer containing a color palette (in native-endian RGBA) used
by this format. The buffer is of size 4*2^depth.
video/x-svq
Sorensen Video.
svqversion
integer
1 or 3
Version of the Sorensen codec that the stream was encoded with.
video/x-tarkin
Tarkin video.
There are currently no specific properties defined or needed for
this type.
video/x-theora
Theora video.
There are currently no specific properties defined or needed for
this type.
video/x-vp3
VP-3 video.
There are currently no specific properties defined or needed for
this type. Note that we have different mimetypes for VP-3 and
Theora, which is not necessarily a good idea. This could probably
be improved.
video/x-wmv
Windows Media Video
wmvversion
integer
1,2 or 3
Version of the WMV codec that the stream was encoded with.
video/x-xvid
XviD video.
There are currently no specific properties defined or needed for
this type.
All image types.
image/jpeg
Joint Picture Expert Group Image.
There are currently no specific properties defined or needed for
this type. Note that image/jpeg only applies to RGB-colourspace
JPEG images; YUY2-colourspace JPEG pictures are referred to as
video/x-jpeg ("Motion JPEG").
image/png
Portable Network Graphics Image.
There are currently no specific properties defined or needed for
this type.
Table of Container Types
Mime Type
Description
Property
Property Type
Property Values
Property Description
video/x-ms-asf
Advanced Streaming Format (ASF).
There are currently no specific properties defined or needed for
this type.
video/x-msvideo
AVI.
There are currently no specific properties defined or needed for
this type.
video/x-dv
Digital Video.
systemstream
boolean
TRUE
Indicates that this is a container system stream rather than an
elementary video stream.
video/x-matroska
Matroska.
There are currently no specific properties defined or needed for
this type.
video/mpeg
Motion Pictures Expert Group System Stream.
systemstream
boolean
TRUE
Indicates that this is a container system stream rather than an
elementary video stream.
application/ogg
Ogg.
There are currently no specific properties defined or needed for
this type.
video/quicktime
Quicktime.
There are currently no specific properties defined or needed for
this type.
video/x-pn-realvideo
Digital Video.
systemstream
boolean
TRUE
Indicates that this is a container system stream rather than an
elementary video stream.
audio/x-wav
WAV.
There are currently no specific properties defined or needed for
this type.
Table of Subtitle Types
Mime Type
Description
Property
Property Type
Property Values
Property Description
None defined yet.
Table of Other Types
Mime Type
Description
Property
Property Type
Property Values
Property Description
None defined yet.