1. Introduction --------------- The type system is used to attach meaning to the bytes in a GstBuffer. A plugin can decide to add metadata to the GstBuffer, this metadata will carry an associated typeid. Types are also used by the plugins to expose the type of their pads to the type system. Types are essential for autoplugging. We will explain the inner workings of the type system in this document, as well as the API used in the plugins. 2. Type properties ------------------ a) identification of the type Types are identified by one or more MIME types. b) detection of types The type of any given GstBuffer can be detected using - a typefind function: a function that will inspect the bytes in the buffer and return a gboolean indicating the buffer is of the given type. - a template for the bytes/bits in the data that must be satisfied in order for the GstBuffer to be of the given type. - other properties that act more like a hint like: the extension of the source filename. the URI of the source. 3. Type registration -------------------- a) the core libraries will create to types: video/raw image/raw audio/raw b) all other types will be provided by the plugins before a type can become available to other plugins, the plugin has to register the type. The system will keep a directed graph of the types and the plugins that operate on them. example: video/mpeg ! mpeg1parse / \ video/mpeg1 audio/mp3 -----\ ! ! \ ! mpeg_play mpeg2dec mpg123 xing ... \ / \ / video/raw audio/raw-----\ ! ! ! ! videosink SDLsink audiosink alsasink ... The system can find the needed plugins to convert video/mpeg to audio/raw using this graph. 4. type equivalence ------------------- some types can have the same meaning for example: audio/mp3 and audio/mpeg or video/raw and image/raw a plugin that exposes its output type as audio/mpeg and another plugins with input type audio/mp3 can be connected. The system has to know about the equivalence of both types, even it they have a different mime type. 5. type hierarchy ----------------- some plugins can ouput a specific subset of an allready existing type. example: mp3parse inputs audio/mp3 and packs the stream into mp3 audio frames with mime type: audio/mp3-frame mpg123 only accepts audio/mp3-frame but not audio/mp3. another mp3 decoder (libmpg123) can accept audio/mp3 (and thus also audio/mp3-frame) it must be possible to connect both libmpg123 and mpg123 to the mp3parse element. it must not be possible to connect mpg123 to an element that outputs only audio/mp3 but not audio/mp3-frame. We say that audio/mp3-frame is a more specific subset of type audio/mp3. we can create a type hierarchy: audio/mp3 / \ audio/mp3-frame audio/mp3-layer12 / \ audio/mp3-layer1 audio/mp3-layer2 6. multi-type pads ------------------ certain plugins might accept multiple non equivalent types in one of their input pads. Consequently a plugin might output non equivalent types in its output pad. It must therefore be possible to specify multiple types for a pad. example: mpegdemux may be able to demux both MPEG1 and MPEG2 system streams. we show the type hierarchy of the video/mpeg as follows: video/mpeg / \ video/mpeg1 video/mpeg2 ---------------\ / \ / \ ! mpeg1-system* mpeg1-video mpeg2-ts mpeg2-ps* mpeg2-video the mpegdemux element might specify the type of the input pad as one of video/mpeg1-system and video/mpeg2-ts 7. definition of types ---------------------- A plugin will provide the following information to the type system: - a mime type string describing the hierarchy and where the types they provide are located in that hierarchy. - typefind functions for some of the types. - extensions for some of the types We will propose a syntax to define the type hierarchy a) equivalent types : separated with a | sign ( audio/mp3 | audio/mpeg ) b) type hierarchy : in braces: ( audio/mp3 ( audio/mp3-frame)) c) multi-type pads ( mpegdemux ( video/mpeg1-system + video/mpeg2-ps) ) the pad will have type mpegdemux which is the parent for type video/mpeg1-system or video/mpeg2-ps mpegdemux / \ video/mpeg1-system video/mpeg2-ps once the type hierarchy has been registered, the typeid of each element can be obtained with: guint16 gst_type_find_by_mime (gchar *mime) extra typefind functions and/or extensions can be added to the typeid afterwards. 8. type matching ---------------- The more specific typefind functions, the functions associated with types in the leaf nodes, are tried first. when a specific type has been found ex. video/mpeg1-system elements that can handle this type or one of its parents are selected: mpegdemux: mpeg1parse: supermpegdecoder: video/mpeg video/mpeg video/mpeg ! ! mpegdemux video/mpeg1-system ! video/mpeg1-system In the above case, also the super mpeg element is selected because it stated that it can handle all sorts of video/mpeg data. example 2: suppose the typefind functions finds an mp3 stream, fillowing elments are selected: libmpg123 mp3parse: audio/mp3 audio/mp3 mpg123 is not selected because its pad type is too specific (audio/mp3-frame): mpg123 audio/mp3 ! audio/mp3-frame if the typefind would find a mp3-frame type, all three objects would be selected. 8. consideration ---------------- It is clear that clear indications have to be given to the type hierarchy, especially for the top nodes. The more specific an element is in its mime type specification, the more flexible and extendible the plugin system can be.