mirror of
https://gitlab.freedesktop.org/gstreamer/gstreamer.git
synced 2024-11-26 03:31:05 +00:00
294 lines
11 KiB
Text
294 lines
11 KiB
Text
Decodebin design
|
|
|
|
GstDecodeBin
|
|
------------
|
|
|
|
Description:
|
|
|
|
Autoplug and decode to raw media
|
|
|
|
Input : single pad with ANY caps Output : Dynamic pads
|
|
|
|
* Contents
|
|
|
|
_ a GstTypeFindElement connected to the single sink pad
|
|
|
|
_ optionally a demuxer/parser
|
|
|
|
_ optionally one or more DecodeGroup
|
|
|
|
* Autoplugging
|
|
|
|
The goal is to reach 'target' caps (by default raw media).
|
|
|
|
This is done by using the GstCaps of a source pad and finding the available
|
|
demuxers/decoders GstElement that can be linked to that pad.
|
|
|
|
The process starts with the source pad of typefind and stops when no more
|
|
non-target caps are left. It is commonly done while pre-rolling, but can also
|
|
happen whenever a new pad appears on any element.
|
|
|
|
Once a target caps has been found, that pad is ghosted and the
|
|
'new-decoded-pad' signal is emitted.
|
|
|
|
If no compatible elements can be found for a GstCaps, the pad is ghosted and
|
|
the 'unknown-type' signal is emitted.
|
|
|
|
|
|
* Assisted auto-plugging
|
|
|
|
When starting the auto-plugging process for a given GstCaps, two signals are
|
|
emitted in the following way in order to allow the application/user to assist or
|
|
fine-tune the process.
|
|
|
|
_ 'autoplug-continue' :
|
|
|
|
gboolean user_function (GstElement * decodebin, GstPad *pad, GstCaps * caps)
|
|
|
|
This signal is fired at the very beginning with the source pad GstCaps. If
|
|
the callback returns TRUE, the process continues normally. If the callback
|
|
returns FALSE, then the GstCaps are considered as a target caps and the
|
|
autoplugging process stops.
|
|
|
|
- 'autoplug-factories' :
|
|
|
|
GValueArray user_function (GstElement* decodebin, GstPad* pad,
|
|
GstCaps* caps);
|
|
|
|
Get a list of elementfactories for @pad with @caps. This function is used to
|
|
instruct decodebin2 of the elements it should try to autoplug. The default
|
|
behaviour when this function is not overridern is to get all elements that
|
|
can handle @caps from the registry sorted by rank.
|
|
|
|
- 'autoplug-select' :
|
|
|
|
gint user_function (GstElement* decodebin, GstPad* pad, GstCaps* caps,
|
|
GValueArray* factories);
|
|
|
|
This signal is fired once autoplugging has got a list of compatible
|
|
GstElementFactory. The signal is emitted with the GstCaps of the source pad
|
|
and a pointer on the GValueArray of compatible factories.
|
|
|
|
The callback should return the index of the elementfactory in @factories
|
|
that should be tried next.
|
|
|
|
If the callback returns -1, the autoplugging process will stop as if no
|
|
compatible factories were found.
|
|
|
|
The default implementation of this function will try to autoplug the first
|
|
factory of the list.
|
|
|
|
* Target Caps
|
|
|
|
The target caps are a read/write GObject property of decodebin.
|
|
|
|
By default the target caps are:
|
|
|
|
_ Raw audio : audio/x-raw-int, audio/x-raw-float
|
|
|
|
_ and raw video : video/x-raw-rgb, video/x-raw-yuv
|
|
|
|
_ and Text : text/plain, text/x-pango-markup
|
|
|
|
|
|
* media chain/group handling
|
|
|
|
When autoplugging, all streams coming out of a demuxer will be grouped in a
|
|
DecodeGroup.
|
|
|
|
All new source pads created on that demuxer after it has emitted the
|
|
'no-more-pads' signal will be put in another DecodeGroup.
|
|
|
|
Only one decodegroup can be active at any given time. If a new decodegroup is
|
|
created while another one exists, that decodegroup will be set as blocking until
|
|
the existing one has drained.
|
|
|
|
|
|
|
|
DecodeGroup
|
|
-----------
|
|
|
|
Description:
|
|
|
|
Streams belonging to the same group/chain of a media file.
|
|
|
|
* Contents
|
|
|
|
The DecodeGroup contains:
|
|
|
|
_ a GstMultiQueue to which all streams of a the media group are connected.
|
|
|
|
_ the eventual decoders which are autoplugged in order to produce the
|
|
requested target pads.
|
|
|
|
* Proper group draining
|
|
|
|
The DecodeGroup takes care that all the streams in the group are completely
|
|
drained (EOS has come through all source ghost pads).
|
|
|
|
* Pre-roll and block
|
|
|
|
The DecodeGroup has a global blocking feature. If enabled, all the ghosted
|
|
source pads for that group will be blocked.
|
|
|
|
A method is available to unblock all blocked pads for that group.
|
|
|
|
|
|
|
|
GstMultiQueue
|
|
-------------
|
|
|
|
Description:
|
|
|
|
Multiple input-output data queue
|
|
|
|
The GstMultiQueue achieves the same functionnality as GstQueue, with a few
|
|
differences:
|
|
|
|
* Multiple streams handling.
|
|
|
|
The element handles queueing data on more than one stream at once. To
|
|
achieve such a feature it has request sink pads (sink_%d) and 'sometimes' src
|
|
pads (src_%d).
|
|
|
|
When requesting a given sinkpad, the associated srcpad for that stream will
|
|
be created. Ex: requesting sink_1 will generate src_1.
|
|
|
|
|
|
* Non-starvation on multiple streams.
|
|
|
|
If more than one stream is used with the element, the streams' queues will
|
|
be dynamically grown (up to a limit), in order to ensure that no stream is
|
|
risking data starvation. This guarantees that at any given time there are at
|
|
least N bytes queued and available for each individual stream.
|
|
|
|
If an EOS event comes through a srcpad, the associated queue should be
|
|
considered as 'not-empty' in the queue-size-growing algorithm.
|
|
|
|
|
|
* Non-linked srcpads graceful handling.
|
|
|
|
A GstTask is started for all srcpads when going to GST_STATE_PAUSED.
|
|
|
|
The task are blocking against a GCondition which will be fired in two
|
|
different cases:
|
|
|
|
_ When the associated queue has received a buffer.
|
|
|
|
_ When the associated queue was previously declared as 'not-linked' and the
|
|
first buffer of the queue is scheduled to be pushed synchronously in
|
|
relation to the order in which it arrived globally in the element (see
|
|
'Synchronous data pushing' below).
|
|
|
|
When woken up by the GCondition, the GstTask will try to push the next
|
|
GstBuffer/GstEvent on the queue. If pushing the GstBuffer/GstEvent returns
|
|
GST_FLOW_NOT_LINKED, then the associated queue is marked as 'not-linked'. If
|
|
pushing the GstBuffer/GstEvent succeeded the queue will no longer be marked as
|
|
'not-linked'.
|
|
|
|
If pushing on all srcpads returns GstFlowReturn different from GST_FLOW_OK,
|
|
then all the srcpads' tasks are stopped and subsequent pushes on sinkpads will
|
|
return GST_FLOW_NOT_LINKED.
|
|
|
|
* Synchronous data pushing for non-linked pads.
|
|
|
|
In order to better support dynamic switching between streams, the multiqueue
|
|
(unlike the current GStreamer queue) continues to push buffers on non-linked
|
|
pads rather than shutting down.
|
|
|
|
In addition, to prevent a non-linked stream from very quickly consuming all
|
|
available buffers and thus 'racing ahead' of the other streams, the element
|
|
must ensure that buffers and inlined events for a non-linked stream are pushed
|
|
in the same order as they were received, relative to the other streams
|
|
controlled by the element. This means that a buffer cannot be pushed to a
|
|
non-linked pad any sooner than buffers in any other stream which were received
|
|
before it.
|
|
|
|
|
|
=====================================
|
|
Parsers, decoders and auto-plugging
|
|
=====================================
|
|
|
|
This section has DRAFT status.
|
|
|
|
Some media formats come in different "flavours" or "stream formats". These
|
|
formats differ in the way the setup data and media data is signalled and/or
|
|
packaged. An example for this is H.264 video, where there is a bytestream
|
|
format (with codec setup data signalled inline and units prefixed by a sync
|
|
code and packet length information) and a "raw" format where codec setup
|
|
data is signalled out of band (via the caps) and the chunking is implicit
|
|
in the way the buffers were muxed into a container, to mention just two of
|
|
the possible variants.
|
|
|
|
Especially on embedded platforms it is common that decoders can only
|
|
handle one particular stream format, and not all of them.
|
|
|
|
Where there are multiple stream formats, parsers are usually expected
|
|
to be able to convert between the different formats. This will, if
|
|
implemented correctly, work as expected in a static pipeline such as
|
|
|
|
... ! parser ! decoder ! sink
|
|
|
|
where the parser can query the decoder's capabilities even before
|
|
processing the first piece of data, and configure itself to convert
|
|
accordingly, if conversion is needed at all.
|
|
|
|
In an auto-plugging context this is not so straight-forward though,
|
|
because elements are plugged incrementally and not before the previous
|
|
element has processes some data and decided what it will output exactly
|
|
(unless the template caps are completely fixed, then it can continue
|
|
right away, this is not always the case here though, see below). A
|
|
parser will thus have to decide on *some* output format so auto-plugging
|
|
can continue. It doesn't know anything about the available decoders and
|
|
their capabilities though, so it's possible that it will choose a format
|
|
that is not supported by any of the available decoders, or by the preferred
|
|
decoder.
|
|
|
|
If the parser had sufficiently concise but fixed source pad template caps,
|
|
decodebin could continue to plug a decoder right away, allowing the
|
|
parser to configure itself in the same way as it would with a static
|
|
pipeline. This is not an option, unfortunately, because often the
|
|
parser needs to process some data to determine e.g. the format's profile or
|
|
other stream properties (resolution, sample rate, channel configuration, etc.),
|
|
and there may be different decoders for different profiles (e.g. DSP codec
|
|
for baseline profile, and software fallback for main/high profile; or a DSP
|
|
codec only supporting certain resolutions, with a software fallback for
|
|
unusual resolutions). So if decodebin just plugged the most highest-ranking
|
|
decoder, that decoder might not be be able to handle the actual stream later
|
|
on, which would yield in an error (this is a data flow error then which would
|
|
be hard to intercept and avoid in decodebin). In other words, we can't solve
|
|
this issue by plugging a decoder right away with the parser.
|
|
|
|
So decodebin need to communicate to the parser the set of available decoder
|
|
caps (which would contain the relevant capabilities/restrictions such as
|
|
supported profiles, resolutions, etc.), after the usual "autoplug-*" signal
|
|
filtering/sorting of course.
|
|
|
|
This could be done in multiple ways, e.g.
|
|
|
|
- plug a capsfilter element right after the parser, and construct
|
|
a set of filter caps from the list of available decoders (one
|
|
could append at the end just the name(s) of the caps structures
|
|
from the parser pad template caps to function as an 'ANY other'
|
|
caps equivalent). This would let the parser negotiate to a
|
|
supported stream format in the same way as with the static
|
|
pipeline mentioned above, but of course incur some overhead
|
|
through the additional capsfilter element.
|
|
|
|
- one could add a filter-caps equivalent property to the parsers
|
|
(and/or GstBaseParse class) (e.g. "prefered-caps" or so).
|
|
|
|
- one could add some kind of "fixate-caps" or "fixate-format"
|
|
signal to such parsers
|
|
|
|
Alternatively, one could simply make all decoders incorporate parsers, so
|
|
that always all formats are supported. This is problematic for other reasons
|
|
though (e.g. we would not be able to detect the profile in all cases then
|
|
before plugging a decoder, which would make it hard to just play the audio
|
|
part of a stream and not the video if a suitable decoder was missing, for
|
|
example).
|
|
|
|
Additional considerations: the same problem exists with sinks that support
|
|
non-raw formats. Consider, for example, an audio sink that accepts DTS audio,
|
|
but only the 14-bit variant, not the 16-bit variant (or only native endiannes).
|
|
Ideally dcaparse would convert into the required stream format here.
|