mirror of
https://gitlab.freedesktop.org/gstreamer/gstreamer.git
synced 2024-12-12 11:26:39 +00:00
656baad90f
Update existing documentation for various behaviours. Add new information on how selection "upstream" of decodebin3 happens. Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/1905>
768 lines
30 KiB
Markdown
768 lines
30 KiB
Markdown
# Stream selection
|
|
|
|
This document describes the events and objects involved in stream selection in
|
|
GStreamer pipelines, elements and applications
|
|
|
|
## History / Availability
|
|
|
|
Since 1.10:
|
|
* `GstStream` and `GstStreamCollection`
|
|
* `enum GstStreamType`
|
|
* `GST_EVENT_SELECT_STREAMS`, `GST_EVENT_STREAM_COLLECTION`,
|
|
`GST_MESSAGE_STREAMS_SELECTED`, `GST_MESSAGE_STREAM_COLLECTION`
|
|
* `GstStream` present in `GST_EVENT_STREAM_START`
|
|
* `GST_BIN_FLAG_STREAMS_AWARE`
|
|
|
|
Since 1.22:
|
|
* `gst_query_new_selectable()`
|
|
* `gst_query_set_selectable()`
|
|
* `gst_query_parse_selectable()`
|
|
|
|
Initial version: June 2015 (Edward Hervey)
|
|
|
|
Last reviewed: March 2022 (Edward Hervey)
|
|
|
|
|
|
## Background
|
|
|
|
This new API is *intended* to address the use cases described in this section:
|
|
|
|
1) As a user/app I want an overview and control of the media streams that can be
|
|
configured within a pipeline for processing, even when some streams are
|
|
mutually exclusive or logical constructs only.
|
|
|
|
2) The user/app can disable entirely streams it's not interested in so they
|
|
don't occupy memory or processing power - discarded as early as possible in
|
|
the pipeline. The user/app can also (re-)enable them at a later time.
|
|
|
|
3) If the set of possible stream configurations is changing, the user/app should
|
|
be aware of the pending change and be able to make configuration choices for
|
|
the new set of streams, as well as possibly still reconfiguring the old set
|
|
|
|
4) Elements that have some other internal mechanism for triggering stream
|
|
selections (DVD, or maybe some scripted playback playlist) should be able to
|
|
trigger 'selection' of some particular stream.
|
|
|
|
5) Indicate known relationships between streams - for example that 2 separate
|
|
video feeds represent the 2 views of a stereoscopic view, or that certain
|
|
streams are mutually exclusive.
|
|
|
|
> Note: the streams that are "available" are not automatically the ones active,
|
|
> or present in the pipeline as pads. Think HLS/DASH alternate streams.
|
|
|
|
|
|
## Example use cases
|
|
|
|
1. Playing an MPEG-TS multi-program stream, we want to tell the app that there
|
|
are multiple programs that could be extracted from the incoming
|
|
feed.
|
|
|
|
Furthermore, we want to provide a mechanism for the app to select which
|
|
program(s) to decode, and once that is known to further tell the app which
|
|
elementary streams are then available within those program(s) so the app/user
|
|
can choose which audio track(s) to decode and/or use.
|
|
|
|
|
|
2. A new PMT arrives for an MPEG-TS stream, due to a codec or channel
|
|
change. The pipeline will need to reconfigure to play the desired streams
|
|
from new program. Equally, there may be multiple seconds of content buffered
|
|
from the old program and it should still be possible to switch (for example)
|
|
subtitle tracks responsively in the draining out data, as well as selecting
|
|
which subs track to play from the new feed.
|
|
|
|
This same scenario applies when doing gapless transition to a new source
|
|
file/URL, except that likely the element providing the list of streams also
|
|
changes as a new demuxer is installed.
|
|
|
|
|
|
3. When playing a multi-angle DVD, the DVD Virtual Machine needs to extract 1
|
|
angle from the data for presentation. It can publish the available angles as
|
|
logical streams, even though only one stream can be chosen.
|
|
|
|
|
|
4. When playing a DVD, the user can make stream selections from the DVD menu to
|
|
choose audio or sub-picture tracks, or the DVD VM can trigger automatic
|
|
selections.
|
|
|
|
In addition, the player UI should be able to show which audio/subtitle tracks
|
|
are available and allow direct selection in a GUI the same as for normal
|
|
files with subtitle tracks in them.
|
|
|
|
|
|
5. Playing a SCHC (3DTV) feed, where one view is MPEG-2 and the other is H.264
|
|
and they should be combined for 3D presentation, or not bother decoding 1
|
|
stream if displaying 2D. (bug
|
|
https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/issues/118)
|
|
|
|
|
|
6. Playing an adaptive stream (HLS or DASH), the user can know the various
|
|
streams it can select, and only the required streams will be downloaded. This
|
|
is independent of the bitrate selection mechanism.
|
|
|
|
|
|
## Design Overview
|
|
|
|
Stream selection in GStreamer is implemented in several parts:
|
|
|
|
1. Objects describing streams : `GstStream`
|
|
2. Objects describing a collection of streams : `GstStreamCollection`
|
|
3. Events from the app allowing selection and activation of some streams:
|
|
`GST_EVENT_SELECT_STREAMS`
|
|
4. Messages informing the user/application about the available streams and
|
|
current status: `GST_MESSAGE_STREAM_COLLECTION` and
|
|
`GST_MESSAGE_STREAMS_SELECTED`
|
|
5. A field on `GST_EVENT_SELECT_STREAMS` to inform downstream elements that a
|
|
certain element can handle the stream selection itself.
|
|
|
|
## GstStream objects
|
|
|
|
API:
|
|
|
|
``` c
|
|
typedef enum {
|
|
GST_STREAM_TYPE_UNKNOWN = 1 << 0,
|
|
GST_STREAM_TYPE_AUDIO = 1 << 1,
|
|
GST_STREAM_TYPE_VIDEO = 1 << 2,
|
|
GST_STREAM_TYPE_CONTAINER = 1 << 3,
|
|
GST_STREAM_TYPE_TEXT = 1 << 4
|
|
} GstStreamType;
|
|
|
|
struct {
|
|
const gchar *stream_id;
|
|
} GstStream;
|
|
|
|
GstStream *gst_stream_new(...);
|
|
|
|
GstStream *gst_stream_get_*(...);
|
|
|
|
GstStream *gst_stream_set_*(...);
|
|
|
|
GstStream *gst_event_set_stream(GstEvent *stream_start, GstStream *stream)
|
|
|
|
void gst_event_parse_stream (GstEvent * event, GstStream ** stream);
|
|
```
|
|
|
|
`GstStream` objects are a high-level convenience object containing information
|
|
regarding a possible data stream that can be exposed by GStreamer elements.
|
|
|
|
They are mostly the aggregation of information present in other GStreamer
|
|
components (`STREAM_START`, `CAPS`, `TAGS` events) but are not tied to the
|
|
presence of a `GstPad`, and for some use-cases provide information that the
|
|
existing components don't provide.
|
|
|
|
|
|
The various properties (available via setters/getters) of a `GstStream` object
|
|
are:
|
|
- stream_id (from the `STREAM_START` event)
|
|
- flags (from the `STREAM_START` event)
|
|
- caps
|
|
- tags
|
|
- type (high-level type of stream: Audio, Video, Container,...)
|
|
|
|
|
|
`GstStream` objects can be subclassed so that they can be re-used by elements
|
|
already using the notion of stream (which is common for example in demuxers).
|
|
|
|
**Note**: While the intent of the design was indeed to allow providers to
|
|
subclass `GstStream`, it turns out it is better to *not* subclass it. Stream
|
|
providers should instead have separate structures to track their internal
|
|
properties of streams, and use a separate `GstStream` for storing information
|
|
they want to share.
|
|
|
|
|
|
Elements that create `GstStream` should also set it on the
|
|
`GST_EVENT_STREAM_START` event of the relevant pad. This helps downstream
|
|
elements have all information in one location.
|
|
|
|
|
|
## Exposing collections of streams
|
|
|
|
API:
|
|
|
|
``` c
|
|
struct GstStreamCollection;
|
|
|
|
gst_stream_collection_new(...);
|
|
gst_stream_collection_add_stream(...);
|
|
gst_stream_collection_get_size(...);
|
|
gst_stream_collection_get_stream(...);
|
|
|
|
GST_MESSAGE_STREAM_COLLECTION
|
|
gst_message_new_stream_collection(...)
|
|
gst_message_parse_stream_collection(...)
|
|
|
|
GST_EVENT_STREAM_COLLECTION
|
|
gst_event_new_stream_collection(...)
|
|
gst_event_parse_stream_collection(...)
|
|
```
|
|
|
|
Elements that create new streams (such as demuxers) or can create new streams
|
|
(like the HLS/DASH alternative streams) can list the streams they can make
|
|
available with the `GstStreamCollection` object.
|
|
|
|
Other elements that might generate `GstStreamCollections` are the DVD-VM, which
|
|
handles internal switching of tracks, or `parsebin` and `decodebin3` when it
|
|
aggregates and presents multiple internal stream sources as a single
|
|
configurable collection.
|
|
|
|
|
|
The `GstStreamCollection` object is a flat listing of `GstStream` objects.
|
|
|
|
The various properties of a `GstStreamCollection` are:
|
|
- 'identifier'
|
|
- the identifier of the collection (unique name)
|
|
- Generated from the 'upstream stream id' (or stream ids, plural)
|
|
|
|
- the list of `GstStreams` in the collection.
|
|
|
|
- **Not implemented** : Flags
|
|
|
|
For now, the only flag is `INFORMATIONAL` - used by container parsers to
|
|
publish information about detected streams without allowing selection of the
|
|
streams. *Solving this might be possible through different means and needs
|
|
more investigation.*
|
|
|
|
- **Not implemented yet** : The relationship between the various streams
|
|
|
|
This specifies which streams are exclusive (can not be selected at the same
|
|
time), are related (such as `LINKED_VIEW` or `ENHANCEMENT`), or need to be
|
|
selected together. *This requires more investigation*
|
|
|
|
|
|
An element will inform outside components about that collection of streams via:
|
|
|
|
* a `GST_MESSAGE_STREAM_COLLECTION` message on the bus.
|
|
* a `GST_EVENT_STREAM_COLLECTION` on each source pads.
|
|
|
|
Applications and container bin elements can listen and collect the various
|
|
stream collections to know the full range of streams available within a
|
|
bin/pipeline.
|
|
|
|
|
|
Once posted on the bus, a `GstStreamCollection` is immutable. It is updated by
|
|
subsequent messages with a matching identifier.
|
|
|
|
|
|
If the element that provided the collection goes away, there is no way to know
|
|
that the streams are no longer valid (without having the user/app track that
|
|
element). The exception to that is if the bin containing that element (such as
|
|
`parsebin` or `decodebin3`) informs that the next collection is a replacement of
|
|
the former one.
|
|
|
|
The mutual exclusion and relationship lists use stream-ids rather than
|
|
`GstStream` references in order to avoid circular referencing problems.
|
|
|
|
|
|
### Usage from elements
|
|
|
|
When a demuxer knows the list of streams it can expose, it creates a new
|
|
`GstStream` for each stream it can provide with the appropriate information
|
|
(stream id, flag, tags, caps, ...).
|
|
|
|
The demuxer then creates a `GstStreamCollection` object in which it will put the
|
|
list of `GstStream` it can expose. That collection is then both posted on the
|
|
bus (via a `GST_MESSAGE_COLLECTION`) and on each pad (via a
|
|
`GST_EVENT_STREAM_COLLECTION`).
|
|
|
|
That new collection must be posted on the bus *before* the changes are made
|
|
available. i.e. before pads corresponding to that selection are added/removed.
|
|
|
|
If an element can handle selecting a sub-set of those streams (for example a
|
|
HLS/DASH demuxer which only download the required streams), it can inform
|
|
downstream elements by answer the `GST_QUERY_SELECTABLE` with `TRUE`. It should
|
|
therefore be able to handle `GST_EVENT_SELECT_STREAMS` at any point in time
|
|
(including while posting the initial collection on the bus).
|
|
|
|
In order to be backwards-compatible and support elements that don't create
|
|
streams/collection yet, the `parsebin` element (used by `decodebin3`) will
|
|
automatically create the stream and collection if not provided by the elements
|
|
within.
|
|
|
|
|
|
### Usage from application
|
|
|
|
Applications can know what streams are available by listening to the
|
|
`GST_MESSAGE_STREAM_COLLECTION` messages posted on the bus.
|
|
|
|
The application can list the available streams per-type (such as all the audio
|
|
streams, or all the video streams) by iterating the streams available in the
|
|
collection by `GST_STREAM_TYPE`.
|
|
|
|
The application will also be able to use these stream information to decide
|
|
which streams should be activated or not (see the stream selection event below).
|
|
|
|
|
|
### Backwards compatibility
|
|
|
|
Not all demuxers will create the various `GstStream` and `GstStreamCollection`
|
|
objects. In order to remain backwards compatible, a parent bin (parsebin in
|
|
decodebin3) will create the `GstStream` and `GstStreamCollection` based on the
|
|
pads being added/removed from an element.
|
|
|
|
This allows providing stream listing/selection for any demuxer-like element even
|
|
if it doesn't implement the `GstStreamCollection` usage.
|
|
|
|
|
|
## Adding and removing pads dynamically
|
|
|
|
API:
|
|
|
|
``` c
|
|
GST_BIN_FLAG_STREAMS_AWARE
|
|
```
|
|
|
|
If an element is within a `GstBin` with the `GST_BIN_FLAG_STREAMS_AWARE` set, it
|
|
can add and remove streams (and corresponding source `GstPad`) dynamically. It
|
|
does **not** have to (re)add all pads of the new stream set, emit `no-more-pads`
|
|
and then remove olds source pads.
|
|
|
|
This method is more efficient than the legacy behaviour, but is opt-in. Several
|
|
elements (`playbin3`, `decodebin3`, `urisourcebin` and more) have this flag set
|
|
by default. Application developers creating custom `GstBin` can set the flag to
|
|
make compatible elements add/remove pads dynamically.
|
|
|
|
|
|
## Stream selection event
|
|
|
|
API:
|
|
|
|
``` c
|
|
GST_EVENT_SELECT_STREAMS
|
|
|
|
gst_event_new_select_streams(...)
|
|
gst_event_parse_select_streams(...)
|
|
```
|
|
|
|
Stream selection events are generated by the application and sent into the
|
|
pipeline to configure the streams.
|
|
|
|
The event carries:
|
|
|
|
* List of `GstStreams` to activate
|
|
* a subset of the `GstStreamCollection`
|
|
|
|
* **Not implemented** List of `GstStreams` to be kept discarded
|
|
* a subset of streams for which hot-swapping will not be desired, allowing
|
|
elements (such as decodebin3, demuxers, ...) to not parse or buffer those
|
|
streams at all. *This will be solved in a different fashion.*
|
|
|
|
|
|
### Usage from application
|
|
|
|
There are two use-cases where an application needs to specify in a generic
|
|
fashion which streams it wants in output:
|
|
|
|
1. When there are several present streams of which it only wants a subset (such
|
|
as one audio, one video and one subtitle stream). Those streams are demuxed
|
|
and present in the pipeline.
|
|
|
|
2. When the stream the user wants require some element to undertake some action
|
|
to expose that stream in the pipeline (such as DASH/HLS alternative streams).
|
|
|
|
|
|
From the point of view of the application, those two use-cases are treated
|
|
identically. The streams are all available through the `GstStreamCollection`
|
|
posted on the bus, and the applications selects a subset.
|
|
|
|
The application can select the streams it wants by creating a
|
|
`GST_EVENT_SELECT_STREAMS` event with the list of stream-id of the streams it
|
|
wants. That event is then sent on the pipeline, eventually traveling all the way
|
|
upstream from each sink.
|
|
|
|
In some cases, selecting one stream may trigger the availability of other
|
|
dependent streams, resulting in new `GstStreamCollection` messages. This can
|
|
happen in the case where choosing a different DVB channel would create a new
|
|
single-program collection.
|
|
|
|
|
|
### Usage in elements
|
|
|
|
Elements that receive the `GST_EVENT_SELECT_STREAMS` event and that can
|
|
activate/deactivate streams need to look at the list of stream-id contained in
|
|
the event and decide if they need to do some action.
|
|
|
|
In the standard demuxer case (demuxing and exposing all streams), there is
|
|
nothing to do by default (see below).
|
|
|
|
In `decodebin3`, activating or deactivating streams is taken care of by linking
|
|
only the streams present in the event to decoders and output ghostpad.
|
|
|
|
In the case of elements that can expose alternate streams that are not present
|
|
in the pipeline as pads, they will take the appropriate action to add/remove
|
|
those streams.
|
|
|
|
Containers that receive the event (via `GstElement::send_event()`) should pass
|
|
it to any elements with no downstream peers, so that streams can be configured
|
|
during pre-roll before a pipeline is completely linked down to sinks.
|
|
|
|
|
|
#### Which elements should handle stream-selection ?
|
|
|
|
By default demuxers *must* process and expose all streams contained within the
|
|
multiplex. This is to allow downstream stream switching to happen as fast as
|
|
possible.
|
|
|
|
The only exception to this is if extra processing (such as network download) can
|
|
be avoided upstream as is the case with HLS/DASH sources. Those elements inform
|
|
other downstream elements of this capability by replying to the
|
|
`GST_QUERY_SELECTABLE`. They are also responsible for selecting a default
|
|
selection of streams to output (one of each available `GstStreamType`) if no
|
|
`GST_EVENT_SELECT_STREAMS` was received before.
|
|
|
|
Furthermore, for all elements, stream-selection can only happen if the `GstBin`
|
|
they are in has the `GST_BIN_FLAG_STREAMS_AWARE` flag set.
|
|
|
|
Later design/API will be researched to allow use-cases where regular demuxers
|
|
could completely disable outputting some of the contained streams (to allow more
|
|
efficient I/O and memory usage).
|
|
|
|
|
|
## decodebin3 usage and example
|
|
|
|
This is an example of how decodebin3 works by using the above-mentioned
|
|
objects/events/messages.
|
|
|
|
For clarity/completeness, we will consider a MPEG-TS stream that has multiple
|
|
audio streams. Furthermore that stream might have changes at some point
|
|
(switching video codec, or adding/removing audio streams).
|
|
|
|
|
|
### Initial differences with `decodebin2`
|
|
|
|
`decodebin3` is different, compared to `decodebin2`, in the sense that, by
|
|
default:
|
|
|
|
* it will only expose as output ghost source pads one stream of each type (one
|
|
audio, one video, ..).
|
|
* It will only decode the exposed streams
|
|
|
|
|
|
The `multiqueue` element is still used and takes in all elementary (non-decoded)
|
|
streams. If parsers are needed/present they are placed before the
|
|
multiqueue. This is needed in order for multiqueue to work only with packetized
|
|
and properly timestamped streams.
|
|
|
|
Note that the whole typefinding of streams, and optional depayloading, demuxing
|
|
and parsing are done in a new `parsebin` element.
|
|
|
|
Just like the current implementation, demuxers will expose all streams present
|
|
within a program as source pads. They will connect to parsers and multiqueue.
|
|
|
|
Initial setup. 1 video stream, 2 audio streams.
|
|
|
|
```
|
|
+---------------------+
|
|
| parsebin |
|
|
| --------- | +-------------+
|
|
| | demux |--[parser]-+-| multiqueue |--[videodec]---[
|
|
]-+-| |--[parser]-+-| |
|
|
| | |--[parser]-+-| |--[audiodec]---[
|
|
| --------- | +-------------+
|
|
+---------------------+
|
|
```
|
|
|
|
### Generating GstStreamCollection
|
|
|
|
When parsing the initial PAT/PMT, the demuxer will:
|
|
|
|
1. create the various `GstStream` objects for each stream.
|
|
|
|
2. create the `GstStreamCollection` for that initial PMT
|
|
|
|
3. post the `GST_MESSAGE_STREAM_COLLECTION`. Decodebin will intercept that
|
|
message and know what the demuxer will be exposing.
|
|
|
|
4. The demuxer creates the various pads and sends the corresponding
|
|
`STREAM_START` event (with the same stream-id as the corresponding
|
|
`GstStream` objects), `CAPS` event, and `TAGS` event.
|
|
|
|
* `parsebin` will add all relevant parsers and expose those streams.
|
|
|
|
* `decodebin3` will be able to correlate, based on `STREAM_START` event
|
|
stream-id, what pad corresponds to which stream. It links each stream from
|
|
parsebin to multiqueue.
|
|
|
|
* `decodebin3` knows all the streams that will be available. Since by default
|
|
it is configured to only expose a stream of each type, it will pick a stream
|
|
of each for which it will complete the auto-plugging (finding a decoder and
|
|
then exposing that stream as a source ghostpad.
|
|
|
|
> Note: If the demuxer doesn't create/post the `GstStreamCollection`, `parsebin`
|
|
> will create it on itself, as explained in the previous sections above.
|
|
|
|
|
|
### Changing the active selection from the application
|
|
|
|
The user wants to change the audio track. The application received the
|
|
`GST_MESSAGE_STREAM_COLLECTION` containing the list of available streams. For
|
|
clarity, we will assume those stream-ids are `"video-main"`, `"audio-english"`
|
|
and `"audio-french"`.
|
|
|
|
|
|
The user prefers to use the french soundtrack (which it knows based on the
|
|
language tag contained in the `GstStream` objects).
|
|
|
|
The application will create and send a `GST_EVENT_SELECT_STREAM` event
|
|
containing the list of streams: `"video-main", "audio-french"`.
|
|
|
|
That event gets sent on the pipeline, the sinks send it upstream and eventually
|
|
reach `decodebin3`.
|
|
|
|
`decodebin3` compares:
|
|
|
|
* The currently active selection (`"video-main", "audio-english"`)
|
|
* The available stream collection (`"video-main", "audio-english",
|
|
"audio-french"`)
|
|
* The list of streams in the event (`"video-main", "audio-french"`)
|
|
|
|
`decodebin3` determines that no change is required for `"video-main"`, but sees
|
|
that it needs to deactivate `"audio-english"` and activate `"audio-french"`.
|
|
|
|
|
|
It unlinks the `multiqueue` source pad connected to the current audio
|
|
decoder. Then it queries that audio decoder, using the `GST_QUERY_ACCEPT_CAPS`,
|
|
whether it can accept as-is the caps from the `"audio-french"` stream.
|
|
|
|
1. If it does, the multiqueue source pad corresponding to `"audio-french"` is
|
|
linked to the decoder.
|
|
|
|
2. If it does not, the existing audio decoder is removed, a new decoder is
|
|
selected (like during initial auto-plugging), and replaces the old audio
|
|
decoder element.
|
|
|
|
The newly selected stream gets decoded and output through the same pad as the
|
|
previous audio stream.
|
|
|
|
|
|
*Note:* The default behavior is to only expose one stream of each type. But
|
|
nothing prevents `decodebin3` from outputting more/less of each type if the
|
|
`GST_EVENT_SELECT_STREAM` event specifies that. This allows covering more
|
|
use-case than the simple playback one. Such examples could be :
|
|
* Wanting just a video stream or just an audio stream
|
|
* Wanting all decoded streams
|
|
* Wanting all audio streams
|
|
...
|
|
|
|
|
|
### Changes of stream collection
|
|
|
|
At some point in time, a PMT change happens. Let's assume a change in
|
|
video-codec and/or PID.
|
|
|
|
The demuxer creates a new `GstStream` for the changed/new stream, creates a new
|
|
`GstStreamCollection` for the updated PMT and posts it.
|
|
|
|
|
|
`decodebin3` sees the new `GstStreamCollection` message.
|
|
|
|
The demuxer (and `parsebin`) then adds and removes pads:
|
|
|
|
1. `decodebin3` will match the new pads to `GstStream` in the *new*
|
|
`GstStreamCollection` the [same way it did](#generating-gststreamcollection)
|
|
for the initial pads previously.
|
|
|
|
2. `decodebin3` will see whether the new stream can re-use a `multiqueue` slot
|
|
used by a stream of the same type no longer present (it compares the old
|
|
collection to the new collection). In this case, `decodebin3` sees that the
|
|
new video stream can re-use the same slot as the previous video stream.
|
|
|
|
3. If the new stream is going to be active by default (in this case it does
|
|
because we are replacing the only video stream, which was active), it will
|
|
check whether the caps are compatible with the existing video decoder (in the
|
|
same way it was done for the [audio decoder
|
|
switch](#changing-the-active-selection-from-the-application)).
|
|
|
|
|
|
Finally, the stream that switched will be decoded and output through the same
|
|
pad as the previous video stream in a gapless fashion.
|
|
|
|
|
|
### Further examples
|
|
|
|
##### DASH/HLS alternates (selection handled upstream of decodebin3)
|
|
|
|
There is a main (multi-bitrate or not) stream with audio and video interleaved
|
|
in MPEG-TS. The manifest also indicates the presence of alternate language
|
|
audio-only streams.
|
|
|
|
* (main) variant stream : `video`, `audio-eng`
|
|
* alternate rendition streams : `audio-kor`, `audio-fre`
|
|
|
|
A new element, `hlsdemux2`, used within `urisourcebin`, exposes one collection
|
|
containing all individual elementary streams (regardless of bitrates). This is
|
|
based on the contents of the manifest:
|
|
|
|
* `[ video , audio-eng, audio-kor, audio-fre ]`
|
|
|
|
Similar to other demuxers, it will post the collection on the bus, and will
|
|
add/remove pads dynamically based on `GST_EVENT_SELECT_STREAMS` (or make a
|
|
default choice if no event was received).
|
|
|
|
Since it can offer a selection of "elementary" streams, it will internally
|
|
demux, parse and buffer the requested streams.
|
|
|
|
If ever it receives the `GST_QUERY_SELECTABLE`, it will set the `selectable`
|
|
field of that query to TRUE. This informs `decodebin3` that it should handle all
|
|
incoming streams and not do any selection (it is done upstream).
|
|
|
|
```
|
|
+-----------+ + decodebin3 -----------------------------+
|
|
| hlsdemux2 | | |
|
|
| | | +------------+ |
|
|
| +- `video` -+-[parsebin]--+ multiqueue +--[videodec]--[
|
|
| +- `audio-eng` -+-[parsebin]--+------------+--[audiodec]--[
|
|
| | | |
|
|
+-----------+ +-----------------------------------------+
|
|
```
|
|
|
|
The user might want to use the korean audio track instead of the default english
|
|
one.
|
|
|
|
```
|
|
=> SELECT_STREAMS ("video", "kor")
|
|
```
|
|
|
|
0) When `decodebin3` received `GST_EVENT_STREAM_START` on the *initial* incoming
|
|
streams, it sent a `GST_QUERY_SELECTABLE` which `hlsdemux2` answered
|
|
succesfully. `decodebin3` therefore knows that the upstream of that stream can
|
|
handle stream-selection itself.
|
|
|
|
1) `decodebin3` receives the new `SELECT_STREAMS` event, and knows that upstream
|
|
can handle stream selection and therefore forwards it upstream.
|
|
|
|
2) `hlsdemux2` receives the event, and prepares the `audio-kor` backing stream
|
|
to be downloaded and returns. `decodebin3` also returns the event as is and
|
|
doesn't do any selection itself.
|
|
|
|
3) Once `hlsdemux2` has downloaded enough from the target `audio-kor` stream,
|
|
which it has demuxed, parsed and buffered internally, it will output it to the
|
|
same output pad (replacing `audio-eng`):
|
|
* Send a new `GST_EVENT_STREAM_START` containing the information about
|
|
`audio-kor`
|
|
* Send the various required events (caps, stream-collection,...)
|
|
* Send the buffers for the new stream
|
|
|
|
4) If the incoming stream isn't compatible with the existing `parsebin`,
|
|
`decodebin3` will reset it so that it reconfigures itself.
|
|
|
|
|
|
```
|
|
+-----------+ + decodebin3 -----------------------------+
|
|
| hlsdemux2 | | |
|
|
| | | +------------+ |
|
|
| +- `video` -+-[parsebin]--+ multiqueue +--[videodec]--[
|
|
| +- `audio-kor` -+-[parsebin]--+------------+--[audiodec]--[
|
|
| | | |
|
|
+-----------+ +-----------------------------------------+
|
|
```
|
|
|
|
|
|
#### Multi-program MPEG-TS
|
|
|
|
**NOTE:** Not properly handled yet.
|
|
|
|
Assuming the case of a MPEG-TS stream which contains multiple programs.
|
|
|
|
There would be three "levels" of collection:
|
|
1. The collection of programs presents in the stream
|
|
2. The collection of elementary streams presents in a stream
|
|
3. The collection of streams decodebin can expose
|
|
|
|
Initially `tsdemux` exposes the first program present (default)
|
|
|
|
```
|
|
[ Collection 1 ] [ Collection 2 ] [ Collection 3 ]
|
|
[ (tsdemux) ] [ (tsdemux) ] [ (decodebin) ]
|
|
[ id:Programs ]<-\ [ id:BBC1 ]<-\ [ id:BBC1-decoded ]
|
|
[ upstream:nil ] \-----[ upstream:Programs] \----[ upstream:BBC1 ]
|
|
[ ] [ ] [ ]
|
|
[ "BBC1" (C) ] [ id:"bbcvideo"(V) ] [ id:"bbcvideo"(V)]
|
|
[ "ITV" (C) ] [ id:"bbcaudio"(A) ] [ id:"bbcaudio"(A)]
|
|
[ "NBC" (C) ] [ ] [ ]
|
|
```
|
|
|
|
At some point the user wants to switch to ITV (of which we do not know the
|
|
topology at this point in time. A `SELECT_STREAMS` event is sent with "ITV" in
|
|
it and the pointer to the Collection1.
|
|
|
|
1. The event travels up the pipeline until `tsdemux` receives it and begins the
|
|
switch.
|
|
|
|
2. `tsdemux` publishes a new 'Collection 2a/ITV' and marks 'Collection 2/BBC' as
|
|
replaced.
|
|
|
|
1. App may send a `SELECT_STREAMS` event configuring which demuxer output
|
|
streams should be selected (parsed)
|
|
|
|
3. `tsdemux` adds/removes pads as needed (flushing pads as it removes them?)
|
|
|
|
4. `decodebin3` feeds new pad streams through existing parsers/decoders as
|
|
needed. As data from the new collection arrives out each decoder,
|
|
`decodebin3` sends new `GstStreamCollection` messages to the app so it can
|
|
know that the new streams are now switchable at that level.
|
|
|
|
1. As new `GstStreamCollections` are published, the app may override the
|
|
default decodebin stream selection to expose more/fewer streams. The
|
|
default is to decode and output 1 stream of each type.
|
|
|
|
|
|
Final state:
|
|
|
|
```
|
|
[ Collection 1 ] [ Collection 4 ] [ Collection 5 ]
|
|
[ (tsdemux) ] [ (tsdemux) ] [ (decodebin) ]
|
|
[ id:Programs ]<-\ [ id:ITV ]<-\ [ id:ITV-decoded ]
|
|
[ upstream:nil ] \-----[ upstream:Programs] \----[ upstream:ITV ]
|
|
[ ] [ ] [ ]
|
|
[ "BBC1" (C) ] [ id:"itvvideo"(V) ] [ id:"itvvideo"(V)]
|
|
[ "ITV" (C) ] [ id:"itvaudio"(A) ] [ id:"itvaudio"(A)]
|
|
[ "NBC" (C) ] [ ] [ ]
|
|
```
|
|
|
|
|
|
### TODO
|
|
|
|
- Figure out how to handle *mutual-exclusion* and *relationship* API with
|
|
collection of streams.
|
|
|
|
- ~~Add flags to `GstStreamCollection`~~
|
|
|
|
- Add helper API to figure out whether a collection is a replacement of another
|
|
or a completely new one. This will require a more generic system to know
|
|
whether a certain stream-id is a replacement of another or not.
|
|
|
|
|
|
### Open Questions
|
|
|
|
- ~~Is a `FLUSHING` flag for stream-selection required or not?~~
|
|
|
|
This would make the handler of the `SELECT_STREAMS` event send `FLUSH
|
|
START/STOP` before switching to the other streams. This is tricky when dealing
|
|
where situations where we keep some streams and only switch some others. Do we
|
|
flush all streams? Do we only flush the new streams, potentially resulting in
|
|
delay to fully switch? Furthermore, due to efficient buffering in
|
|
`decodebin3`, the switching time has been minimized extensively, to the point
|
|
where flushing might not bring a noticeable improvement.
|
|
|
|
**Note:** This flushing capability was not needed in the end. Leveraging
|
|
minimal buffering in `decodebin3` and various other elements allows
|
|
*almost-instantaneous* stream switching without this.
|
|
|
|
|
|
- Store the stream collection in bins/pipelines?
|
|
|
|
A Bin/Pipeline could store all active collection internally, so that it could
|
|
be queried later on. This could be useful to then get, on any pipeline, at any
|
|
point in time, the full list of collections available without having to listen
|
|
to all COLLECTION messages on the bus. This would require fixing the "is a
|
|
collection a replacement or not" issue first.
|
|
|
|
|
|
- When switching to new collections, should `decodebin3` make any effort to
|
|
*map* corresponding streams from the old to new PMT - that is, try and stick
|
|
to the *english* language audio track, for example? Alternatively, rely on
|
|
the app to do such smarts with stream-select messages?
|
|
|
|
**Note:** If an application listens synchronously to the collection bus
|
|
messages, it can indeed decide which streams it wants to select. And therefore
|
|
stick to the (new) english audio track.
|