mirrors/gstreamer
mirrors/gstreamer
1
1
Fork 0
mirror of https://gitlab.freedesktop.org/gstreamer/gstreamer.git synced 2024-07-26 08:18:22 +00:00
Code Issues 1 Projects Releases Wiki Activity

design: stream-selection: section breakdown, markup and readability fixes

Browse source
This commit is contained in:
Reynaldo H. Verdejo Pinochet 2016-12-30 00:11:33 -08:00
parent 3e315b6b2e
commit f2929cfdb6
1 changed files with 98 additions and 84 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

182
markdown/design/stream-selection.md
View file

@ -45,7 +45,7 @@ this section:
> the ones active, or present in the pipeline as pads. Think HLS/DASH > the ones active, or present in the pipeline as pads. Think HLS/DASH
> alternate streams. > alternate streams.
Use case examples: ## Example use cases
1) Playing an MPEG-TS multi-program stream, we want to tell the 1) Playing an MPEG-TS multi-program stream, we want to tell the
app that there are multiple programs that could be extracted app that there are multiple programs that could be extracted
@ -89,97 +89,104 @@ FIXME - need some use cases indicating what alternate streams in
## Design Overview ## Design Overview
Stream selection in GStreamer is implemented in several parts: Stream selection in GStreamer is implemented in several parts:
1) Objects describing streams : GstStream 1) Objects describing streams : `GstStream`
2) Objects describing a collection of streams : GstStreamCollection 2) Objects describing a collection of streams : `GstStreamCollection`
3) Events from the app allowing selection and activation of some streams: 3) Events from the app allowing selection and activation of some streams:
GST_EVENT_SELECT_STREAMS `GST_EVENT_SELECT_STREAMS`
4) Messages informing the user/application about the available 4) Messages informing the user/application about the available
streams and current status: streams and current status: `GST_MESSAGE_STREAM_COLLECTION` and
GST_MESSAGE_STREAM_COLLECTION `GST_MESSAGE_STREAMS_SELECTED`
GST_MESSAGE_STREAMS_SELECTED
## GstStream objects ## GstStream objects
* API: GstStream ```
* API: gst_stream_new(..) API:
* API: gst_stream_get_\*(...)
* API: gst_stream_set_\*()
* API: gst_event_set_stream(...)
* API: gst_event_parse_stream(...)
GstStream objects are a high-level convenience object containing GstStream
gst_stream_new(..)
gst_stream_get_\*(...)
gst_stream_set_\*()
gst_event_set_stream(...)
gst_event_parse_stream(...)
```
`GstStream` objects are a high-level convenience object containing
information regarding a possible data stream that can be exposed by information regarding a possible data stream that can be exposed by
GStreamer elements. GStreamer elements.
They are mostly the aggregation of information present in other They are mostly the aggregation of information present in other
GStreamer components (STREAM_START, CAPS, TAGS event) but are not GStreamer components (`STREAM_START`, `CAPS`, `TAGS` events) but are not
tied to the presence of a GstPad, and for some use-cases provide tied to the presence of a `GstPad`, and for some use-cases provide
information that the existing components don't provide. information that the existing components don't provide.
The various properties of a GstStream object are: The various properties of a `GstStream` object are:
- stream_id (from the STREAM_START event) - stream_id (from the `STREAM_START` event)
- flags (from the STREAM_START event) - flags (from the `STREAM_START` event)
- caps - caps
- tags - tags
- type (high-level type of stream: Audio, Video, Container,...) - type (high-level type of stream: Audio, Video, Container,...)
GstStream objects can be subclassed so that they can be re-used by `GstStream` objects can be subclassed so that they can be re-used by
elements already using the notion of stream (which is common for elements already using the notion of stream (which is common for
example in demuxers). example in demuxers).
Elements that create GstStream should also set it on the Elements that create GstStream should also set it on the
GST_EVENT_STREAM_START event of the relevant pad. This helps `GST_EVENT_STREAM_START` event of the relevant pad. This helps
downstream elements to have all information in one location. downstream elements to have all information in one location.
## Exposing collections of streams ## Exposing collections of streams
* API: GstStreamCollection ```
* API: gst_stream_collection_new(...) API:
* API: gst_stream_collection_add_stream(...)
* API: gst_stream_collection_get_size(...) GstStreamCollection
* API: gst_stream_collection_get_stream(...) gst_stream_collection_new(...)
* API: GST_MESSAGE_STREAM_COLLECTION gst_stream_collection_add_stream(...)
* API: gst_message_new_stream_collection(...) gst_stream_collection_get_size(...)
* API: gst_message_parse_stream_collection(...) gst_stream_collection_get_stream(...)
* API: GST_EVENT_STREAM_COLLECTION GST_MESSAGE_STREAM_COLLECTION
* API: gst_event_new_stream_collection(...) gst_message_new_stream_collection(...)
* API: gst_event_parse_stream_collection(...) gst_message_parse_stream_collection(...)
GST_EVENT_STREAM_COLLECTION
gst_event_new_stream_collection(...)
gst_event_parse_stream_collection(...)
```
Elements that create new streams (such as demuxers) or can create Elements that create new streams (such as demuxers) or can create
new streams (like the HLS/DASH alternative streams) can list the new streams (like the HLS/DASH alternative streams) can list the
streams they can make available with the GstStreamCollection object. streams they can make available with the GstStreamCollection object.
Other elements that might generate GstStreamCollections are the Other elements that might generate `GstStreamCollections` are the
DVD-VM, which handles internal switching of tracks, or parsebin and DVD-VM, which handles internal switching of tracks, or parsebin and
decodebin3 when it aggregates and presents multiple internal stream decodebin3 when it aggregates and presents multiple internal stream
sources as a single configurable collection. sources as a single configurable collection.
The GstStreamCollection object is a flat listing of GstStream objects. The `GstStreamCollection` object is a flat listing of `GstStream` objects.
The various properties of a GstStreamCollection are: The various properties of a `GstStreamCollection` are:
- 'identifier' - 'identifier'
- the identifier of the collection (unique name) - the identifier of the collection (unique name)
- Generated from the 'upstream stream id' (or stream ids, plural) - Generated from the 'upstream stream id' (or stream ids, plural)
- the list of GstStreams in the collection. - the list of `GstStreams` in the collection.
- (Not implemented) : Flags - - (Not implemented) : Flags -
For now, the only flag is 'INFORMATIONAL' - used by container parsers to For now, the only flag is `INFORMATIONAL` - used by container parsers to
publish information about detected streams without allowing selection of publish information about detected streams without allowing selection of
the streams. the streams.
- (Not implemented yet) : The relationship between the various streams - (Not implemented yet) : The relationship between the various streams
This specifies which streams are exclusive (can not be selected at the This specifies which streams are exclusive (can not be selected at the
same time), are related (such as LINKED_VIEW or ENHANCEMENT), or need to same time), are related (such as `LINKED_VIEW` or `ENHANCEMENT`), or need to
be selected together. be selected together.
An element will inform outside components about that collection via: An element will inform outside components about that collection via:
* a GST_MESSAGE_STREAM_COLLECTION message on the bus. * a `GST_MESSAGE_STREAM_COLLECTION` message on the bus.
* a GST_EVENT_STREAM_COLLECTION on each source pads. * a `GST_EVENT_STREAM_COLLECTION` on each source pads.
Applications and container bin elements can listen and collect the Applications and container bin elements can listen and collect the
various stream collections to know the full range of streams various stream collections to know the full range of streams
available within a bin/pipeline. available within a bin/pipeline.
Once posted on the bus, a GstStreamCollection is immutable. It is Once posted on the bus, a `GstStreamCollection` is immutable. It is
updated by subsequent messages with a matching identifier. updated by subsequent messages with a matching identifier.
If the element that provided the collection goes away, there is no way If the element that provided the collection goes away, there is no way
@ -189,7 +196,7 @@ containing that element (such as parsebin or decodebin3) informs that
the next collection is a replacement of the former one. the next collection is a replacement of the former one.
The mutual exclusion and relationship lists use stream-ids The mutual exclusion and relationship lists use stream-ids
rather than GstStream references in order to avoid circular rather than `GstStream` references in order to avoid circular
referencing problems. referencing problems.
### Usage from elements ### Usage from elements
@ -200,8 +207,8 @@ appropriate information (stream id, flag, tags, caps, ...).
The demuxer then creates a GstStreamCollection object in which it The demuxer then creates a GstStreamCollection object in which it
will put the list of GstStream it can expose. That collection is will put the list of GstStream it can expose. That collection is
then both posted on the bus (via a GST_MESSAGE_COLLECTION) and on then both posted on the bus (via a `GST_MESSAGE_COLLECTION`) and on
each pad (via a GST_EVENT_STREAM_COLLECTION). each pad (via a `GST_EVENT_STREAM_COLLECTION`).
That new collection must be posted on the bus *before* the changes That new collection must be posted on the bus *before* the changes
are made available. i.e. before pads corresponding to that selection are made available. i.e. before pads corresponding to that selection
@ -214,11 +221,11 @@ decodebin3 will automatically create those if not provided.
### Usage from application ### Usage from application
Applications can know what streams are available by listening to the Applications can know what streams are available by listening to the
GST_MESSAGE_STREAM_COLLECTION messages posted on the bus. `GST_MESSAGE_STREAM_COLLECTION` messages posted on the bus.
The application can list the available streams per-type (such as all The application can list the available streams per-type (such as all
the audio streams, or all the video streams) by iterating the the audio streams, or all the video streams) by iterating the
streams available in the collection by GST_STREAM_TYPE. streams available in the collection by `GST_STREAM_TYPE`.
The application will also be able to use these stream information to The application will also be able to use these stream information to
decide which streams should be activated or not (see the stream decide which streams should be activated or not (see the stream
@ -226,27 +233,31 @@ selection event below).
### Backwards compatibility ### Backwards compatibility
Not all demuxers will create the various GstStream and Not all demuxers will create the various `GstStream` and
GstStreamCollection objects. In order to remain backwards `GstStreamCollection` objects. In order to remain backwards
compatible, a parent bin (parsebin in decodebin3) will create the compatible, a parent bin (parsebin in decodebin3) will create the
GstStream and GstStreamCollection based on the pads being `GstStream` and `GstStreamCollection` based on the pads being
added/removed from an element. added/removed from an element.
This allows providing stream listing/selection for any demuxer-like This allows providing stream listing/selection for any demuxer-like
element even if it doesn't implement the GstStreamCollection usage. element even if it doesn't implement the `GstStreamCollection` usage.
## Stream selection event ## Stream selection event
* API: GST_EVENT_SELECT_STREAMS ```
* API: gst_event_new_select_streams(...) API:
* API: gst_event_parse_select_streams(...)
Stream selection events are generated by the application and GST_EVENT_SELECT_STREAMS
sent into the pipeline to configure the streams. gst_event_new_select_streams(...)
gst_event_parse_select_streams(...)
```
Stream selection events are generated by the application and sent into the
pipeline to configure the streams.
The event carries: The event carries:
* List of GstStreams to activate - a subset of the GstStreamCollection * List of `GstStreams` to activate - a subset of the `GstStreamCollection`
* (Not implemented) - List of GstStreams to be kept discarded - a * (Not implemented) - List of `GstStreams` to be kept discarded - a
subset of streams for which hot-swapping will not be desired, subset of streams for which hot-swapping will not be desired,
allowing elements (such as decodebin3, demuxers, ...) to not parse or allowing elements (such as decodebin3, demuxers, ...) to not parse or
buffer those streams at all. buffer those streams at all.
@ -265,21 +276,21 @@ generic fashion which streams it wants in output:
From the point of view of the application, those two use-cases are From the point of view of the application, those two use-cases are
treated identically. The streams are all available through the treated identically. The streams are all available through the
GstStreamCollection posted on the bus, and it will select a subset. `GstStreamCollection` posted on the bus, and it will select a subset.
The application can select the streams it wants by creating a The application can select the streams it wants by creating a
GST_EVENT_SELECT_STREAMS event with the list of stream-id of the `GST_EVENT_SELECT_STREAMS` event with the list of stream-id of the
streams it wants. That event is then sent on the pipeline, streams it wants. That event is then sent on the pipeline,
eventually traveling all the way upstream from each sink. eventually traveling all the way upstream from each sink.
In some cases, selecting one stream may trigger the availability of In some cases, selecting one stream may trigger the availability of
other dependent streams, resulting in new GstStreamCollection other dependent streams, resulting in new `GstStreamCollection`
messages. This can happen in the case where choosing a different DVB messages. This can happen in the case where choosing a different DVB
channel would create a new single-program collection. channel would create a new single-program collection.
### Usage in elements ### Usage in elements
Elements that receive the GST_EVENT_SELECT_STREAMS event and that Elements that receive the `GST_EVENT_SELECT_STREAMS` event and that
can activate/deactivate streams need to look at the list of can activate/deactivate streams need to look at the list of
stream-id contained in the event and decide if they need to do some stream-id contained in the event and decide if they need to do some
action. action.
@ -347,15 +358,15 @@ Initial setup. 1 video stream, 2 audio streams.
When parsing the initial PAT/PMT, the demuxer will: When parsing the initial PAT/PMT, the demuxer will:
1) create the various GstStream objects for each stream. 1) create the various GstStream objects for each stream.
2) create the GstStreamCollection for that initial PMT 2) create the GstStreamCollection for that initial PMT
3) post the GST_MESSAGE_STREAM_COLLECTION Decodebin will intercept that message 3) post the `GST_MESSAGE_STREAM_COLLECTION` Decodebin will intercept that
and know what the demuxer will be exposing. message and know what the demuxer will be exposing.
4) The demuxer creates the various pads and sends the corresponding 4) The demuxer creates the various pads and sends the corresponding
STREAM_START event (with the same stream-id as the corresponding `STREAM_START` event (with the same stream-id as the corresponding
GstStream objects), CAPS event, and TAGS event. `GstStream` objects), `CAPS` event, and `TAGS` event.
* parsebin will add all relevant parsers and expose those streams. * parsebin will add all relevant parsers and expose those streams.
* Decodebin will be able to correlate, based on STREAM_START event * Decodebin will be able to correlate, based on `STREAM_START` event
stream-id, what pad corresponds to which stream. It links each stream stream-id, what pad corresponds to which stream. It links each stream
from parsebin to multiqueue. from parsebin to multiqueue.
@ -365,21 +376,21 @@ will pick a stream of each for which it will complete the
auto-plugging (finding a decoder and then exposing that stream as a auto-plugging (finding a decoder and then exposing that stream as a
source ghostpad. source ghostpad.
> Note: If the demuxer doesn't create/post the GstStreamCollection, > Note: If the demuxer doesn't create/post the `GstStreamCollection`,
> parsebin will create it on itself, as explained in section 2.3 > parsebin will create it on itself, as explained in section 2.3
> above. > above.
### Changing the active selection from the application ### Changing the active selection from the application
The user wants to change the audio track. The application received The user wants to change the audio track. The application received
the GST_MESSAGE_STREAM_COLLECTION containing the list of available the `GST_MESSAGE_STREAM_COLLECTION` containing the list of available
streams. For clarity, we will assume those stream-ids are streams. For clarity, we will assume those stream-ids are
"video-main", "audio-english" and "audio-french". "video-main", "audio-english" and "audio-french".
The user prefers to use the french soundtrack (which it knows based The user prefers to use the french soundtrack (which it knows based
on the language tag contained in the GstStream objects). on the language tag contained in the `GstStream` objects).
The application will create and send a GST_EVENT_SELECT_STREAM event The application will create and send a `GST_EVENT_SELECT_STREAM` event
containing the list of streams: "video-main", "audio-french". containing the list of streams: "video-main", "audio-french".
That event gets sent on the pipeline, the sinks send it upstream and That event gets sent on the pipeline, the sinks send it upstream and
@ -396,7 +407,7 @@ but sees that it needs to deactivate "audio-english" and activate
"audio-french". "audio-french".
It unlinks the multiqueue source pad connected to the audiodec. Then It unlinks the multiqueue source pad connected to the audiodec. Then
it queries audiodec, using the GST_QUERY_ACCEPT_CAPS, whether it can it queries audiodec, using the `GST_QUERY_ACCEPT_CAPS`, whether it can
accept as-is the caps from the "audio-french" stream. accept as-is the caps from the "audio-french" stream.
1) If it does, the multiqueue source pad corresponding to 1) If it does, the multiqueue source pad corresponding to
"audio-french" is linked to the decoder. "audio-french" is linked to the decoder.
@ -410,7 +421,7 @@ pad as the previous audio stream.
Note: Note:
The default behaviour would be to only expose one stream of each The default behaviour would be to only expose one stream of each
type. But nothing prevents decodebin from outputting more/less of type. But nothing prevents decodebin from outputting more/less of
each type if the GST_EVENT_SELECT_STREAM event specifies that. This each type if the `GST_EVENT_SELECT_STREAM` event specifies that. This
allows covering more use-case than the simple playback one. allows covering more use-case than the simple playback one.
Such examples could be : Such examples could be :
* Wanting just a video stream or just an audio stream * Wanting just a video stream or just an audio stream
@ -423,14 +434,14 @@ Such examples could be :
At some point in time, a PMT change happens. Let's assume a change At some point in time, a PMT change happens. Let's assume a change
in video-codec and/or PID. in video-codec and/or PID.
The demuxer creates a new GstStream for the changed/new stream, The demuxer creates a new `GstStream` for the changed/new stream,
creates a new GstStreamCollection for the updated PMT and posts it. creates a new GstStreamCollection for the updated PMT and posts it.
Decodebin sees the new GstStreamCollection message. Decodebin sees the new `GstStreamCollection` message.
The demuxer (and parsebin) then adds and removes pads. The demuxer (and parsebin) then adds and removes pads.
1) decodebin will match the new pads to GstStream in the "new" 1) decodebin will match the new pads to `GstStream` in the "new"
GstStreamCollection the same way it did for the initial pads in `GstStreamCollection` the same way it did for the initial pads in
section 4.2 above. section 4.2 above.
2) decodebin will see whether the new stream can re-use a multiqueue 2) decodebin will see whether the new stream can re-use a multiqueue
slot used by a stream of the same type no longer present (it slot used by a stream of the same type no longer present (it
@ -474,7 +485,10 @@ a collection containing
The user might want to use the korean audio track instead of the The user might want to use the korean audio track instead of the
default english one. default english one.
```
=> SELECT_STREAMS ("video", "kor") => SELECT_STREAMS ("video", "kor")
```
1) decodebin3 receives and sends the event further upstream 1) decodebin3 receives and sends the event further upstream
2) tsdemux sees that "video" is part of its current upstream, 2) tsdemux sees that "video" is part of its current upstream,
@ -511,20 +525,20 @@ Initially tsdemux exposes the first program present (default)
``` ```
At some point the user wants to switch to ITV (of which we do not At some point the user wants to switch to ITV (of which we do not
know the topology at this point in time. A SELECT_STREAMS event know the topology at this point in time. A `SELECT_STREAMS` event
is sent with "ITV" in it and the pointer to the Collection1. is sent with "ITV" in it and the pointer to the Collection1.
1) The event travels up the pipeline until tsdemux receives it 1) The event travels up the pipeline until tsdemux receives it
and begins the switch. and begins the switch.
2) tsdemux publishes a new 'Collection 2a/ITV' and marks 'Collection 2/BBC' 2) tsdemux publishes a new 'Collection 2a/ITV' and marks 'Collection 2/BBC'
as replaced. as replaced.
2a) App may send a SELECT_STREAMS event configuring which demuxer output 2a) App may send a `SELECT_STREAMS` event configuring which demuxer output
streams should be selected (parsed) streams should be selected (parsed)
3) tsdemux adds/removes pads as needed (flushing pads as it removes them?) 3) tsdemux adds/removes pads as needed (flushing pads as it removes them?)
4) Decodebin feeds new pad streams through existing parsers/decoders as 4) Decodebin feeds new pad streams through existing parsers/decoders as
needed. As data from the new collection arrives out each decoder, needed. As data from the new collection arrives out each decoder,
decodebin sends new GstStreamCollection messages to the app so it decodebin sends new `GstStreamCollection` messages to the app so it
can know that the new streams are now switchable at that level. can know that the new streams are now switchable at that level.
4a) As new GstStreamCollections are published, the app may override 4a) As new `GstStreamCollections` are published, the app may override
the default decodebin stream selection to expose more/fewer streams. the default decodebin stream selection to expose more/fewer streams.
The default is to decode and output 1 stream of each type. The default is to decode and output 1 stream of each type.
@ -545,9 +559,9 @@ Final state:
- Add missing implementation - Add missing implementation
- Add flags to GstStreamCollection - Add flags to `GstStreamCollection`
- Add mutual-exclusion and relationship API to GstStreamCollection - Add mutual-exclusion and relationship API to `GstStreamCollection`
- Add helper API to figure out whether a collection is a replacement - Add helper API to figure out whether a collection is a replacement
of another or a completely new one. This will require a more generic of another or a completely new one. This will require a more generic
@ -556,8 +570,8 @@ another or not.
### OPEN QUESTIONS ### OPEN QUESTIONS
- Is a FLUSHING flag for stream-selection required or not ? This would - Is a `FLUSHING` flag for stream-selection required or not ? This would
make the handler of the SELECT\_STREAMS event send FLUSH START/STOP make the handler of the `SELECT_STREAMS` event send `FLUSH START/STOP`
before switching to the other streams. This is tricky when dealing before switching to the other streams. This is tricky when dealing
where situations where we keep some streams and only switch some where situations where we keep some streams and only switch some
others. Do we flush all streams ? Do we only flush the new streams, others. Do we flush all streams ? Do we only flush the new streams,