mirrors/gstreamer
mirrors/gstreamer
1
1
Fork 0
mirror of https://gitlab.freedesktop.org/gstreamer/gstreamer.git synced 2025-01-09 08:55:33 +00:00
Code Issues 1 Projects Releases Wiki Activity

design-docs: add html output using asciidoc

Browse source 
Unify the ad-hoc markup to be asciidoc style in many places. Add a "html" target
to Makefile to generate the output.
This commit is contained in:
Stefan Kost 2010-11-01 15:32:43 +02:00
parent 1c50dcd54f
commit 16ce2d4ea4
46 changed files with 1637 additions and 1589 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

16
docs/design/Makefile.am
View file

@ -47,6 +47,22 @@ EXTRA_DIST = \
part-TODO.txt \
part-trickmodes.txt
CLEANFILES = index.html index.txt
html:
if ! test -z `which asciidoc`; then \
echo >index.txt "GStreamer design"; \
echo >>index.txt "================"; \
echo >>index.txt "The Gstreamer developers"; \
echo >>index.txt "Version $(PACKAGE_VERSION)"; \
echo >>index.txt ""; \
( cd $(srcdir) && \
cat >>$(abs_builddir)/index.txt $(EXTRA_DIST) ); \
asciidoc -o index.html index.txt; \
else \
echo "need asciidoc to generate html"; \
fi;
upload:
@echo nothing to upload

40
docs/design/draft-buffer2.txt
View file

@ -23,7 +23,7 @@ Some examples of metadata:
Requirements
------------
~~~~~~~~~~~~
- It must be fast
* allocation, free, low fragmentation
@ -41,7 +41,7 @@ Requirements
GstMiniObject
-------------
~~~~~~~~~~~~~
We make GstMiniObject a simple refcounted C structure and also a GLib boxed
type. The following fields will be in the structure:
@ -66,7 +66,7 @@ objects.
GstEvent, GstCaps, GstQuery, GstMessage
---------------------------------------
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Have the new GstMiniObject be the first field in these objects. They will probably
also replace the copy and free functions with their own implementations.
@ -77,7 +77,7 @@ the custom functions.
GstBuffer
---------
~~~~~~~~~
A GstMiniObject will be the parent instance of the GstBuffer object, which is a
regular C structure.
@ -105,7 +105,7 @@ Buffers point to a GstCaps structure that contains the caps of the buffer data.
GstBufferMeta
-------------
~~~~~~~~~~~~~
A GstBufferMeta is a structure as follows:
@ -208,7 +208,7 @@ GstBufferMetaMemory | | next ---+
. .
API examples
------------
~~~~~~~~~~~~
Buffers are created using the normal gst_buffer_new functions. The standard fields
are initialized as usual. A memory area that is bigger than the structure size
@ -276,7 +276,7 @@ possible simple API would look like this:
Memory management
-----------------
~~~~~~~~~~~~~~~~~
* allocation
@ -316,7 +316,7 @@ Memory management
Subbuffers
----------
~~~~~~~~~~
Subbuffers are a first class feature of the GstBuffer.
@ -334,7 +334,7 @@ timing metadata needs to be reset to NONE when the start offset is different.
Serialization
-------------
~~~~~~~~~~~~~
When buffer should be sent over the wire or be serialized in GDP, we need a way
to perform custom serialization and deserialization on the metadata.
@ -345,7 +345,7 @@ and endianness.
Transformations
---------------
~~~~~~~~~~~~~~~
After certain transformations, the metadata on a buffer might not be relevant
anymore.
@ -368,7 +368,7 @@ so on).
Other use cases
---------------
~~~~~~~~~~~~~~~
Making the GstBufferMetaMemory (for making the buffer point to the associated
memory region) as metadata on a GstBuffer, as opposed to making it an integral
@ -397,7 +397,7 @@ implementations add (or use, in the case of a file reader) the memory metadata.
Relationship with GstCaps
-------------------------
~~~~~~~~~~~~~~~~~~~~~~~~~
The difference between GstCaps, used in negotiation, and the metadata is not
clearly defined.
@ -415,7 +415,7 @@ video resolution while the implementation details would be inside the metadata.
Compatibility
-------------
~~~~~~~~~~~~~
We need to make sure that elements exchange metadata that they both understand,
This is particulary important when the metadata describes the data layout in
@ -436,16 +436,16 @@ that are metadata aware to set a flag on their pads; any buffer passed on that
pad will be converted to the old layout when the flag is not set.
Notes:
------
Notes
~~~~~
Some structures that we need to be able to add to buffers.
Clean Aperture
Arbitrary Matrix Transform
Aspect ratio
Pan/crop/zoom
Video strides
* Clean Aperture
* Arbitrary Matrix Transform
* Aspect ratio
* Pan/crop/zoom
* Video strides
Some of these overlap, we need to find a minimal set of metadata structures that
allows us to define all use cases.

280
docs/design/draft-klass.txt
View file

@ -1,189 +1,187 @@
Element Klass definition
------------------------
Status:
Purpose
~~~~~~~
DRAFT.
Applications should be able to retrieve elements from the registry of existing
elements based on specific capabilities or features of the element.
A playback application might want to retrieve all the elements that can be
used for visualisation, for example, or a video editor might want to select
all video effect filters.
Purpose:
The topic of defining the klass of elements should be based on use cases.
Applications should be able to retrieve elements from the registry of existing
elements based on specific capabilities or features of the element.
A list of classes that are used in a installation can be generated using:
gst-inspect-0.10 -a | grep -ho Class:.* | cut -c8- | sed "s/\//\\n/g" | sort | uniq
A playback application might want to retrieve all the elements that can be
used for visualisation, for example, or a video editor might want to select
all video effect filters.
Proposal
~~~~~~~~
The topic of defining the klass of elements should be based on use cases.
The GstElementDetails contains a field named klass that is a pointer to a
string describing the element type.
A list of classes that are used in a installation can be generated using:
gst-inspect-0.10 -a | grep -ho Class:.* | cut -c8- | sed "s/\//\\n/g" | sort | uniq
In this document we describe the format and contents of the string. Elements
should adhere to this specification although that is not enforced to allow
for wild (application specific) customisation.
Proposal:
1) string format
The GstElementDetails contains a field named klass that is a pointer to a
string describing the element type.
<keyword>['/'<keyword]*
In this document we describe the format and contents of the string. Elements
should adhere to this specification although that is not enforced to allow
for wild (application specific) customisation.
The string consists of an _unordered_ list of keywords separated with a '/'
character. While the / suggests a hierarchy, this is _not_ the case.
1) string format
2) keyword categories
<keyword>['/'<keyword]*
- functional
The string consists of an _unordered_ list of keywords separated with a '/'
character. While the / suggests a hierarchy, this is _not_ the case.
Categories are base on _intended usage_ of the element. Some elements
might have other side-effects (especially for filers/effects). The purpose
is to list enough keywords so that applications can do meaningfull filtering,
not to completely describe the functionality, that is expressed in caps etc..
2) keyword categories
* Source : produces data
* Sink : consumes data
* Filter : filters/transforms data, no modification on the data is
intended (although it might be unavoidable). The
filter can decide on input and output caps independently
of the stream contents (GstBaseTransform).
* Effect : applies an effect to some data, changes to data are
intended. Examples are colorbalance, volume. These
elements can also be implemented with GstBaseTransform.
* Demuxer : splits audio, video, ... from a stream
* Muxer : interleave audio, video, ... into one stream, this is
like mixing but without losing or degrading each separate
input stream. The reverse operation is possible with a
Demuxer that reproduces the exact same input streams.
* Decoder : decodes encoded data into a raw format, there is typically
no relation between input caps and output caps. The output
caps are defined in the stream data. This separates the
Decoder from the Filter and Effect.
* Encoder : encodes raw data into an encoded format.
* Mixer : combine audio, video, .. this is like muxing but with
applying some algorithm so that the individual streams
are not extractable anymore, there is therefore no
reverse operation to mixing. (audio mixer, video mixer, ...)
* Converter : convert audio into video, text to audio, ... The converter
typically works on raw types only. The source media type
is listed first.
* Analyzer : reports about the stream contents.
* Control : controls some aspect of a hardware device
* Extracter : extracts tags/headers from a stream
* Formatter : adds tags/headers to a stream
* Connector : allows for new connections in the pipeline. (tee, ...)
* ...
- functional
- Based on media type
Categories are base on _intended usage_ of the element. Some elements
might have other side-effects (especially for filers/effects). The purpose
is to list enough keywords so that applications can do meaningfull filtering,
not to completely describe the functionality, that is expressed in caps etc..
Purpose is to make a selection for elements operating on the different
types of media. An audio application must be able to filter out the
elements operating on audio, for example.
* Source : produces data
* Sink : consumes data
* Filter : filters/transforms data, no modification on the data is
intended (although it might be unavoidable). The
filter can decide on input and output caps independently
of the stream contents (GstBaseTransform).
* Effect : applies an effect to some data, changes to data are
intended. Examples are colorbalance, volume. These
elements can also be implemented with GstBaseTransform.
* Demuxer : splits audio, video, ... from a stream
* Muxer : interleave audio, video, ... into one stream, this is
like mixing but without losing or degrading each separate
input stream. The reverse operation is possible with a
Demuxer that reproduces the exact same input streams.
* Decoder : decodes encoded data into a raw format, there is typically
no relation between input caps and output caps. The output
caps are defined in the stream data. This separates the
Decoder from the Filter and Effect.
* Encoder : encodes raw data into an encoded format.
* Mixer : combine audio, video, .. this is like muxing but with
applying some algorithm so that the individual streams
are not extractable anymore, there is therefore no
reverse operation to mixing. (audio mixer, video mixer, ...)
* Converter : convert audio into video, text to audio, ... The converter
typically works on raw types only. The source media type
is listed first.
* Analyzer : reports about the stream contents.
* Control : controls some aspect of a hardware device
* Extracter : extracts tags/headers from a stream
* Formatter : adds tags/headers to a stream
* Connector : allows for new connections in the pipeline. (tee, ...)
* ...
* Audio : operates on audio data
* Video : operates on video data
* Image : operates on image data. Usually this media type can also
be used to make a video stream in which case it is added
together with the Video media type.
* Text : operates on text data
* Metadata : operates on metadata
* ...
- Based on media type
- Extra features
Purpose is to make a selection for elements operating on the different
types of media. An audio application must be able to filter out the
elements operating on audio, for example.
The purpose is to further specialize the element, mostly for
application specific needs.
* Audio : operates on audio data
* Video : operates on video data
* Image : operates on image data. Usually this media type can also
be used to make a video stream in which case it is added
together with the Video media type.
* Text : operates on text data
* Metadata : operates on metadata
* ...
* Network : element is used in networked situations
* Protocol : implements some protocol (RTSP, HTTP, ...)
* Payloader : encapsulate as payload (RTP, RDT,.. )
* Depayloader : strip a payload (RTP, RDT,.. )
* RTP : intended to be used in RTP applications
* Device : operates on some hardware device (disk, network,
audio card, video card, usb, ...)
* Visualisation : intended to be used for audio visualisation
* Debug : intended usage is more for debugging purposes.
- Extra features
- Categories found, but not yet in one of the above lists
The purpose is to further specialize the element, mostly for
application specific needs.
* Bin : playbin, decodebin, bin, pipeline
* Codec : lots of decoders, encoder, demuxers
should be removed?
* Generic : should be removed?
* File : like network, should go to Extra?
* Editor : gnonlin, textoverlays
* DVD, GDP, LADSPA, Parser, Player, Subtitle, Testing, ...
* Network : element is used in networked situations
* Protocol : implements some protocol (RTSP, HTTP, ...)
* Payloader : encapsulate as payload (RTP, RDT,.. )
* Depayloader : strip a payload (RTP, RDT,.. )
* RTP : intended to be used in RTP applications
* Device : operates on some hardware device (disk, network,
audio card, video card, usb, ...)
* Visualisation : intended to be used for audio visualisation
* Debug : intended usage is more for debugging purposes.
3) suggested order:
- Categories found, but not yet in one of the above lists
<functional>[/<media type>]*[/<extra...>]*
* Bin : playbin, decodebin, bin, pipeline
* Codec : lots of decoders, encoder, demuxers
should be removed?
* Generic : should be removed?
* File : like network, should go to Extra?
* Editor : gnonlin, textoverlays
* DVD, GDP, LADSPA, Parser, Player, Subtitle, Testing, ...
4) examples:
3) suggested order:
apedemux : Extracter/Metadata
audiotestsrc : Source/Audio
autoaudiosink : Sink/Audio/Device
cairotimeoverlay : Mixer/Video/Text
dvdec : Decoder/Video
dvdemux : Demuxer
goom : Converter/Audio/Video
id3demux : Extracter/Metadata
udpsrc : Source/Network/Protocol/Device
videomixer : Mixer/Video
ffmpegcolorspace : Filter/Video (intended use to convert video with as little
visible change as possible)
vertigotv : Effect/Video (intended use is to change the video)
volume : Effect/Audio (intended use is to change the audio data)
vorbisdec : Decoder/Audio
vorbisenc : Encoder/Audio
oggmux : Muxer
adder : Mixer/Audio
videobox : Effect/Video
alsamixer : Control/Audio/Device
audioconvert : Filter/Audio
audioresample : Filter/Audio
xvimagesink : Sink/Video/Device
navseek : Filter/Debug
decodebin : Decoder/Demuxer
level : Filter/Analyzer/Audio
tee : Connector/Debug
<functional>[/<media type>]*[/<extra...>]*
5) open issues:
4) examples:
apedemux : Extracter/Metadata
audiotestsrc : Source/Audio
- how to differencial physical devices from logical ones?
autoaudiosink : Sink/Audio/Device
cairotimeoverlay : Mixer/Video/Text
dvdec : Decoder/Video
dvdemux : Demuxer
goom : Converter/Audio/Video
id3demux : Extracter/Metadata
udpsrc : Source/Network/Protocol/Device
videomixer : Mixer/Video
ffmpegcolorspace : Filter/Video (intended use to convert video with as little
visible change as possible)
vertigotv : Effect/Video (intended use is to change the video)
volume : Effect/Audio (intended use is to change the audio data)
vorbisdec : Decoder/Audio
vorbisenc : Encoder/Audio
oggmux : Muxer
adder : Mixer/Audio
videobox : Effect/Video
alsamixer : Control/Audio/Device
audioconvert : Filter/Audio
audioresample : Filter/Audio
xvimagesink : Sink/Video/Device
navseek : Filter/Debug
decodebin : Decoder/Demuxer
level : Filter/Analyzer/Audio
tee : Connector/Debug
alsasink : Sink/Audio/Device
5) open issues:
Use cases
~~~~~~~~~
- how to differencial physical devices from logical ones?
autoaudiosink : Sink/Audio/Device
alsasink : Sink/Audio/Device
- get a list of all elements implementing a video effect (pitivi):
Use cases:
klass.contains (Effect & Video)
- get a list of all elements implementing a video effect (pitivi):
- get list of muxers (pitivi):
klass.contains (Effect & Video)
klass.contains (Muxer)
- get list of muxers (pitivi):
- get list of video encoders (pitivi):
klass.contains (Muxer)
klass.contains (Encoder & video)
- get list of video encoders (pitivi):
- Get a list of all audio/video visualisations (totem):
klass.contains (Encoder & video)
klass.contains (Visualisation)
- Get a list of all audio/video visualisations (totem):
- Get a list of all decoders/demuxer/metadata parsers/vis (playbin):
klass.contains (Visualisation)
klass.contains (Visualisation | Demuxer | Decoder | (Extractor & Metadata))
- Get a list of all decoders/demuxer/metadata parsers/vis (playbin):
- Get a list of elements that can capture from an audio device (gst-properties):
klass.contains (Visualisation | Demuxer | Decoder | (Extractor & Metadata))
klass.contains (Source & Audio & Device)
- Get a list of elements that can capture from an audio device (gst-properties):
klass.contains (Source & Audio & Device)
* filters out audiotestsrc, since it is not a device
* filters out audiotestsrc, since it is not a device

17
docs/design/draft-metadata.txt
View file

@ -6,7 +6,7 @@ additions.
Supported Metadata standards
----------------------------
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The paragraphs below list supported native metadata standards sorted by type and
then in alphabetical order. Some standards have been extended to support
@ -94,7 +94,7 @@ http://wiki.creativecommons.org/Tracker_CC_Indexing
Current Metadata handling
-------------------------
~~~~~~~~~~~~~~~~~~~~~~~~~
When reading files, demuxers or parsers extract the metadata. It will be sent as
a GST_MESSAGE_TAG to the GstBus and GST_EVENT_TAG to downstream elements.
@ -130,10 +130,11 @@ metadata.
Issues
------
~~~~~~
Unknown/Unmapped metadata
- - - - - - - - - - - - -
^^^^^^^^^^^^^^^^^^^^^^^^^
Right now GStreamer can lose metadata when transcoding, remuxing content. This
can happend as we don't map all metadata fields to generic ones.
@ -159,7 +160,8 @@ We would basically need this for each container format.
See also https://bugzilla.gnome.org/show_bug.cgi?id=345352
Lost metadata
- - - - - - -
^^^^^^^^^^^^^
A case slighly different from the previous is that when an application sets a
GstTagList on a pipeline. Right elements consuming tags do not report which tags
have been consumed. Especially when using elements that make metadata
@ -181,7 +183,8 @@ of elements and their tags. As a convenience we could flatten the list of tags
for the top-level element (if the query was sent to a bin) and add that.
Tags are per Element
- - - - - - - - - -
^^^^^^^^^^^^^^^^^^^^
In many cases we want tags per stream. Even metadata standards like mp4/3gp
metadata supports that. Right now GST_MESSAGE_SRC(tags) is the element. We tried
changing that to the pad, but that broke applications.
@ -189,7 +192,7 @@ Also we miss the symmetric functionality in GstTagSetter. This interface is
usually implemented by elements.
Open bugs
- - - - -
^^^^^^^^^
https://bugzilla.gnome.org/buglist.cgi?query_format=advanced;short_desc=tag;bug_status=UNCONFIRMED;bug_status=NEW;bug_status=ASSIGNED;bug_status=REOPENED;bug_status=NEEDINFO;short_desc_type=allwordssubstr;product=GStreamer

28
docs/design/draft-tagreading.txt
View file

@ -20,7 +20,7 @@ peeked buffers (a queue element after sink, but that would change pull to push).
Design
------
~~~~~~
The plan is that applications can do the following:
pipeline = "filesrc ! tagbin"
@ -38,7 +38,8 @@ The plan is that applications can do the following:
decodebin
Interface
---------
~~~~~~~~~
* gboolean iface property "tag-reading"
Switches the element to tagreading mode. Needed if normal element implement
that behaviour. Elements will skip parsing unneeded data, don't build a
@ -47,7 +48,7 @@ Interface
Equivalent of EOS.
Use Cases
---------
~~~~~~~~~
* mp3 with id3- and apetags
* plug id3demux ! apedemux
@ -57,7 +58,8 @@ Use Cases
* plug vorbisdec or special vorbiscomment reader
Additional Thoughts
-------------------
~~~~~~~~~~~~~~~~~~~
* would it make sense to have 2-phase tag-reading (property on tagbin and/or
tagread elements)
* 1st phase: get tag-data that are directly embedded in the data
@ -73,23 +75,25 @@ Additional Thoughts
gst_tag_cache_store_tag_data (GstTagCache *self, const gchar *uri, GstTagList *tags);
Tests
-----
~~~~~
* write a generic test for parsers/demuxers to ensure they send tags until they
reached PAUSED (elements need to parse file for prerolling anyway):
set pipeline to paused, check for tags, set to playing, error out if tags come
after paused
Code Locations
--------------
~~~~~~~~~~~~~~
* tagreadbin -> gst-plugins-base/gst/tagread
* tagreaderiface -> gst-plugins-base/gst-libs/gst/tag
Reuse
-----
~~~~~
ogg : gst-plugins-base/ext/ogg
avi : gst-plugins-good/gst/avi
mp3 : gst-plugins-good/gst/id3demux
wav : gst-plugins-good/gst/wavparse
qt : gst-plugins-bad/gst/qtdemux
* ogg : gst-plugins-base/ext/ogg
* avi : gst-plugins-good/gst/avi
* mp3 : gst-plugins-good/gst/id3demux
* wav : gst-plugins-good/gst/wavparse
* qt : gst-plugins-bad/gst/qtdemux

5
docs/design/part-MT-refcounting.txt
View file

@ -10,7 +10,7 @@ This design decision has implications for the usage of the API and the objects
which this document explains.
MT safety techniques
--------------------
~~~~~~~~~~~~~~~~~~~~
Several design patterns are used to guarantee object consistency in GStreamer.
This is an overview of the methods used in various GStreamer subsystems.
@ -140,7 +140,7 @@ Compare and swap
Objects
-------
~~~~~~~
* Locking involved:
@ -414,3 +414,4 @@ Objects
}
}
gst_iterator_free (it);

9
docs/design/part-TODO.txt
View file

@ -1,5 +1,8 @@
TODO - Future Development
-------------------------
API/ABI
-------
~~~~~~~
- implement return values from events in addition to the gboolean. This should
be done by making the event contain a GstStructure with input/output values,
@ -66,7 +69,7 @@ API/ABI
IMPLEMENTATION
--------------
~~~~~~~~~~~~~~
- implement more QOS, see part-qos.txt.
@ -76,7 +79,7 @@ IMPLEMENTATION
DESIGN
------
~~~~~~
- unlinking pads in the PAUSED state needs to make sure the stream thread is not
executing code. Can this be done with a flush to unlock all downstream chain

12
docs/design/part-activation.txt
View file

@ -1,5 +1,8 @@
Pad activation
--------------
Pad (de)activation
------------------
Activation
~~~~~~~~~~
When changing states, a bin will set the state on all of its children in
sink-to-source order. As elements undergo the READY->PAUSED transition,
@ -73,7 +76,7 @@ already in PAUSED with an active sink pad by the time fakesrc starts
pushing data.
Deactivation
------------
~~~~~~~~~~~~
Pad deactivation occurs when its parent goes into the READY state or when the
pad is deactivated explicitly by the application or element.
@ -82,7 +85,8 @@ activate_push() or activate_pull() with a FALSE argument, depending on the
activation mode of the pad.
Mode switching
--------------
~~~~~~~~~~~~~~
Changing from push to pull modes needs a bit of thought. This is actually
possible and implemented but not yet documented here.

126
docs/design/part-block.txt
View file

@ -18,66 +18,69 @@ and will succeed when the following events happen on the pad:
Flushing
--------
~~~~~~~~
The flushing event is used to clear any data out of the
downstream elements.
The flushing event is used to clear any data out of the
downstream elements.
* Generic case
Generic case
^^^^^^^^^^^^
Consider the following pipeline:
Consider the following pipeline:
.-----. .-------. .-------.
| src | | elem1 |\/ | elem2 |
| src -> sink src -> sink src ....
'-----' '-------'/\ '-------'
Where elem1.src is blocked. If the pad block is taken (the callback
is called or the sync block returned) no data is flowing in elem2.sink.
In this situation, the streaming thread is blocked on a GCond and is
waiting to be unblocked.
Where elem1.src is blocked. If the pad block is taken (the callback
is called or the sync block returned) no data is flowing in elem2.sink.
In this situation, the streaming thread is blocked on a GCond and is
waiting to be unblocked.
When sending a flushing seek upstream on elem1.src, the FLUSH_START and
will temporary unblock the streaming thread and make all pad functions that
triggers a block (_push/_alloc_buffer/_push_event/_pull_range) return
GST_FLOW_WRONG_STATE. This will then eventually pause the streaming thread
and release the STREAM_LOCK.
Since no STREAM lock is taken after the pad block it is not needed to send
the FLUSH_START event further downstream.
When sending a flushing seek upstream on elem1.src, the FLUSH_START and
will temporary unblock the streaming thread and make all pad functions that
triggers a block (_push/_alloc_buffer/_push_event/_pull_range) return
GST_FLOW_WRONG_STATE. This will then eventually pause the streaming thread
and release the STREAM_LOCK.
The FLUSH_STOP will set the srcpad to non-flushing again and is dropped
for the same reason. From then on, the new data after the flushing seek
will be queued when the pad block is taken again.
Since no STREAM lock is taken after the pad block it is not needed to send
the FLUSH_START event further downstream.
* Case where the stream is blocking downstream
The FLUSH_STOP will set the srcpad to non-flushing again and is dropped
for the same reason. From then on, the new data after the flushing seek
will be queued when the pad block is taken again.
The example above is only valid if the elem1.src pad is really blocking
(callback called or sync block returned).
In the case where the stream is blocking further downstream (on elem2.src
for example, or on a blocking queue), extra care has to be taken.
Case where the stream is blocking downstream
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Consider the following pipeline:
The example above is only valid if the elem1.src pad is really blocking
(callback called or sync block returned).
In the case where the stream is blocking further downstream (on elem2.src
for example, or on a blocking queue), extra care has to be taken.
Consider the following pipeline:
.-----. .-------. .-------.
| src | | elem1 |\/ | elem2 |
| src -> sink src -> sink src .... Blocking somewhere downstream
'-----' '-------'/\ '-------'
A pad block has been requested by the user on elem1.src , but since the stream
is blocking somewhere downstream, the callback is not called or the sync block
does not return.
A pad block has been requested by the user on elem1.src , but since the stream
is blocking somewhere downstream, the callback is not called or the sync block
does not return.
In order for the block to happen, a FLUSH_START needs to be sent directly on
the downstream blocking element/pad so that it releases the stream lock, and it
gives a chance for the elem1.src pad to block.
In order for the block to happen, a FLUSH_START needs to be sent directly on
the downstream blocking element/pad so that it releases the stream lock, and it
gives a chance for the elem1.src pad to block.
Use cases:
----------
Use cases
~~~~~~~~~
* Prerolling a partial pipeline
Prerolling a partial pipeline
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.---------. .---------. .----------.
| filesrc | | demuxer | .-----. | decoder1 |
@ -89,27 +92,28 @@ Use cases:
'---------' '-----' '----------' X
The purpose is to create the pipeline dynamically up to the
decoders but not yet connect them to a sink and without losing
any data.
The purpose is to create the pipeline dynamically up to the
decoders but not yet connect them to a sink and without losing
any data.
To do this, the source pads of the decoders is blocked so that no
events or buffers can escape and we don't interrupt the stream.
To do this, the source pads of the decoders is blocked so that no
events or buffers can escape and we don't interrupt the stream.
When all of the dynamic pad are created (no-more-pads emited by the
branching point, ie, the demuxer or the queues filled) and the pads
are blocked (blocked callback received) the pipeline is completely
prerolled.
When all of the dynamic pad are created (no-more-pads emited by the
branching point, ie, the demuxer or the queues filled) and the pads
are blocked (blocked callback received) the pipeline is completely
prerolled.
It should then be possible to perform the following actions on the
prerolled pipeline:
It should then be possible to perform the following actions on the
prerolled pipeline:
- query duration/position
- perform a flushing seek to preroll a new position
- connect other elements and unblock the blocked pads.
- query duration/position
- perform a flushing seek to preroll a new position
- connect other elements and unblock the blocked pads.
* dynamically switching an element in a PLAYING pipeline.
dynamically switching an element in a PLAYING pipeline
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.----------. .----------. .----------.
@ -121,8 +125,8 @@ Use cases:
sink src
'----------'
The purpose is to replace element2 with element4 in the PLAYING
pipeline.
The purpose is to replace element2 with element4 in the PLAYING
pipeline.
1) block element1 src pad. This can be done async.
2) wait for block to happen. at that point nothing is flowing between
@ -144,18 +148,18 @@ Use cases:
element should at least be PAUSED.
9) unblock element1 src
The same flow can be used to replace an element in a PAUSED pipeline. Only
special care has to be taken when performing step 2) which has to be done
async or it might deadlock. In the async callback one can then perform the
steps from 3). In a playing pipeline one can of course use the async block
as well, so that there is a generic method for both PAUSED and PLAYING.
The same flow can be used to replace an element in a PAUSED pipeline. Only
special care has to be taken when performing step 2) which has to be done
async or it might deadlock. In the async callback one can then perform the
steps from 3). In a playing pipeline one can of course use the async block
as well, so that there is a generic method for both PAUSED and PLAYING.
The same flow works as well for any chain of multiple elements and might
be implemented with a helper function in the future.
The same flow works as well for any chain of multiple elements and might
be implemented with a helper function in the future.
Issues
------
~~~~~~
When an EOS event has passed a pad and the pad is set to blocked, the block will
never happen because no data is going to flow anymore. One possibility is to

19
docs/design/part-buffering.txt
View file

@ -21,7 +21,7 @@ We want to be able to implement the following features:
- easy (backward compatible) application notification of buffering
- the possibility for the application to do more complex buffering
Some use cases:
Some use cases:
* Stream buffering:
@ -112,7 +112,7 @@ We want to be able to implement the following features:
Messages
--------
~~~~~~~~
A GST_MESSAGE_BUFFERING must be posted on the bus when playback temporarily
stops to buffer and when buffering finishes. When percentage field in the
@ -137,27 +137,24 @@ are added to the buffering message:
"avg-in-rate", G_TYPE_INT
- gives the average input buffering speed in bytes/second. -1 is unknown.
This is the average number of bytes per second that is received on the
buffering element input (sink) pads. It is a measurement of the network
speed in most cases.
"avg-out-rate", G_TYPE_INT
- gives the average consumption speed in bytes/second. -1 is unknown.
This is the average number of bytes per second that is consumed by the
downstream element of the buffering element.
"buffering-left", G_TYPE_INT64
- gives the estimated time that bufferring will take in milliseconds.
-1 unknown.
This is measured based on the avg-in-rate and the filled level of the
queue. The application can use this hint to update the GUI about the
estimated remaining time that buffering will take.
This is measured based on the avg-in-rate and the filled level of the
queue. The application can use this hint to update the GUI about the
estimated remaining time that buffering will take.
Application
-----------
~~~~~~~~~~~
While data is buffered, the pipeline should remain in the PAUSED state. It is
also possible that more data should be buffered while the pipeline is PLAYING,
@ -180,7 +177,7 @@ time to resume playback to get uninterrupted playback.
Buffering Query
---------------
~~~~~~~~~~~~~~~
In addition to the BUFFERING messages posted by the buffering elements we want
to be able to query the same information from the application. We also want to
@ -222,7 +219,7 @@ and newest item (expressed in "format") in the buffer.
Defaults
--------
~~~~~~~~
Some defaults for common elements:

38
docs/design/part-bufferlist.txt
View file

@ -21,17 +21,18 @@ for the following extra functionality:
Use cases
---------
~~~~~~~~~
A typical use case for multimedia pipelines is to append or remove 'headers'
from packets of data.
* Generating RTP packets from h264 video
Generating RTP packets from h264 video
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
We receive as input a GstBuffer with an encoded h264 image and we need to
create RTP packets containing this h264 data as the payload. We typically need
to fragment the h264 data into multiple packets, each with their own RTP and
payload specific header.
We receive as input a GstBuffer with an encoded h264 image and we need to
create RTP packets containing this h264 data as the payload. We typically need
to fragment the h264 data into multiple packets, each with their own RTP and
payload specific header.
+-------+-------+---------------------------+--------+
input H264 buffer: | NALU1 | NALU2 | ..... | NALUx |
@ -41,25 +42,25 @@ from packets of data.
+-+ +-------+ +-+ +-------+ +-+ +-------+
output bufferlist: | | | NALU1 | | | | NALU2 | .... | | | NALUx |
+-+ +-------+ +-+ +-------+ +-+ +-------+
: : : :
\-----------/ \-----------/
group 1 group 2
The output bufferlist consists of x groups consisting of an RTP payload header
and a subbuffer of the original input H264 buffer. Since the rtp headers and
the h264 data don't need to be contiguous in memory, we can avoid to memcpy the
h264 data into the rtp packets.
The output bufferlist consists of x groups consisting of an RTP payload header
and a subbuffer of the original input H264 buffer. Since the rtp headers and
the h264 data don't need to be contiguous in memory, we can avoid to memcpy the
h264 data into the rtp packets.
Since we can generate a bufferlist with multiple groups, we can push all the
RTP packets for the input data to the next element in one operation.
Since we can generate a bufferlist with multiple groups, we can push all the
RTP packets for the input data to the next element in one operation.
A typical udpsink will then use something like sendmsg to send the groups on
the network inside one UDP packet. This will further avoid having to memcpy
data into contiguous memory.
A typical udpsink will then use something like sendmsg to send the groups on
the network inside one UDP packet. This will further avoid having to memcpy
data into contiguous memory.
API
---
~~~
The GstBufferList is an opaque data structure and is operated on using an
iterator. It derives from GstMiniObject so that it has basic refcounting and
@ -78,7 +79,7 @@ group.
Metadata
--------
~~~~~~~~
Each of the buffers inside the bufferlist can have metadata assiociated with it.
@ -92,3 +93,4 @@ first group in the bufferlist. This means that:
This allows for efficient (re)timestamping and re-typing (caps) of a group of
buffers without having to modify each of the buffer's metadata.

6
docs/design/part-clocks.txt
View file

@ -11,7 +11,7 @@ The time reported by the clock is called the absolute_time.
Clock Selection
---------------
~~~~~~~~~~~~~~~
To synchronize the different elements, the GstPipeline is responsible for
selecting and distributing a global GstClock for all the elements in it.
@ -35,7 +35,7 @@ PLAYING and is described in part-states.txt.
Clock features
--------------
~~~~~~~~~~~~~~
The clock supports periodic and single shot clock notifications both
synchronous and asynchronous.
@ -73,7 +73,7 @@ the clock is not PLAYING.
Clock implementations
---------------------
~~~~~~~~~~~~~~~~~~~~~
The GStreamer core provides a GstSystemClock based on the system time.
Asynchronous callbacks are scheduled from an internal thread.

30
docs/design/part-conventions.txt
View file

@ -1,5 +1,5 @@
Documentation conventions
=========================
-------------------------
Due to the potential for exponential growth, several abbreviating conventions will be used throughout this
documentation. These conventions have grown primarily from extremely in-depth discussions of the architecure in IRC.
@ -7,21 +7,21 @@ This has verified the safety of these conventions, if used properly. There are
context is rigorously observed.
Object classes
--------------
~~~~~~~~~~~~~~
Since everything starts with Gst, we will generally refer to objects by the shorter name, i.e. Element or Pad. These
names will always have their first letter capitalized.
Function names
--------------
~~~~~~~~~~~~~~
Within the context of a given object, functions defined in that object's header and/or source file will have their
object-specific prefix stripped. For instance, gst_element_add_pad() would be referred to as simply _add_pad(). Note
that the trailing parentheses should always be present, but sometimes may not be. A prefixing underscore (_) will
always tell you it's a function, however, regardless of the presence or absence of the trailing parentheses.
#defines and enums
------------------
defines and enums
~~~~~~~~~~~~~~~~~
Values and macros defined as enums and preprocessor macros will be referred to in all capitals, as per their
definition. This includes object flags and element states, as well as general enums. Examples are the states NULL,
@ -31,23 +31,25 @@ element flags should be cross-checked with the header, as there are currently tw
_FLAGS_ in the middle.
Drawing conventions
===================
~~~~~~~~~~~~~~~~~~~
When drawing pictures the folowing conventions apply:
objects
-------
^^^^^^^
Objects are drawn with a box like:
Objects are drawn with a box like
+------+
| |
+------+
pointers
--------
^^^^^^^^
a pointer to an object.
+-----+
*--->| |
+-----+
@ -58,7 +60,7 @@ an invalid pointer, this is a pointer that should not be used.
elements
--------
^^^^^^^^
+----------+
| name |
@ -66,16 +68,10 @@ elements
+----------+
pad links
---------
^^^^^^^^^
-----+ +---
| |
src--sink
-----+ +---

1
docs/design/part-dynamic.txt
View file

@ -11,3 +11,4 @@ following features of gstreamer:
- pad blocking (part-block.txt)
- playback segments.
- streaming vs application threads.

4
docs/design/part-element-sink.txt
View file

@ -13,7 +13,7 @@ Sinks are harder to construct than other element types as they are
treated specially by the GStreamer core.
state changes
-------------
~~~~~~~~~~~~~
A sink always returns ASYNC from the state change to PAUSED, this
includes a state change from READY->PAUSED and PLAYING->PAUSED. The
@ -29,7 +29,7 @@ and take no part in the preroll procedure.
Events other than EOS do not complete the preroll stage.
sink overview
-------------
~~~~~~~~~~~~~
- TODO: PREROLL_LOCK can be removed and we can safely use the STREAM_LOCK.

39
docs/design/part-element-source.txt
View file

@ -13,7 +13,7 @@ Typical source elements include:
Live sources
------------
~~~~~~~~~~~~
A source is said to be a live source when it has the following property:
@ -52,7 +52,7 @@ Let's look at some example sources.
Source types
------------
~~~~~~~~~~~~
A source element can operate in three ways:
@ -90,36 +90,36 @@ source's state change function is called.
Source base classes
-------------------
~~~~~~~~~~~~~~~~~~~
GstBaseSrc:
This base class provides an implementation of a random access source and
is very well suited for file reader like sources.
This base class provides an implementation of a random access source and
is very well suited for file reader like sources.
GstPushSrc:
Base class for block-based sources. This class is mostly useful for
elements that cannot do random access, or at least very slowly. The
source usually prefers to push out a fixed size buffer.
Base class for block-based sources. This class is mostly useful for
elements that cannot do random access, or at least very slowly. The
source usually prefers to push out a fixed size buffer.
Classes extending this base class will usually be scheduled in a push
based mode. It the peer accepts to operate without offsets and withing
the limits of the allowed block size, this class can operate in getrange
based mode automatically.
Classes extending this base class will usually be scheduled in a push
based mode. It the peer accepts to operate without offsets and withing
the limits of the allowed block size, this class can operate in getrange
based mode automatically.
The subclass should extend the methods from the baseclass in
addition to the create method. If the source is seekable, it
needs to override GstBaseSrc::event() in addition to
GstBaseSrc::is_seekable() in order to retrieve the seek offset,
which is the offset of the next buffer to be requested.
The subclass should extend the methods from the baseclass in
addition to the create method. If the source is seekable, it
needs to override GstBaseSrc::event() in addition to
GstBaseSrc::is_seekable() in order to retrieve the seek offset,
which is the offset of the next buffer to be requested.
Flushing, scheduling and sync is all handled by this base class.
Flushing, scheduling and sync is all handled by this base class.
Timestamps
----------
~~~~~~~~~~
A non-live source should timestamp the buffers it produces starting from 0. If
it is not possible to timestamp every buffer (filesrc), the source is allowed to
@ -134,3 +134,4 @@ the pipeline, which is expressed as:
With absolute_time the time obtained from the global pipeline with
gst_clock_get_time() and base_time being the time of that clock when the
pipeline was last set to PLAYING.

127
docs/design/part-element-transform.txt
View file

@ -49,7 +49,7 @@ each of the above use cases. We focus mostly on the buffer allocation strategies
and caps negotiation.
Processing
----------
~~~~~~~~~~
A transform has 2 main processing functions:
@ -94,10 +94,10 @@ When no functions are provided, we can only process in passthrough mode.
Negotiation
-----------
~~~~~~~~~~~
Typical (re)negotiation of the transform element in push mode always goes from
sink to src, this means triggers the following sequence:
Typical (re)negotiation of the transform element in push mode always goes from
sink to src, this means triggers the following sequence:
- the sinkpad receives a buffer with new caps, this triggers the setcaps
function on the sinkpad before handing the buffer to transform.
@ -108,7 +108,7 @@ Negotiation
target src caps
- the transform processes and sets the output caps on the src pad
We call this downstream negotiation (DN) and it goes roughly like this:
We call this downstream negotiation (DN) and it goes roughly like this:
sinkpad transform srcpad
setcaps() | | |
@ -119,10 +119,10 @@ Negotiation
| <configure caps> <-| |
These steps configure the element for a transformation from the input caps to
the output caps.
These steps configure the element for a transformation from the input caps to
the output caps.
The transform has 3 function to perform the negotiation:
The transform has 3 function to perform the negotiation:
- transform_caps():
@ -140,17 +140,17 @@ Negotiation
Configure the transform for a transformation between src caps and dest
caps. Both caps are guaranteed to be fixed caps.
If no transform_caps() is defined, we can only perform the identity transform,
by default.
If no transform_caps() is defined, we can only perform the identity transform,
by default.
If no set_caps() is defined, we don't care about caps. In that case we also
assume nothing is going to write to the buffer and we don't enforce a writable
buffer for the transform_ip function, when present.
If no set_caps() is defined, we don't care about caps. In that case we also
assume nothing is going to write to the buffer and we don't enforce a writable
buffer for the transform_ip function, when present.
One common function that we need for the transform element is to find the best
transform from one format (src) to another (dest). Since the function is
bidirectional, we will use the src->dest negotiation. Some requirements of this
function are:
One common function that we need for the transform element is to find the best
transform from one format (src) to another (dest). Since the function is
bidirectional, we will use the src->dest negotiation. Some requirements of this
function are:
- has a fixed src caps
- finds a fixed dest caps that the transform element can transform to
@ -158,7 +158,7 @@ Negotiation
- the transform function prefers to make src caps == dest caps
- the transform function can optionally fixate dest caps.
The find_transform() function goes like this:
The find_transform() function goes like this:
- start from src aps, these caps are fixed.
- check if the caps are acceptable for us as src caps. This is usually
@ -177,8 +177,8 @@ Negotiation
- if we run out of caps, we fail to find a transform.
- if we found a destination caps, configure the transform with set_caps().
After this negotiation process, the transform element is usually in a steady
state. We can identify these steady states:
After this negotiation process, the transform element is usually in a steady
state. We can identify these steady states:
- src and sink pads both have the same caps. Note that when the caps are equal
on both pads, the input and output buffers automatically have the same size.
@ -266,14 +266,14 @@ Negotiation
input buffer is transformed into the output buffer. The flow is exactly
the same as the case with the same-caps negotiation. (DCC)
We can immeditatly observe that the copy transform states will need to
allocate a buffer from a downstream element using pad-alloc. When the transform
element is receiving a non-writable buffer in the in-place state, it will also
need to perform a pad-alloc. There is no reason why the passthrough state would
perform a pad-alloc. This is important because upstream re-negotiation can only
happen when the transform uses pad-alloc for all outgoing buffers.
We can immeditatly observe that the copy transform states will need to
allocate a buffer from a downstream element using pad-alloc. When the transform
element is receiving a non-writable buffer in the in-place state, it will also
need to perform a pad-alloc. There is no reason why the passthrough state would
perform a pad-alloc. This is important because upstream re-negotiation can only
happen when the transform uses pad-alloc for all outgoing buffers.
This steady state changes when one of the following actions occur:
This steady state changes when one of the following actions occur:
- the sink pad receives new caps, this triggers the above downstream
renegotation process, see above for the flow.
@ -284,9 +284,9 @@ Negotiation
for example). This essentially clears the current steady state and
triggers the downstream and upstream renegotiation process.
Parallel to the downstream negotiation process there is an upstream negotiation
process. The handling and proxy of buffer-alloc is the most comple part of the
transform element. This upstream negotiation process has 3 cases: (UN)
Parallel to the downstream negotiation process there is an upstream negotiation
process. The handling and proxy of buffer-alloc is the most comple part of the
transform element. This upstream negotiation process has 3 cases: (UN)
- upstream calls the buffer-alloc function of the transform sinkpad and this
call is proxied downstream (UNP)
@ -295,13 +295,13 @@ Negotiation
- the transform calls the pad-alloc function downstream to allocate a new
output buffer (but not because of a proxied buffer-alloc) (UNA)
The case where the pad-alloc is called because an output buffer must be
generated in the chain function is handled above in the copy-transform and the
in-place transform when the input buffer is not writable or the input buffer
size is smaller than the output size.
The case where the pad-alloc is called because an output buffer must be
generated in the chain function is handled above in the copy-transform and the
in-place transform when the input buffer is not writable or the input buffer
size is smaller than the output size.
We are left with the last case (proxy an incomming pad-alloc or not). We have 2
possibilities here:
We are left with the last case (proxy an incomming pad-alloc or not). We have 2
possibilities here:
- pad-alloc is called with the same caps as are currently being handled by
the transform on the sinkcaps. Note that this will only be true when the
@ -400,14 +400,14 @@ Negotiation
<----------------------------------| |
| | |
In order to perform passthrough buffer-alloc or pad-alloc, we need to be able
to get the size of the output buffer after the transform.
For passthrough buffer-alloc, this is trivial: the input size equals the output
size.
In order to perform passthrough buffer-alloc or pad-alloc, we need to be able
to get the size of the output buffer after the transform.
For the copy transform or the in-place transform we need additional function to
retrieve the size. There are two functions:
For passthrough buffer-alloc, this is trivial: the input size equals the output
size.
For the copy transform or the in-place transform we need additional function to
retrieve the size. There are two functions:
- transform_size()
@ -426,35 +426,35 @@ Negotiation
Issues
------
~~~~~~
passthrough and in-place transforms (with writable buffers) never need to
perform a pad-alloc on the srcpad. This means that if upstream negotiation
happens, the transform element will never know about it.
passthrough and in-place transforms (with writable buffers) never need to
perform a pad-alloc on the srcpad. This means that if upstream negotiation
happens, the transform element will never know about it.
The transform element will keep therefore track of the allocation pattern of
the peer elements. We can see the following cases:
The transform element will keep therefore track of the allocation pattern of
the peer elements. We can see the following cases:
- upstream peer calls buffer-alloc on the sinkpad of the transform. In some
cases (see above) this call gets proxied or not.
- upstream peer calls buffer-alloc on the sinkpad of the transform. In some
cases (see above) this call gets proxied or not.
- upstream peer does never call buffer-alloc.
- upstream peer does never call buffer-alloc.
We will keeps state about this allocation pattern and perform the following in
each case respectively:
We will keeps state about this allocation pattern and perform the following in
each case respectively:
- Upstream calls buffer-alloc: In passthrough and (some) in-place we proxy
this call onto the downstream element. If the caps are changed, we mark
a flag that we will require a new pad-alloc for the output of the next
output buffer.
- Upstream calls buffer-alloc: In passthrough and (some) in-place we proxy
this call onto the downstream element. If the caps are changed, we mark
a flag that we will require a new pad-alloc for the output of the next
output buffer.
- upstream peer does not call buffer-alloc: We always perform a pad-alloc
when processing buffers. We can further optimize by only looking at the
returned caps instead of doing a full, needless buffer copy.
- upstream peer does not call buffer-alloc: We always perform a pad-alloc
when processing buffers. We can further optimize by only looking at the
returned caps instead of doing a full, needless buffer copy.
Use cases
---------
~~~~~~~~~
videotestsrc ! ximagesink
@ -478,7 +478,8 @@ Use cases
- resizing the videosink makes videoscale perform the scaling.
Problematic
-----------
~~~~~~~~~~~
filesrc location=~/media/moveyourfeet.mov ! decodebin !
ffmpegcolorspace ! videoscale ! ffmpegcolorspace ! ximagesink -v

26
docs/design/part-events.txt
View file

@ -27,7 +27,7 @@ Different types of events exist to implement various functionalities.
in DVD.
FLUSH_START/STOP
----------------
~~~~~~~~~~~~~~~~
A flush event is sent both downstream and upstream to clear any pending data
from the pipeline. This might be needed to make the graph more responsive
@ -65,7 +65,7 @@ base_time (see part-clocks.txt and part-synchronisation.txt).
EOS
---
~~~
The EOS event can only be sent on a sinkpad. It is typically emited by the
source element when it has finished sending data. This event is mainly sent
@ -107,7 +107,7 @@ A FLUSH_STOP event on an element flushes the EOS state and all pending EOS messa
NEWSEGMENT
-------------
~~~~~~~~~~
A newsegment event is sent downstream by an element to indicate that the following
group of buffers start and end at the specified positions. The newsegment event
@ -146,7 +146,7 @@ make the buffer timestamps increasing (part-segments.txt).
TAG
---
~~~
The tag event is sent downstream when an element has discovered metadata
tags in a media file. Encoders can use this event to adjust their tagging
@ -154,15 +154,17 @@ system. A tag is serialized with buffers.
BUFFERSIZE
----------
~~~~~~~~~~
NOTE: This event is not yet implemented.
An element can suggest a buffersize for downstream elements. This is
typically done by elements that produce data on multiple source pads
such as demuxers. This event is currently not yet defined nor used.
such as demuxers.
QOS
---
~~~
A QOS, or quality of service message, is generated in an element to report
to the upstream elements about the current quality of real-time performance
@ -171,7 +173,7 @@ of framedrops they have. (see part-qos.txt)
SEEK
----
~~~~
A seek event is issued by the application to configure the playback range
of a stream. It is called form the application thread and travels upstream.
@ -227,7 +229,7 @@ part-seeking.txt.
NAVIGATION
----------
~~~~~~~~~~~
A navigation event is generated by a sink element to signal the elements
of a navigation event such as a mouse movement or button click.
@ -235,7 +237,7 @@ Navigation events travel upstream.
LATENCY
-------
~~~~~~~
A latency event is used to configure a certain latency in the pipeline. It
contains a single GstClockTime with the required latency. The latency value is
@ -246,9 +248,9 @@ timestamps of the buffer in order to delay their presentation.
DRAIN
-----
~~~~~
This event is not yet implemented.
NOTE: This event is not yet implemented.
Drain event indicates that upstream is about to perform a real-time event, such
as pausing to present an interactive menu or such, and needs to wait for all

77
docs/design/part-framestep.txt
View file

@ -26,7 +26,7 @@ segments of data in PLAYING.
Use Cases
---------
~~~~~~~~~
* frame stepping in video only pipeline in PAUSED
@ -92,10 +92,10 @@ Use Cases
events
------
~~~~~~
A new GST_EVENT_STEP event is introduced to start the step operation.
The step event is created with the following fields in the structure:
A new GST_EVENT_STEP event is introduced to start the step operation.
The step event is created with the following fields in the structure:
"format", GST_TYPE_FORMAT
The format of the step units
@ -146,29 +146,29 @@ events
flag is passed to the corresponding GST_MESSAGE_STEP_DONE.
The application will create a STEP event to start or stop the stepping
operation. Both stepping in PAUSED and PLAYING can be performed by means of
the flush flag.
The application will create a STEP event to start or stop the stepping
operation. Both stepping in PAUSED and PLAYING can be performed by means of
the flush flag.
The event is usually sent to the pipeline, which will typically distribute the
event to all of its sinks. For some use cases, like frame stepping on video
frames only, the event should only be sent to the video sink and upon reception
of the STEP_DONE message, one can step the other sinks to align the streams
again.
The event is usually sent to the pipeline, which will typically distribute the
event to all of its sinks. For some use cases, like frame stepping on video
frames only, the event should only be sent to the video sink and upon reception
of the STEP_DONE message, one can step the other sinks to align the streams
again.
For large stepping amounts, there needs to be enough queueing in front of all
the sinks. If large steps need to be performed, they can be split up into
smaller step operations using the "intermediate" flag on the step.
For large stepping amounts, there needs to be enough queueing in front of all
the sinks. If large steps need to be performed, they can be split up into
smaller step operations using the "intermediate" flag on the step.
Since the step event does not update the base_time of any of the elements, the
sinks should keep track of the amount of stepped data in order to remain
synchronized against the clock.
Since the step event does not update the base_time of any of the elements, the
sinks should keep track of the amount of stepped data in order to remain
synchronized against the clock.
messages
--------
~~~~~~~~
A GST_MESSAGE_STEP_START is created. It contains the following fields.
A GST_MESSAGE_STEP_START is created. It contains the following fields.
"active"
If the step was queued or activated.
@ -188,7 +188,7 @@ messages
"intermediate", G_TYPE_BOOLEAN
If this is an intermediate step operation that queued/activated.
The STEP_START message is emited 2 times:
The STEP_START message is emited 2 times:
* first when an element received the STEP event and queued it. The "active"
field will be FALSE in this case.
@ -197,12 +197,12 @@ messages
field is TRUE in this case. After this message is emited, the application
can queue a new step operation.
The purpose of this message is to find out how many elements participate in the
step operation and to queue new step operations at the earliest possible
moment.
The purpose of this message is to find out how many elements participate in the
step operation and to queue new step operations at the earliest possible
moment.
A new GST_MESSAGE_STEP_DONE message is created. It contains the following
fields:
A new GST_MESSAGE_STEP_DONE message is created. It contains the following
fields:
"format", GST_TYPE_FORMAT
The format of the step units that completed.
@ -225,25 +225,24 @@ messages
"eos", G_TYPE_BOOLEAN
The step ended because of EOS.
The message is emited by the element that performs the step operation. The
purpose is to return the duration in GST_FORMAT_TIME of the stepped media. This
especially interesting to align other stream in case of stepping frames on the
video sink element.
The message is emited by the element that performs the step operation. The
purpose is to return the duration in GST_FORMAT_TIME of the stepped media. This
especially interesting to align other stream in case of stepping frames on the
video sink element.
Direction switch
----------------
~~~~~~~~~~~~~~~~
When quickly switching between a forwards and a backwards step of, for example,
one video frame, we need either:
When quickly switching between a forwards and a backwards step of, for example,
one video frame, we need either:
a) issue a new seek to change the direction from the current position.
b) cache a certain number of stepped frames and walk the cache.
option a) might be very slow.
For option b) we would ideally like to offload this caching functionality to a
separate element, which means that we need to forward the STEP event upstream.
It's unclear how this could work in a generic way. What is a demuxer supposed
to do when it received a step event? a flushing seek to what stream position?
option a) might be very slow.
For option b) we would ideally like to offload this caching functionality to a
separate element, which means that we need to forward the STEP event upstream.
It's unclear how this could work in a generic way. What is a demuxer supposed
to do when it received a step event? a flushing seek to what stream position?

16
docs/design/part-gstbin.txt
View file

@ -7,7 +7,7 @@ GstElement. A GstBin provides a GstBus for the children and collates messages
from them.
Add/removing elements
---------------------
~~~~~~~~~~~~~~~~~~~~~
The basic functionality of a bin is to add and remove GstElements to/from it.
gst_bin_add() and gst_bin_remove() perform these operations respectively.
@ -17,7 +17,7 @@ relations.txt).
Retrieving elements
-------------------
~~~~~~~~~~~~~~~~~~~
GstBin provides a number of functions to retrieve one or more children from
itself. A few examples of the provided functions:
@ -28,7 +28,7 @@ gst_bin_iterate_elements() returns an iterator to all the children.
element management
------------------
~~~~~~~~~~~~~~~~~~
The most important function of the GstBin is to distribute all GstElement
operations on itself to all of its children. This includes:
@ -41,7 +41,7 @@ The state change distribution is the most complex and is explained in
part-states.txt.
GstBus
------
~~~~~~
The GstBin creates a GstBus for its children and distributes it when child
elements are added to the bin. The bin attaches a sync handler to receive
@ -63,7 +63,7 @@ When a bin goes to READY it will clear all cached messages.
EOS
---
~~~
The sink elements will post an EOS message on the bus when they reach EOS. The
EOS message is only posted to the bus when the sink element is in PLAYING.
@ -77,7 +77,7 @@ to PLAYING again.
SEGMENT_START/DONE
------------------
~~~~~~~~~~~~~~~~~~
A bin collects SEGMENT_START messages but does not post them to the application.
It counts the number of SEGMENT_START messages and posts a SEGMENT_STOP message
@ -87,7 +87,7 @@ The cached SEGMENT_START/STOP messages are cleared when going to READY.
DURATION
--------
~~~~~~~~
When a DURATION query is performed on a bin, it will forward the query to all
its sink elements. The bin will calculate the total duration as the MAX of all
@ -103,7 +103,7 @@ posted to the application, which can then fetch the updated DURATION.
Subclassing
-----------
~~~~~~~~~~~
Subclasses of GstBin are free to implement their own add/remove implementations.
It is a good idea to update the GList of children so that the _iterate() functions

1
docs/design/part-gstbus.txt
View file

@ -38,3 +38,4 @@ to flushing, ie. the bus will drop all existing and new messages on the bus,
This is necessary because bus messages hold references to the bin/pipeline
or its elements, so there are circular references that need to be broken if
one ever wants to be able to destroy a bin or pipeline properly.

21
docs/design/part-gstelement.txt
View file

@ -1,14 +1,5 @@
gstelement
name
pads
state
loopfunction
threadstate
manager
ghostpads
GstElement
==========
----------
The Element is the most important object in the entire GStreamer system, as it
defines the structure of the pipeline. Elements include sources, filters,
@ -26,7 +17,7 @@ Element. This allows deeply nested pipelines, and the possibility of
"black-box" meta-elements.
Name
----
~~~~
All elements are named, and while they should ideally be unique in any given
pipeline, they do not have to be. The only guaranteed unique name for an
@ -39,7 +30,7 @@ or name of an element is changed.
Pads
----
~~~~
GstPads are the property of a given GstElement. They provide the connection
capability, with allowing arbitrary structure in the graph. For any Element
@ -67,10 +58,12 @@ pad = gst_element_get_pad(element,"padname"):
Ghost Pads
----------
~~~~~~~~~~
More info in part-gstghostpad.txt.
State
-----
~~~~~
An element has a state. More info in part-states.txt.

41
docs/design/part-gstghostpad.txt
View file

@ -1,11 +1,11 @@
Ghostpads
---------
GhostPads are used to build complex compound elements out of
existing elements. They are used to expose internal element pads
on the complex element.
GhostPads are used to build complex compound elements out of
existing elements. They are used to expose internal element pads
on the complex element.
Some design requirements
Some design requirements
- Must look like a real GstPad on both sides.
- target of Ghostpad must be changeable
@ -346,23 +346,22 @@ Ghostpads
Activation
==========
~~~~~~~~~~
Sometimes ghost pads should proxy activation functions. This thingie
attempts to explain how it should work in the different cases.
+---+ +----+ +----+ +----+
| A +-----+ B | | C |-------+ D |
+---+ +---=+ +=---+ +----+
+--=-----------------------------=-+
| +=---+ +----+ +----+ +---=+ |
| | a +---+ b ==== c +--+ d | |
| +----+ +----+ +----+ +----+ |
| |
+----------------------------------+
state change goes from right to left
<-----------------------------------------------------------
+---+ +----+ +----+ +----+
| A +-----+ B | | C |-------+ D |
+---+ +---=+ +=---+ +----+
+--=-----------------------------=-+
| +=---+ +----+ +----+ +---=+ |
| | a +---+ b ==== c +--+ d | |
| +----+ +----+ +----+ +----+ |
| |
+----------------------------------+
state change goes from right to left
<-----------------------------------------------------------
All of the labeled boxes are pads. The dashes (---) show pad links, and
the double-lines (===) are internal connections. The box around a, b, c,
@ -375,10 +374,10 @@ Now, in the state change from READY to PAUSED, assuming the pipeline
does not have a live source, all of the pads will end up activated at
the end. There are 4 possible activation modes:
1) AD and ab in PUSH, cd and CD in PUSH
2) AD and ab in PUSH, cd and CD in PULL
3) AD and ab in PULL, cd and CD in PUSH
4) AD and ab in PULL, cd and CD in PULL
1) AD and ab in PUSH, cd and CD in PUSH
2) AD and ab in PUSH, cd and CD in PULL
3) AD and ab in PULL, cd and CD in PUSH
4) AD and ab in PULL, cd and CD in PULL
When activating (1), the state change algorithm will first visit the
parent of D and activate D in push mode. Then it visits the bin. The bin

17
docs/design/part-gstobject.txt
View file

@ -1,10 +1,10 @@
GstObject
=========
---------
The base class for the entire GStreamer hierarchy is the GstObject.
Parentage
---------
~~~~~~~~~
A pointer is available to store the current parent of the object. This is one
of the two fundamental requirements for a hierarchical system such as GStreamer
@ -20,7 +20,8 @@ allows for new additions later.
Naming
------
~~~~~~
- names of objects cannot be changed when they are parented
- names of objects should be unique across parent
- set_name() can fail because of this
@ -36,7 +37,7 @@ Naming
Locking
-------
~~~~~~~
The GstObject contains the necessary primitives to lock the object in a
thread-safe manner. This will be used to provide general thread-safety as
@ -62,7 +63,7 @@ GstObject.
Locking order
-------------
~~~~~~~~~~~~~
In parent-child situations the lock of the parent must always be taken first
before taking the lock of the child. It is NOT allowed to hold the child
@ -76,7 +77,7 @@ parent-child relation (eg. pads), an explictic locking order has to be defined.
Path Generation
---------------
~~~~~~~~~~~~~~~
Due to the base nature of the GstObject, it becomes the only reasonable place
to put this particular function (_get_path_string). It will generate a string
@ -84,7 +85,7 @@ describing the parent hierarchy of a given GstObject.
Flags
-----
~~~~~
Each object in the GStreamer object hierarchy can have flags associated with it,
which are used to describe a state or a feature of the object.
@ -92,7 +93,7 @@ GstObject has flags to mark its lifecycle: FLOATING and DISPOSING.
Class signals
-------------
~~~~~~~~~~~~~
It is possible to know when a new object is loaded by connecting to the
GstObjectClass signal. This feature is not very much used and might be removed

6
docs/design/part-gstpipeline.txt
View file

@ -13,7 +13,7 @@ The pipeline will calculate a global latency for the elements in the pipeline.
(See also part-latency.txt).
State changes
-------------
~~~~~~~~~~~~~
In addition to the normal state change procedure of its parent class
GstBin, the pipeline performs the following actions during a state change:
@ -43,7 +43,7 @@ The running_time is set to 0 after a flushing seek.
Clock selection
---------------
~~~~~~~~~~~~~~~
Since all of the children of a GstPipeline must use the same clock, the
pipeline must select a clock. This clock selection happens when the pipeline
@ -80,7 +80,7 @@ matic clock selection algorithm described above.
GstBus
------
~~~~~~
A GstPipeline provides a GstBus to the application. The bus can be retrieved
with gst_pipeline_get_bus() and can then be used to retrieve messages posted by

10
docs/design/part-latency.txt
View file

@ -35,7 +35,7 @@ application needs (required minimal latency).
Pipelines without latency compensation
--------------------------------------
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
We show some examples to demonstrate the problem of latency in typical
capture pipelines.
@ -196,7 +196,7 @@ capture pipelines.
State Changes revised
---------------------
~~~~~~~~~~~~~~~~~~~~~
As a first step in a generic solution we propose to modify the state changes so
that no sink is set to PLAYING before it is prerolled.
@ -229,7 +229,7 @@ implications:
Latency compensation
--------------------
~~~~~~~~~~~~~~~~~~~~
As an extension to the revised state changes we can perform latency calculation
and compensation before we proceed to the PLAYING state.
@ -277,7 +277,7 @@ the same for all sinks, all sinks will render data relatively synchronised.
Flushing a playing pipeline
---------------------------
~~~~~~~~~~~~~~~~~~~~~~~~~~~
Using the new state change mechanism we can implement resynchronisation after an
uncontrolled FLUSH in (part of) a pipeline. Indeed, when a flush is performed on
@ -306,7 +306,7 @@ to perform additional latency calculations and adjustments before doing this.
Dynamically adjusting latency
-----------------------------
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
An element that want to change the latency in the pipeline can do this by
posting a LATENCY message on the bus. This message instructs the pipeline to:

6
docs/design/part-live-source.txt
View file

@ -28,14 +28,14 @@ a blocking wait.
Scheduling
----------
~~~~~~~~~~
Live sources will not produce data in the paused state. They block in the
getrange function or in the loop function until they go to PLAYING.
Latency
-------
~~~~~~~
The live source timestamps its data with the time of the clock at the
time the data was captured. Normally it will take some time to capture
@ -49,7 +49,7 @@ See part-latency.txt.
Timestamps
----------
~~~~~~~~~~
Live sources always timestamp their buffers with the running_time of the
pipeline. This is needed to be able to match the timestamps of different live

195
docs/design/part-messages.txt
View file

@ -14,122 +14,123 @@ application using the GstBus (See also part-gstbus.txt and part-gstpipeline.txt)
Message types
-------------
~~~~~~~~~~~~~
GST_MESSAGE_EOS:
GST_MESSAGE_EOS:
Posted by sink elements. This message is posted to the application when
all the sinks in a pipeline posted an EOS message. When performing a seek,
the EOS state of the pipeline and sinks is undone.
Posted by sink elements. This message is posted to the application when
all the sinks in a pipeline posted an EOS message. When performing a seek,
the EOS state of the pipeline and sinks is undone.
GST_MESSAGE_ERROR:
An element in the pipeline got into an error state. The message carries
a GError and a debug string describing the error. This usually means that
part of the pipeline is not streaming anymore.
GST_MESSAGE_WARNING:
An element in the pipeline encountered a condition that made it produce a
warning. This could be a recoverable decoding error or some other non fatal
event. The pipeline continues streaming after a warning.
GST_MESSAGE_INFO:
An element produced an informational message.
GST_MESSAGE_TAG:
An element decoded metadata about the stream. The message carries a GstTagList
with the tag information.
GST_MESSAGE_BUFFERING:
An element is buffering data and that could potentially take some time. This
message is typically emited by elements that perform some sort of network
buffering. While the pipeline is buffering it should remain in the PAUSED
state. When the buffering is finished, it can resume PLAYING.
GST_MESSAGE_STATE_CHANGED:
An element changed state in the pipeline. The message carries the old, new
and pending state of the element.
GST_MESSAGE_STATE_DIRTY:
GST_MESSAGE_ERROR:
An internal message used to instruct a pipeline hierarchy that a state
recalculation must be performed because of an ASYNC state change completed.
This message is not used anymore.
GST_MESSAGE_STEP_DONE:
An element stepping frames has finished. This is currently not used.
GST_MESSAGE_CLOCK_PROVIDE:
An element notifies it capability of providing a clock for the pipeline.
GST_MESSAGE_CLOCK_LOST:
The current clock as selected by the pipeline became unusable. The pipeline
will select a new clock on the next PLAYING state change.
GST_MESSAGE_NEW_CLOCK:
A new clock was selected for the pipeline.
GST_MESSAGE_STRUCTURE_CHANGE:
The pipeline changed of structure, This means elements were added or removed or
pads were linked or unlinked. This messages is not yet used.
GST_MESSAGE_STREAM_STATUS:
Posted by an element when it start/stop/pauses a streaming task. It
contains information about the reason why the stream state changed along
with the thread id. The application can use this information to detect
failures in streaming threads. It can also be used to adjust streaming
thread priorities by the application.
An element in the pipeline got into an error state. The message carries
a GError and a debug string describing the error. This usually means that
part of the pipeline is not streaming anymore.
GST_MESSAGE_WARNING:
An element in the pipeline encountered a condition that made it produce a
warning. This could be a recoverable decoding error or some other non fatal
event. The pipeline continues streaming after a warning.
GST_MESSAGE_INFO:
GST_MESSAGE_APPLICATION:
An element produced an informational message.
GST_MESSAGE_TAG:
The application posted a message. This message must be used when the
application posts a message on the bus.
An element decoded metadata about the stream. The message carries a GstTagList
with the tag information.
GST_MESSAGE_BUFFERING:
An element is buffering data and that could potentially take some time. This
message is typically emited by elements that perform some sort of network
buffering. While the pipeline is buffering it should remain in the PAUSED
state. When the buffering is finished, it can resume PLAYING.
GST_MESSAGE_STATE_CHANGED:
GST_MESSAGE_ELEMENT:
An element changed state in the pipeline. The message carries the old, new
and pending state of the element.
GST_MESSAGE_STATE_DIRTY:
An internal message used to instruct a pipeline hierarchy that a state
recalculation must be performed because of an ASYNC state change completed.
This message is not used anymore.
Element-specific message, see the specific element's documentation
GST_MESSAGE_STEP_DONE:
GST_MESSAGE_SEGMENT_START:
An element stepping frames has finished. This is currently not used.
GST_MESSAGE_CLOCK_PROVIDE:
An element started playback of a new segment. This message is not forwarded
the the application but is used internally to schedule SEGMENT_DONE messages.
An element notifies it capability of providing a clock for the pipeline.
GST_MESSAGE_SEGMENT_DONE:
GST_MESSAGE_CLOCK_LOST:
An element or bin completed playback of a segment. This message is only posted
on the bus if a SEGMENT seek is performed on a pipeline.
The current clock as selected by the pipeline became unusable. The pipeline
will select a new clock on the next PLAYING state change.
GST_MESSAGE_DURATION:
GST_MESSAGE_NEW_CLOCK:
An element posts this message when it has detected or updated the stream duration.
A new clock was selected for the pipeline.
GST_MESSAGE_ASYNC_START:
GST_MESSAGE_STRUCTURE_CHANGE:
Posted by sinks when they start an asynchronous state change.
The pipeline changed of structure, This means elements were added or removed or
pads were linked or unlinked. This messages is not yet used.
GST_MESSAGE_ASYNC_DONE:
GST_MESSAGE_STREAM_STATUS:
Posted by sinks when they receive the first data buffer and complete the
asynchronous state change.
Posted by an element when it start/stop/pauses a streaming task. It
contains information about the reason why the stream state changed along
with the thread id. The application can use this information to detect
failures in streaming threads. It can also be used to adjust streaming
thread priorities by the application.
GST_MESSAGE_APPLICATION:
GST_MESSAGE_LATENCY:
The application posted a message. This message must be used when the
application posts a message on the bus.
Posted by elements when the latency in a pipeline changed and a new global
latency should be calculated by the pipeline or application.
GST_MESSAGE_ELEMENT:
GST_MESSAGE_REQUEST_STATE:
Element-specific message, see the specific element's documentation
Posted by elements when they want to change the state of the pipeline they
are in. A typical use case would be an audio sink that requests the pipeline
to pause in order to play a higher priority stream.
GST_MESSAGE_SEGMENT_START:
An element started playback of a new segment. This message is not forwarded
the the application but is used internally to schedule SEGMENT_DONE messages.
GST_MESSAGE_SEGMENT_DONE:
An element or bin completed playback of a segment. This message is only posted
on the bus if a SEGMENT seek is performed on a pipeline.
GST_MESSAGE_DURATION:
An element posts this message when it has detected or updated the stream duration.
GST_MESSAGE_ASYNC_START:
Posted by sinks when they start an asynchronous state change.
GST_MESSAGE_ASYNC_DONE:
Posted by sinks when they receive the first data buffer and complete the
asynchronous state change.
GST_MESSAGE_LATENCY:
Posted by elements when the latency in a pipeline changed and a new global
latency should be calculated by the pipeline or application.
GST_MESSAGE_REQUEST_STATE:
Posted by elements when they want to change the state of the pipeline they
are in. A typical use case would be an audio sink that requests the pipeline
to pause in order to play a higher priority stream.

2
docs/design/part-missing-plugins.txt
View file

@ -1,5 +1,5 @@
What to do when a plugin is missing
===================================
-----------------------------------
The mechanism and API described in this document requires GStreamer core and
gst-plugins-base versions >= 0.10.12. Further information on some aspects of

9
docs/design/part-negotiation.txt
View file

@ -13,7 +13,7 @@ themselves to different mechanisms to achieve this goal. As it is more
common we describe push mode negotiation first.
Push-mode negotiation
---------------------
~~~~~~~~~~~~~~~~~~~~~
Push-mode negotiation happens when elements want to push buffers and
need to decide on the format. This is called downstream negotiation
@ -168,10 +168,10 @@ videotestsrc ! queue ! xvimagesink
Pull-mode negotiation
---------------------
~~~~~~~~~~~~~~~~~~~~~
Rationale
.........
^^^^^^^^^
A pipeline in pull mode has different negotiation needs than one
activated in push mode. Push mode is optimized for two use cases:
@ -237,7 +237,7 @@ but before the sink has spawned the pulling thread.
Mechanism
.........
^^^^^^^^^
The sink determines that the upstream elements support pull based scheduling by
calling gst_pad_check_pull_range().
@ -276,3 +276,4 @@ GST_FLOW_NOT_NEGOTIATED. Because of the low-latency requirements,
changing caps in an activate pull-mode pipeline is not supported, as it
might require e.g. the sound card to reconfigure its hardware buffers,
and start capsnego again.

688
docs/design/part-overview.txt
View file

@ -1,32 +1,32 @@
Overview
--------
This part gives an overview of the design of GStreamer with references to
the more detailed explanations of the different topics.
This part gives an overview of the design of GStreamer with references to
the more detailed explanations of the different topics.
This document is intented for people that want to have a global overview of
the inner workings of GStreamer.
This document is intented for people that want to have a global overview of
the inner workings of GStreamer.
Introduction
------------
~~~~~~~~~~~~
GStreamer is a set of libraries and plugins that can be used to implement various
multimedia applications ranging from desktop players, audio/video recorders,
multimedia servers, transcoders, etc.
GStreamer is a set of libraries and plugins that can be used to implement various
multimedia applications ranging from desktop players, audio/video recorders,
multimedia servers, transcoders, etc.
Applications are built by constructing a pipeline composed of elements. An element
is an object that performs some action on a multimedia stream such as:
Applications are built by constructing a pipeline composed of elements. An element
is an object that performs some action on a multimedia stream such as:
- read a file
- decode or encode between formats
- capture from a hardware device
- render to a hardware device
- mix or multiplex multiple streams
- read a file
- decode or encode between formats
- capture from a hardware device
- render to a hardware device
- mix or multiplex multiple streams
Elements have input and output pads called sink and source pads in GStreamer. An
application links elements together on pads to construct a pipeline. Below is
an example of an ogg/vorbis playback pipeline.
Elements have input and output pads called sink and source pads in GStreamer. An
application links elements together on pads to construct a pipeline. Below is
an example of an ogg/vorbis playback pipeline.
+-----------------------------------------------------------+
| ----------> downstream -------------------> |
@ -40,54 +40,54 @@ Introduction
| <---------< upstream <-------------------< |
+-----------------------------------------------------------+
The filesrc element reads data from a file on disk. The oggdemux element parses
the data and sends the compressed audio data to the vorbisdec element. The
vorbisdec element decodes the compressed data and sends it to the alsasink
element. The alsasink element sends the samples to the audio card for playback.
The filesrc element reads data from a file on disk. The oggdemux element parses
the data and sends the compressed audio data to the vorbisdec element. The
vorbisdec element decodes the compressed data and sends it to the alsasink
element. The alsasink element sends the samples to the audio card for playback.
Downstream and upstream are the terms used to describe the direction in the
Pipeline. From source to sink is called "downstream" and "upstream" is
from sink to source. Dataflow always happens downstream.
Downstream and upstream are the terms used to describe the direction in the
Pipeline. From source to sink is called "downstream" and "upstream" is
from sink to source. Dataflow always happens downstream.
The task of the application is to construct a pipeline as above using existing
elements. This is further explained in the pipeline building topic.
The task of the application is to construct a pipeline as above using existing
elements. This is further explained in the pipeline building topic.
The application does not have to manage any of the complexities of the
actual dataflow/decoding/conversions/synchronsiation etc. but only calls high
level functions on the pipeline object such as PLAY/PAUSE/STOP.
The application does not have to manage any of the complexities of the
actual dataflow/decoding/conversions/synchronsiation etc. but only calls high
level functions on the pipeline object such as PLAY/PAUSE/STOP.
The application also receives messages and notifications from the pipeline such
as metadata, warning, error and EOS messages.
If the application needs more control over the graph it is possible to directly
access the elements and pads in the pipeline.
The application also receives messages and notifications from the pipeline such
as metadata, warning, error and EOS messages.
If the application needs more control over the graph it is possible to directly
access the elements and pads in the pipeline.
Design overview
---------------
~~~~~~~~~~~~~~~
GStreamer design goals include:
GStreamer design goals include:
- Process large amounts of data quickly
- Allow fully multithreaded processing
- Ability to deal with multiple formats
- Synchronize different dataflows
- Ability to deal with multiple devices
- Process large amounts of data quickly
- Allow fully multithreaded processing
- Ability to deal with multiple formats
- Synchronize different dataflows
- Ability to deal with multiple devices
The capabilities presented to the application depends on the number of elements
installed on the system and their functionality.
The GStreamer core is designed to be media agnostic but provides many features
to elements to describe media formats.
The capabilities presented to the application depends on the number of elements
installed on the system and their functionality.
The GStreamer core is designed to be media agnostic but provides many features
to elements to describe media formats.
Elements
--------
~~~~~~~~
The smallest building blocks in a pipeline are elements. An element provides a
number of pads which can be source or sinkpads. Sourcepads provide data and
sinkpads consume data. Below is an example of an ogg demuxer element that has
one pad that takes (sinks) data and two source pads that produce data.
The smallest building blocks in a pipeline are elements. An element provides a
number of pads which can be source or sinkpads. Sourcepads provide data and
sinkpads consume data. Below is an example of an ogg demuxer element that has
one pad that takes (sinks) data and two source pads that produce data.
+-----------+
| oggdemux |
@ -95,57 +95,57 @@ Elements
sink src1
+-----------+
An element can be in four different states: NULL, READY, PAUSED, PLAYING. In the
NULL and READY state, the element is not processing any data. In the PLAYING state
it is processing data. The intermediate PAUSED state is used to preroll data in
the pipeline. A state change can be performed with gst_element_set_state().
An element can be in four different states: NULL, READY, PAUSED, PLAYING. In the
NULL and READY state, the element is not processing any data. In the PLAYING state
it is processing data. The intermediate PAUSED state is used to preroll data in
the pipeline. A state change can be performed with gst_element_set_state().
An element always goes through all the intermediate state changes. This means that
when en element is in the READY state and is put to PLAYING, it will first go
through the intermediate PAUSED state.
An element always goes through all the intermediate state changes. This means that
when en element is in the READY state and is put to PLAYING, it will first go
through the intermediate PAUSED state.
An element state change to PAUSED will activate the pads of the element. First the
source pads are activated, then the sinkpads. When the pads are activated, the
pad activate function is called. Some pads will start a thread (GstTask) or some
other mechanism to start producing or consuming data.
An element state change to PAUSED will activate the pads of the element. First the
source pads are activated, then the sinkpads. When the pads are activated, the
pad activate function is called. Some pads will start a thread (GstTask) or some
other mechanism to start producing or consuming data.
The PAUSED state is special as it is used to preroll data in the pipeline. The purpose
is to fill all connected elements in the pipeline with data so that the subsequent
PLAYING state change happens very quickly. Some elements will therefore not complete
the state change to PAUSED before they have received enough data. Sink elements are
required to only complete the state change to PAUSED after receiving the first data.
The PAUSED state is special as it is used to preroll data in the pipeline. The purpose
is to fill all connected elements in the pipeline with data so that the subsequent
PLAYING state change happens very quickly. Some elements will therefore not complete
the state change to PAUSED before they have received enough data. Sink elements are
required to only complete the state change to PAUSED after receiving the first data.
Normally the state changes of elements are coordinated by the pipeline as explained
in [part-states.txt].
Normally the state changes of elements are coordinated by the pipeline as explained
in [part-states.txt].
Different categories of elements exist:
Different categories of elements exist:
- source elements, these are elements that do not consume data but only provide data
for the pipeline.
- sink elements, these are elements that do not produce data but renders data to
an output device.
- transform elements, these elements transform an input stream in a certain format
into a stream of another format. Encoder/decoder/converters are examples.
- demuxer elements, these elements parse a stream and produce several output streams.
- mixer/muxer elements, combine several input streams into one output stream.
Other categories of elements can be constructed (see part-klass.txt).
- source elements, these are elements that do not consume data but only provide data
for the pipeline.
- sink elements, these are elements that do not produce data but renders data to
an output device.
- transform elements, these elements transform an input stream in a certain format
into a stream of another format. Encoder/decoder/converters are examples.
- demuxer elements, these elements parse a stream and produce several output streams.
- mixer/muxer elements, combine several input streams into one output stream.
Other categories of elements can be constructed (see part-klass.txt).
Bins
----
~~~~
A bin is an element subclass and acts as a container for other elements so that multiple
elements can be combined into one element.
A bin is an element subclass and acts as a container for other elements so that multiple
elements can be combined into one element.
A bin coordinates its children's state changes as explained later. It also distributes
events and various other functionality to elements.
A bin coordinates its children's state changes as explained later. It also distributes
events and various other functionality to elements.
A bin can have its own source and sinkpads by ghostpadding one or more of its children's
pads to itself.
A bin can have its own source and sinkpads by ghostpadding one or more of its children's
pads to itself.
Below is a picture of a bin with two elements. The sinkpad of one element is ghostpadded
to the bin.
Below is a picture of a bin with two elements. The sinkpad of one element is ghostpadded
to the bin.
+---------------------------+
| bin |
@ -157,131 +157,131 @@ Bins
Pipeline
--------
~~~~~~~~
A pipeline is a special bin subclass that provides the following features to its
children:
A pipeline is a special bin subclass that provides the following features to its
children:
- Select and manage a global clock for all its children.
- Manage running_time based on the selected clock. Running_time is the elapsed
time the pipeline spent in the PLAYING state and is used for
synchronisation.
- Manage latency in the pipeline.
- Provide means for elements to comunicate with the application by the GstBus.
- Manage the global state of the elements such as Errors and end-of-stream.
Normally the application creates one pipeline that will manage all the elements
in the application.
- Select and manage a global clock for all its children.
- Manage running_time based on the selected clock. Running_time is the elapsed
time the pipeline spent in the PLAYING state and is used for
synchronisation.
- Manage latency in the pipeline.
- Provide means for elements to comunicate with the application by the GstBus.
- Manage the global state of the elements such as Errors and end-of-stream.
Normally the application creates one pipeline that will manage all the elements
in the application.
Dataflow and buffers
--------------------
~~~~~~~~~~~~~~~~~~~~
GStreamer supports two possible types of dataflow, the push and pull model. In the
push model, an upstream element sends data to a downstream element by calling a
method on a sinkpad. In the pull model, a downstream element requests data from
an upstream element by calling a method on a source pad.
GStreamer supports two possible types of dataflow, the push and pull model. In the
push model, an upstream element sends data to a downstream element by calling a
method on a sinkpad. In the pull model, a downstream element requests data from
an upstream element by calling a method on a source pad.
The most common dataflow is the push model. The pull model can be used in specific
circumstances by demuxer elements. The pull model can also be used by low latency
audio applications.
The most common dataflow is the push model. The pull model can be used in specific
circumstances by demuxer elements. The pull model can also be used by low latency
audio applications.
The data passed between pads is encapsulated in Buffers. The buffer contains a
pointer to the actual data and also metadata describing the data. This metadata
includes:
- timestamp of the data, this is the time instance at which the data was captured
or the time at which the data should be played back.
- offset of the data: a media specific offset, this could be samples for audio or
frames for video.
- the duration of the data in time.
- the media type of the data described with caps, these are key/value pairs that
describe the media type in a unique way.
- additional flags describing special properties of the data such as
discontinuities or delta units.
When an element whishes to send a buffer to another element is does this using one
of the pads that is linked to a pad of the other element. In the push model, a
buffer is pushed to the peer pad with gst_pad_push(). In the pull model, a buffer
is pulled from the peer with the gst_pad_pull_range() function.
Before an element pushes out a buffer, it should make sure that the peer element
can understand the buffer contents. It does this by querying the peer element
for the supported formats and by selecting a suitable common format. The selected
format is then attached to the buffer with gst_buffer_set_caps() before pushing
out the buffer.
When an element pad receives a buffer, if has to check if it understands the media
type of the buffer before starting processing it. The GStreamer core does this
automatically and will call the gst_pad_set_caps() function of the element before
sending the buffer to the element.
Both gst_pad_push() and gst_pad_pull_range() have a return value indicating whether
the operation succeeded. An error code means that no more data should be sent
to that pad. A source element that initiates the data flow in a thread typically
pauses the producing thread when this happens.
The data passed between pads is encapsulated in Buffers. The buffer contains a
pointer to the actual data and also metadata describing the data. This metadata
includes:
A buffer can be created with gst_buffer_new() or by requesting a usable buffer
from the peer pad using gst_pad_alloc_buffer(). Using the second method, it is
possible for the peer element to suggest the element to produce data in another
format by attaching another media type caps to the buffer.
- timestamp of the data, this is the time instance at which the data was captured
or the time at which the data should be played back.
- offset of the data: a media specific offset, this could be samples for audio or
frames for video.
- the duration of the data in time.
- the media type of the data described with caps, these are key/value pairs that
describe the media type in a unique way.
- additional flags describing special properties of the data such as
discontinuities or delta units.
The process of selecting a media type and attaching it to the buffers is called
caps negotiation.
When an element whishes to send a buffer to another element is does this using one
of the pads that is linked to a pad of the other element. In the push model, a
buffer is pushed to the peer pad with gst_pad_push(). In the pull model, a buffer
is pulled from the peer with the gst_pad_pull_range() function.
Before an element pushes out a buffer, it should make sure that the peer element
can understand the buffer contents. It does this by querying the peer element
for the supported formats and by selecting a suitable common format. The selected
format is then attached to the buffer with gst_buffer_set_caps() before pushing
out the buffer.
When an element pad receives a buffer, if has to check if it understands the media
type of the buffer before starting processing it. The GStreamer core does this
automatically and will call the gst_pad_set_caps() function of the element before
sending the buffer to the element.
Both gst_pad_push() and gst_pad_pull_range() have a return value indicating whether
the operation succeeded. An error code means that no more data should be sent
to that pad. A source element that initiates the data flow in a thread typically
pauses the producing thread when this happens.
A buffer can be created with gst_buffer_new() or by requesting a usable buffer
from the peer pad using gst_pad_alloc_buffer(). Using the second method, it is
possible for the peer element to suggest the element to produce data in another
format by attaching another media type caps to the buffer.
The process of selecting a media type and attaching it to the buffers is called
caps negotiation.
Caps
----
A media type (Caps) is described using a generic list of key/value pairs. The key is
a string and the value can be a single/list/range of int/float/string.
~~~~
Caps that have no ranges/list or other variable parts are said to be fixed and
can be used to put on a buffer.
A media type (Caps) is described using a generic list of key/value pairs. The key is
a string and the value can be a single/list/range of int/float/string.
Caps with variables in them are used to describe possible media types that can be
handled by a pad.
Caps that have no ranges/list or other variable parts are said to be fixed and
can be used to put on a buffer.
Caps with variables in them are used to describe possible media types that can be
handled by a pad.
Dataflow and events
-------------------
~~~~~~~~~~~~~~~~~~~
Parallel to the dataflow is a flow of events. Unlike the buffers, events can pass
both upstream and downstream. Some events only travel upstream others only downstream.
Parallel to the dataflow is a flow of events. Unlike the buffers, events can pass
both upstream and downstream. Some events only travel upstream others only downstream.
the events are used to denote special conditions in the dataflow such as EOS or
to inform plugins of special events such as flushing or seeking.
the events are used to denote special conditions in the dataflow such as EOS or
to inform plugins of special events such as flushing or seeking.
Some events must be serialized with the buffer flow, others don't. Serialized
events are inserted between the buffers. Non serialized events jump in front
of any buffers current being processed.
Some events must be serialized with the buffer flow, others don't. Serialized
events are inserted between the buffers. Non serialized events jump in front
of any buffers current being processed.
An example of a serialized event is a TAG event that is inserted between buffers
to mark metadata for those buffers.
An example of a serialized event is a TAG event that is inserted between buffers
to mark metadata for those buffers.
An example of a non serialized event is the FLUSH event.
An example of a non serialized event is the FLUSH event.
Pipeline construction
---------------------
~~~~~~~~~~~~~~~~~~~~~
The application starts by creating a Pipeline element using gst_pipeline_new ().
Elements are added to and removed from the pipeline with gst_bin_add() and
gst_bin_remove().
The application starts by creating a Pipeline element using gst_pipeline_new ().
Elements are added to and removed from the pipeline with gst_bin_add() and
gst_bin_remove().
After adding the elements, the pads of an element can be retrieved with
gst_element_get_pad(). Pads can then be linked together with gst_pad_link().
After adding the elements, the pads of an element can be retrieved with
gst_element_get_pad(). Pads can then be linked together with gst_pad_link().
Some elements create new pads when actual dataflow is happening in the pipeline.
With g_signal_connect() one can receive a notification when an element has created
a pad. These new pads can then be linked to other unlinked pads.
Some elements create new pads when actual dataflow is happening in the pipeline.
With g_signal_connect() one can receive a notification when an element has created
a pad. These new pads can then be linked to other unlinked pads.
Some elements cannot be linked together because they operate on different
incompatible data types. The possible datatypes a pad can provide or consume can
be retrieved with gst_pad_get_caps().
Some elements cannot be linked together because they operate on different
incompatible data types. The possible datatypes a pad can provide or consume can
be retrieved with gst_pad_get_caps().
Below is a simple mp3 playback pipeline that we constructed. We will use this
pipeline in further examples.
Below is a simple mp3 playback pipeline that we constructed. We will use this
pipeline in further examples.
+-------------------------------------------+
| pipeline |
@ -293,195 +293,195 @@ Pipeline construction
Pipeline clock
--------------
~~~~~~~~~~~~~~
One of the important functions of the pipeline is to select a global clock
for all the elements in the pipeline.
One of the important functions of the pipeline is to select a global clock
for all the elements in the pipeline.
The purpose of the clock is to provide a stricly increasing value at the rate
of one GST_SECOND per second. Clock values are expressed in nanoseconds.
Elements use the clock time to synchronize the playback of data.
The purpose of the clock is to provide a stricly increasing value at the rate
of one GST_SECOND per second. Clock values are expressed in nanoseconds.
Elements use the clock time to synchronize the playback of data.
Before the pipeline is set to PLAYING, the pipeline asks each element if they can
provide a clock. The clock is selected in the following order:
Before the pipeline is set to PLAYING, the pipeline asks each element if they can
provide a clock. The clock is selected in the following order:
- If the application selected a clock, use that one.
- If a source element provides a clock, use that clock.
- Select a clock from any other element that provides a clock, start with the
sinks.
- If no element provides a clock a default system clock is used for the pipeline.
- If the application selected a clock, use that one.
- If a source element provides a clock, use that clock.
- Select a clock from any other element that provides a clock, start with the
sinks.
- If no element provides a clock a default system clock is used for the pipeline.
In a typical playback pipeline this algorithm will select the clock provided by
a sink element such as an audio sink.
In a typical playback pipeline this algorithm will select the clock provided by
a sink element such as an audio sink.
In capture pipelines, this will typically select the clock of the data producer, which
in most cases can not control the rate at which it produces data.
In capture pipelines, this will typically select the clock of the data producer, which
in most cases can not control the rate at which it produces data.
Pipeline states
---------------
~~~~~~~~~~~~~~~
When all the pads are linked and signals have been connected, the pipeline can
be put in the PAUSED state to start dataflow.
When all the pads are linked and signals have been connected, the pipeline can
be put in the PAUSED state to start dataflow.
When a bin (and hence a pipeline) performs a state change, it will change the state
of all its children. The pipeline will change the state of its children from the
sink elements to the source elements, this to make sure that no upstream element
produces data to an element that is not yet ready to accept it.
When a bin (and hence a pipeline) performs a state change, it will change the state
of all its children. The pipeline will change the state of its children from the
sink elements to the source elements, this to make sure that no upstream element
produces data to an element that is not yet ready to accept it.
In the mp3 playback pipeline, the state of the elements is changed in the order
alsasink, mp3dec, filesrc.
In the mp3 playback pipeline, the state of the elements is changed in the order
alsasink, mp3dec, filesrc.
All intermediate states are traversed for each element resulting in the following
chain of state changes:
All intermediate states are traversed for each element resulting in the following
chain of state changes:
alsasink to READY: the audio device is probed
mp3dec to READY: nothing happens.
filesrc to READY: the file is probed
alsasink to PAUSED: the audio device is opened. alsasink is a sink and returns
ASYNC because it did not receive data yet.
mp3dec to PAUSED: the decoding library is initialized
filesrc to PAUSED: the file is opened and a thread is started to push data to
mp3dec
alsasink to READY: the audio device is probed
mp3dec to READY: nothing happens.
filesrc to READY: the file is probed
alsasink to PAUSED: the audio device is opened. alsasink is a sink and returns
ASYNC because it did not receive data yet.
mp3dec to PAUSED: the decoding library is initialized
filesrc to PAUSED: the file is opened and a thread is started to push data to
mp3dec
At this point data flows from filesrc to mp3dec and alsasink. Since mp3dec is PAUSED,
it accepts the data from filesrc on the sinkpad and starts decoding the compressed
data to raw audio samples.
At this point data flows from filesrc to mp3dec and alsasink. Since mp3dec is PAUSED,
it accepts the data from filesrc on the sinkpad and starts decoding the compressed
data to raw audio samples.
The mp3 decoder figures out the samplerate, the number of channels and other audio
properties of the raw audio samples, puts the decoded samples into a Buffer,
attaches the media type caps to the buffer and pushes this buffer to the next
element.
The mp3 decoder figures out the samplerate, the number of channels and other audio
properties of the raw audio samples, puts the decoded samples into a Buffer,
attaches the media type caps to the buffer and pushes this buffer to the next
element.
Alsasink then receives the buffer, inspects the caps and reconfigures itself to process
the buffer. Since it received the first buffer of samples, it completes the state change
to the PAUSED state. At this point the pipeline is prerolled and all elements have
samples. Alsasink is now also capable of providing a clock to the pipeline.
Alsasink then receives the buffer, inspects the caps and reconfigures itself to process
the buffer. Since it received the first buffer of samples, it completes the state change
to the PAUSED state. At this point the pipeline is prerolled and all elements have
samples. Alsasink is now also capable of providing a clock to the pipeline.
Since alsasink is now in the PAUSED state it blocks while receiving the first buffer. This
effectively blocks both mp3dec and filesrc in their gst_pad_push().
Since alsasink is now in the PAUSED state it blocks while receiving the first buffer. This
effectively blocks both mp3dec and filesrc in their gst_pad_push().
Since all elements now return SUCCESS from the gst_element_get_state() function,
the pipeline can be put in the PLAYING state.
Since all elements now return SUCCESS from the gst_element_get_state() function,
the pipeline can be put in the PLAYING state.
Before going to PLAYING, the pipeline select a clock and samples the current time of
the clock. This is the base_time. It then distributes this time to all elements.
Elements can then synchronize against the clock using the buffer running_time +
base_time (See also part-synchronisation.txt).
Before going to PLAYING, the pipeline select a clock and samples the current time of
the clock. This is the base_time. It then distributes this time to all elements.
Elements can then synchronize against the clock using the buffer running_time +
base_time (See also part-synchronisation.txt).
The following chain of state changes then takes place:
The following chain of state changes then takes place:
alsasink to PLAYING: the samples are played to the audio device
mp3dec to PLAYING: nothing happens
filesrc to PLAYING: nothing happens
alsasink to PLAYING: the samples are played to the audio device
mp3dec to PLAYING: nothing happens
filesrc to PLAYING: nothing happens
Pipeline status
---------------
~~~~~~~~~~~~~~~
The pipeline informs the application of any special events that occur in the
pipeline with the bus. The bus is an object that the pipeline provides and that
can be retrieved with gst_pipeline_get_bus().
The pipeline informs the application of any special events that occur in the
pipeline with the bus. The bus is an object that the pipeline provides and that
can be retrieved with gst_pipeline_get_bus().
The bus can be polled or added to the glib mainloop.
The bus can be polled or added to the glib mainloop.
The bus is distributed to all elements added to the pipeline. The elements use the bus
to post messages on. Various message types exist such as ERRORS, WARNINGS, EOS,
STATE_CHANGED, etc..
The bus is distributed to all elements added to the pipeline. The elements use the bus
to post messages on. Various message types exist such as ERRORS, WARNINGS, EOS,
STATE_CHANGED, etc..
The pipeline handles EOS messages received from elements in a special way. It will
only forward the message to the application when all sink elements have posted an
EOS message.
The pipeline handles EOS messages received from elements in a special way. It will
only forward the message to the application when all sink elements have posted an
EOS message.
Other methods for obtaining the pipeline status include the Query functionality that
can be performed with gst_element_query() on the pipeline. This type of query
is useful for obtaining information about the current position and total time of
the pipeline. It can also be used to query for the supported seeking formats and
ranges.
Other methods for obtaining the pipeline status include the Query functionality that
can be performed with gst_element_query() on the pipeline. This type of query
is useful for obtaining information about the current position and total time of
the pipeline. It can also be used to query for the supported seeking formats and
ranges.
Pipeline EOS
------------
~~~~~~~~~~~~
When the source filter encounters the end of the stream, it sends an EOS event to
the peer element. This event will then travel downstream to all of the connected
elements to inform them of the EOS. The element is not supposed to accept any more
data after receiving an EOS event on a sinkpad.
When the source filter encounters the end of the stream, it sends an EOS event to
the peer element. This event will then travel downstream to all of the connected
elements to inform them of the EOS. The element is not supposed to accept any more
data after receiving an EOS event on a sinkpad.
The element providing the streaming thread stops sending data after sending the
EOS event.
The element providing the streaming thread stops sending data after sending the
EOS event.
The EOS event will eventually arrive in the sink element. The sink will then post
an EOS message on the bus to inform the pipeline that a particular stream has
finished. When all sinks have reported EOS, the pipeline forwards the EOS message
to the application. The EOS message is only forwarded to the application in the
PLAYING state.
The EOS event will eventually arrive in the sink element. The sink will then post
an EOS message on the bus to inform the pipeline that a particular stream has
finished. When all sinks have reported EOS, the pipeline forwards the EOS message
to the application. The EOS message is only forwarded to the application in the
PLAYING state.
When in EOS, the pipeline remains in the PLAYING state, it is the applications
responsability to PAUSE or READY the pipeline. The application can also issue
a seek, for example.
When in EOS, the pipeline remains in the PLAYING state, it is the applications
responsability to PAUSE or READY the pipeline. The application can also issue
a seek, for example.
Pipeline READY
--------------
~~~~~~~~~~~~~~
When a running pipeline is set from the PLAYING to READY state, the following
actions occur in the pipeline:
alsasink to PAUSED: alsasink blocks and completes the state change on the
next sample. If the element was EOS, it does not wait for
a sample to complete the state change.
mp3dec to PAUSED: nothing
filesrc to PAUSED: nothing
When a running pipeline is set from the PLAYING to READY state, the following
actions occur in the pipeline:
Going to the intermediate PAUSED state will block all elements in the _push()
functions. This happens because the sink element blocks on the first buffer
it receives.
alsasink to PAUSED: alsasink blocks and completes the state change on the
next sample. If the element was EOS, it does not wait for
a sample to complete the state change.
mp3dec to PAUSED: nothing
filesrc to PAUSED: nothing
Some elements might be performing blocking operations in the PLAYING state that
must be unblocked when they go into the PAUSED state. This makes sure that the
state change happens very fast.
Going to the intermediate PAUSED state will block all elements in the _push()
functions. This happens because the sink element blocks on the first buffer
it receives.
In the next PAUSED to READY state change the pipeline has to shut down and all
streaming threads must stop sending data. This happens in the following sequence:
Some elements might be performing blocking operations in the PLAYING state that
must be unblocked when they go into the PAUSED state. This makes sure that the
state change happens very fast.
alsasink to READY: alsasink unblocks from the _chain() function and returns a
WRONG_STATE return value to the peer element. The sinkpad is
deactivated and becomes unusable for sending more data.
mp3dec to READY: the pads are deactivated and the state change completes when
mp3dec leaves its _chain() function.
filesrc to READY: the pads are deactivated and the thread is paused.
In the next PAUSED to READY state change the pipeline has to shut down and all
streaming threads must stop sending data. This happens in the following sequence:
alsasink to READY: alsasink unblocks from the _chain() function and returns a
WRONG_STATE return value to the peer element. The sinkpad is
deactivated and becomes unusable for sending more data.
mp3dec to READY: the pads are deactivated and the state change completes when
mp3dec leaves its _chain() function.
filesrc to READY: the pads are deactivated and the thread is paused.
The upstream elements finish their chain() function because the downstream element
returned an error code (WRONG_STATE) from the _push() functions. These error codes
are eventually returned to the element that started the streaming thread (filesrc),
which pauses the thread and completes the state change.
This sequence of events ensure that all elements are unblocked and all streaming
threads stopped.
The upstream elements finish their chain() function because the downstream element
returned an error code (WRONG_STATE) from the _push() functions. These error codes
are eventually returned to the element that started the streaming thread (filesrc),
which pauses the thread and completes the state change.
This sequence of events ensure that all elements are unblocked and all streaming
threads stopped.
Pipeline seeking
----------------
~~~~~~~~~~~~~~~~
Seeking in the pipeline requires a very specific order of operations to make
sure that the elements remain synchronized and that the seek is performed with
a minimal amount of latency.
Seeking in the pipeline requires a very specific order of operations to make
sure that the elements remain synchronized and that the seek is performed with
a minimal amount of latency.
An application issues a seek event on the pipeline using gst_element_send_event()
on the pipeline element. The event can be a seek event in any of the formats
supported by the elements.
An application issues a seek event on the pipeline using gst_element_send_event()
on the pipeline element. The event can be a seek event in any of the formats
supported by the elements.
The pipeline first pauses the pipeline to speed up the seek operations.
The pipeline then issues the seek event to all sink elements. The sink then forwards
the seek event upstream until some element can perform the seek operation, which is
typically the source or demuxer element. All intermediate elements can transform the
requested seek offset to another format, this way a decoder element can transform a
seek to a frame number to a timestamp, for example.
When the seek event reaches an element that will perform the seek operation, that
element performs the following steps.
The pipeline first pauses the pipeline to speed up the seek operations.
The pipeline then issues the seek event to all sink elements. The sink then forwards
the seek event upstream until some element can perform the seek operation, which is
typically the source or demuxer element. All intermediate elements can transform the
requested seek offset to another format, this way a decoder element can transform a
seek to a frame number to a timestamp, for example.
When the seek event reaches an element that will perform the seek operation, that
element performs the following steps.
1) send a FLUSH_START event to all downstream and upstream peer elements.
2) make sure the streaming thread is not running. The streaming thread will
@ -491,31 +491,31 @@ Pipeline seeking
5) send NEWSEGMENT event to inform all elements of the new position and to complete
the seek.
In step 1) all dowstream elements have to return from any blocking operations
and have to refuse any further buffers or events different from a FLUSH done.
In step 1) all dowstream elements have to return from any blocking operations
and have to refuse any further buffers or events different from a FLUSH done.
The first step ensures that the streaming thread eventually unblocks and that
step 2) can be performed. At this point, dataflow is completely stopped in the
pipeline.
The first step ensures that the streaming thread eventually unblocks and that
step 2) can be performed. At this point, dataflow is completely stopped in the
pipeline.
In step 3) the element performs the seek to the requested position.
In step 3) the element performs the seek to the requested position.
In step 4) all peer elements are allowed to accept data again and streaming
can continue from the new position. A FLUSH done event is sent to all the peer
elements so that they accept new data again and restart their streaming threads.
In step 4) all peer elements are allowed to accept data again and streaming
can continue from the new position. A FLUSH done event is sent to all the peer
elements so that they accept new data again and restart their streaming threads.
Step 5) informs all elements of the new position in the stream. After that the
event function returns back to the application. and the streaming threads start
to produce new data.
Step 5) informs all elements of the new position in the stream. After that the
event function returns back to the application. and the streaming threads start
to produce new data.
Since the pipeline is still PAUSED, this will preroll the next media sample in the
sinks. The application can wait for this preroll to complete by performing a
_get_state() on the pipeline.
Since the pipeline is still PAUSED, this will preroll the next media sample in the
sinks. The application can wait for this preroll to complete by performing a
_get_state() on the pipeline.
The last step in the seek operation is then to adjust the stream time of the pipeline
to 0 and to set the pipeline back to PLAYING.
The last step in the seek operation is then to adjust the stream time of the pipeline
to 0 and to set the pipeline back to PLAYING.
The sequence of events in our mp3 playback example.
The sequence of events in our mp3 playback example.
| a) seek on pipeline
| b) PAUSE pipeline
@ -537,4 +537,4 @@ Pipeline seeking
| e) update stream time to 0
| f) PLAY pipeline

4
docs/design/part-preroll.txt
View file

@ -19,7 +19,7 @@ sinks (see part-element-sink.txt)
Committing the state
--------------------
~~~~~~~~~~~~~~~~~~~~
When going to PAUSED and PLAYING a buffer should be queued in the pad. We also
make this requirement for going to PLAYING since a flush event in the PAUSED
@ -39,7 +39,7 @@ blocking wait.
Unlocking the preroll
---------------------
~~~~~~~~~~~~~~~~~~~~~
The following conditions unlock the preroll:

2
docs/design/part-push-pull.txt
View file

@ -23,7 +23,7 @@ srcpads implementing a getrange function will be the exception.
state changes
-------------
~~~~~~~~~~~~~
The GstBin sets the state of all the sink elements. These are the elements
without source pads.

11
docs/design/part-qos.txt
View file

@ -18,7 +18,7 @@ made:
Sources of quality problems
---------------------------
~~~~~~~~~~~~~~~~~~~~~~~~~~~
- High CPU load
- Network problems
@ -26,7 +26,7 @@ Sources of quality problems
QoS event
---------
~~~~~~~~~
The QoS event is generated by an element that synchronizes against the clock. It
travels upstream and contains the following fields:
@ -51,7 +51,7 @@ operations.
QoS message
-----------
~~~~~~~~~~~
A QOS message is posted on the bus whenever an element decides to:
@ -122,7 +122,7 @@ relevant elements or baseclasses.
Collecting statistics
---------------------
~~~~~~~~~~~~~~~~~~~~~
A buffer with timestamp B1 arrives in the sink at time T1. The buffer
timestamp is then synchronized against the clock which yields a jitter J1
@ -179,7 +179,7 @@ An element that is not receiving enough data is said to be starved.
Element measurements
--------------------
~~~~~~~~~~~~~~~~~~~~
In addition to the measurements of the datarate of the upstream element, a
typical element must also measure its own performance. Global pipeline
@ -389,3 +389,4 @@ Live sources will automatically drop data when it takes too long to process the
that the element pushes out.
Live sources should post a QoS message when data is dropped.

16
docs/design/part-query.txt
View file

@ -1,12 +1,16 @@
DRAFT Query
-----------
Query
-----
NOTE: this is implemented as proposed
Purpose
~~~~~~~
Queries are used to get information about the stream.
A query is started on a specific pad and travels up or downstream.
Queries are used to get information about the stream.
A query is started on a specific pad and travels up or downstream.
Types of queries
~~~~~~~~~~~~~~~~
- get length of stream
- get position in stream
@ -17,6 +21,7 @@ Types of queries
- query internal links.
Current implementation
~~~~~~~~~~~~~~~~~~~~~~
The current implementation of query requires pads to implement the
following functions:
@ -36,12 +41,14 @@ Current implementation
Requirements
~~~~~~~~~~~~
- multiple return values, grouped together when they make sense.
- one pad function to perform the query
- extensible queries.
Proposition
~~~~~~~~~~~
- define GstQuery extending GstMiniObject and containing a GstStructure (see GstMessage)
- define standard query types (see proposed types)
@ -54,6 +61,7 @@ Proposition
query is not supported.
Proposed types
~~~~~~~~~~~~~~
- GST_QUERY_SEEKING:

848
docs/design/part-relations.txt
View file

@ -5,487 +5,487 @@ This document describes the relations between objects that exist in GStreamer.
It will also describe the way of handling the relation wrt locking and
refcounting.
1) parent-child relation
parent-child relation
~~~~~~~~~~~~~~~~~~~~~
+---------+ +-------+
| parent | | child |
*--->| *----->| |
| F1|<-----* 1|
+---------+ +-------+
+---------+ +-------+
| parent | | child |
*--->| *----->| |
| F1|<-----* 1|
+---------+ +-------+
- properties
- properties
- parent has references to multiple children
- child has reference to parent
- reference fields protected with LOCK
- the reference held by each child to the parent is
NOT reflected in the refcount of the parent.
- the parent removes the floating flag of the child when taking
ownership.
- the application has valid reference to parent
- creation/destruction requires two unnested locks and 1 refcount.
- parent has references to multiple children
- child has reference to parent
- reference fields protected with LOCK
- the reference held by each child to the parent is
NOT reflected in the refcount of the parent.
- the parent removes the floating flag of the child when taking
ownership.
- the application has valid reference to parent
- creation/destruction requires two unnested locks and 1 refcount.
- usage in GStreamer
- usage in GStreamer
GstBin -> GstElement
GstElement -> GstRealPad
GstBin -> GstElement
GstElement -> GstRealPad
- lifecycle
a) object creation
- lifecycle
a) object creation
The application creates two object and holds a pointer
to them. The objects are initially FLOATING with a refcount
of 1.
The application creates two object and holds a pointer
to them. The objects are initially FLOATING with a refcount
of 1.
+---------+ +-------+
*--->| parent | *--->| child |
| * | | |
| F1| | * F1|
+---------+ +-------+
+---------+ +-------+
*--->| parent | *--->| child |
| * | | |
| F1| | * F1|
+---------+ +-------+
b) establishing the parent-child relationship
b) establishing the parent-child relationship
The application then calls a method on the parent object to take
ownership of the child object. The parent performs the following
actions:
The application then calls a method on the parent object to take
ownership of the child object. The parent performs the following
actions:
result = _set_parent (child, parent);
if (result) {
LOCK (parent);
ref_pointer = child;
result = _set_parent (child, parent);
if (result) {
LOCK (parent);
ref_pointer = child;
.. update other data structures ..
UNLOCK (parent);
}
else {
.. child had parent ..
}
.. update other data structures ..
UNLOCK (parent);
}
else {
.. child had parent ..
}
The _set_parent() method performs the following actions:
LOCK (child);
if (child->parent != NULL) {
UNLOCK (child);
return FALSE;
}
if (IS_FLOATING (child)) {
UNSET (child, FLOATING);
}
else {
_ref (child);
}
child->parent = parent;
The _set_parent() method performs the following actions:
LOCK (child);
if (child->parent != NULL) {
UNLOCK (child);
_signal (PARENT_SET, child, parent);
return TRUE;
The function atomically checks if the child has no parent yet
and will set the parent if not. It will also sink the child, meaning
all floating references to the child are invalid now as it takes
over the refcount of the object.
return FALSE;
}
if (IS_FLOATING (child)) {
UNSET (child, FLOATING);
}
else {
_ref (child);
}
child->parent = parent;
UNLOCK (child);
_signal (PARENT_SET, child, parent);
return TRUE;
The function atomically checks if the child has no parent yet
and will set the parent if not. It will also sink the child, meaning
all floating references to the child are invalid now as it takes
over the refcount of the object.
Visually:
Visually:
after _set_parent() returns TRUE:
after _set_parent() returns TRUE:
+---------+ +-------+
*---->| parent | *-//->| child |
| * | | |
| F1|<-------------* 1|
+---------+ +-------+
+---------+ +-------+
*---->| parent | *-//->| child |
| * | | |
| F1|<-------------* 1|
+---------+ +-------+
after parent updates ref_pointer to child.
after parent updates ref_pointer to child.
+---------+ +-------+
*---->| parent | *-//->| child |
| *--------->| |
| F1|<---------* 1|
+---------+ +-------+
+---------+ +-------+
*---->| parent | *-//->| child |
| *--------->| |
| F1|<---------* 1|
+---------+ +-------+
- only one parent is able to _sink the same object because the
_set_parent() method is atomic.
- since only one parent is able to _set_parent() the object, only
one will add a reference to the object.
- since the parent can hold multiple references to children, we don't
need to lock the parent when locking the child. Many threads can
call _set_parent() on the children with the same parent, the parent
can then add all those to its lists.
Note: that the signal is emited before the parent has added the
element to its internal data structures. This is not a problem
since the parent usually has his own signal to inform the app that
the child was reffed. One possible solution would be to update the
internal structure first and then perform a rollback if the _set_parent()
failed. This is not a good solution as iterators might grab the
'half-added' child too soon.
c) using the parent-child relationship
- since the initial floating reference to the child object became
invalid after giving it to the parent, any reference to a child
has at least a refcount > 1.
- this means that unreffing a child object cannot decrease the refcount
to 0. In fact, only the parent can destroy and dispose the child
object.
- given a reference to the child object, the parent pointer is only
valid when holding the child LOCK. Indeed, after unlocking the child
LOCK, the parent can unparent the child or the parent could even become
disposed. To avoid the parent dispose problem, when obtaining the
parent pointer, if should be reffed before releasing the child LOCK.
1) getting a reference to the parent.
- only one parent is able to _sink the same object because the
_set_parent() method is atomic.
- since only one parent is able to _set_parent() the object, only
one will add a reference to the object.
- since the parent can hold multiple references to children, we don't
need to lock the parent when locking the child. Many threads can
call _set_parent() on the children with the same parent, the parent
can then add all those to its lists.
- a referece is held to the child, so it cannot be disposed.
LOCK (child);
parent = _ref (child->parent);
UNLOCK (child);
Note: that the signal is emited before the parent has added the
element to its internal data structures. This is not a problem
since the parent usually has his own signal to inform the app that
the child was reffed. One possible solution would be to update the
internal structure first and then perform a rollback if the _set_parent()
failed. This is not a good solution as iterators might grab the
'half-added' child too soon.
c) using the parent-child relationship
- since the initial floating reference to the child object became
invalid after giving it to the parent, any reference to a child
has at least a refcount > 1.
.. use parent ..
- this means that unreffing a child object cannot decrease the refcount
to 0. In fact, only the parent can destroy and dispose the child
object.
_unref (parent);
- given a reference to the child object, the parent pointer is only
valid when holding the child LOCK. Indeed, after unlocking the child
LOCK, the parent can unparent the child or the parent could even become
disposed. To avoid the parent dispose problem, when obtaining the
parent pointer, if should be reffed before releasing the child LOCK.
2) getting a reference to a child
- a reference to a child can be obtained by reffing it before
adding it to the parent or by querying the parent.
- when requesting a child from the parent, a reference is held to
the parent so it cannot be disposed. The parent will use its
internal data structures to locate the child element and will
return a reference to it with an incremented refcount. The
requester should _unref() the child after usage.
d) destroying the parent-child relationship
- only the parent can actively destroy the parent-child relationship
this typically happens when a method is called on the parent to release
ownership of the child.
- a child shall never remove itself from the parent.
- since calling a method on the parent with the child as an argument
requires the caller to obtain a valid reference to the child, the child
refcount is at least > 1.
- the parent will perform the folowing actions:
LOCK (parent);
if (ref_pointer == child) {
ref_pointer = NULL;
.. update other data structures ..
UNLOCK (parent);
_unparent (child);
}
else {
UNLOCK (parent);
.. not our child ..
}
The _unparent() method performs the following actions:
LOCK (child);
if (child->parent != NULL) {
child->parent = NULL;
UNLOCK (child);
_signal (PARENT_UNSET, child, parent);
_unref (child);
}
else {
UNLOCK (child);
}
I) getting a reference to the parent.
- a referece is held to the child, so it cannot be disposed.
LOCK (child);
parent = _ref (child->parent);
UNLOCK (child);
.. use parent ..
_unref (parent);
II) getting a reference to a child
- a reference to a child can be obtained by reffing it before
adding it to the parent or by querying the parent.
- when requesting a child from the parent, a reference is held to
the parent so it cannot be disposed. The parent will use its
internal data structures to locate the child element and will
return a reference to it with an incremented refcount. The
requester should _unref() the child after usage.
d) destroying the parent-child relationship
- only the parent can actively destroy the parent-child relationship
this typically happens when a method is called on the parent to release
ownership of the child.
- a child shall never remove itself from the parent.
- since calling a method on the parent with the child as an argument
requires the caller to obtain a valid reference to the child, the child
refcount is at least > 1.
- the parent will perform the folowing actions:
LOCK (parent);
if (ref_pointer == child) {
ref_pointer = NULL;
.. update other data structures ..
UNLOCK (parent);
_unparent (child);
}
else {
UNLOCK (parent);
.. not our child ..
}
The _unparent() method performs the following actions:
LOCK (child);
if (child->parent != NULL) {
child->parent = NULL;
UNLOCK (child);
_signal (PARENT_UNSET, child, parent);
_unref (child);
}
else {
UNLOCK (child);
}
Since the _unparent() method unrefs the child object, it is possible that
the child pointer is invalid after this function. If the parent wants to
perform other actions on the child (such as signal emmision) it should
_ref() the child first.
Since the _unparent() method unrefs the child object, it is possible that
the child pointer is invalid after this function. If the parent wants to
perform other actions on the child (such as signal emmision) it should
_ref() the child first.
2) single-reffed relation
single-reffed relation
~~~~~~~~~~~~~~~~~~~~~~
+---------+ +---------+
*--->| object1 | *--->| object2 |
| *--------->| |
| 1| | 2|
+---------+ +---------+
+---------+ +---------+
*--->| object1 | *--->| object2 |
| *--------->| |
| 1| | 2|
+---------+ +---------+
- properties
- properties
- one object has a reference to another
- reference field protected with LOCK
- the reference held by the object is reflected in the
refcount of the other object.
- typically the other object can be shared among multiple
other objects where each ref is counted for in the
refcount.
- no object has ownership of the other.
- either shared state or copy-on-write.
- creation/destruction requires one lock and one refcount.
- one object has a reference to another
- reference field protected with LOCK
- the reference held by the object is reflected in the
refcount of the other object.
- typically the other object can be shared among multiple
other objects where each ref is counted for in the
refcount.
- no object has ownership of the other.
- either shared state or copy-on-write.
- creation/destruction requires one lock and one refcount.
- usage
- usage
GstRealPad -> GstCaps
GstBuffer -> GstCaps
GstEvent -> GstCaps
GstEvent -> GstObject
GstMessage -> GstCaps
GstMessage -> GstObject
GstRealPad -> GstCaps
GstBuffer -> GstCaps
GstEvent -> GstCaps
GstEvent -> GstObject
GstMessage -> GstCaps
GstMessage -> GstObject
- lifecycle
- lifecycle
a) Two objects exist unlinked.
+---------+ +---------+
*--->| object1 | *--->| object2 |
| * | | |
| 1| | 1|
+---------+ +---------+
a) Two objects exist unlinked.
+---------+ +---------+
*--->| object1 | *--->| object2 |
| * | | |
| 1| | 1|
+---------+ +---------+
b) establishing the single-reffed relationship
b) establishing the single-reffed relationship
The second object is attached to the first one using a method
on the first object. The second object is reffed and a pointer
is updated in the first object using the following algorithm:
The second object is attached to the first one using a method
on the first object. The second object is reffed and a pointer
is updated in the first object using the following algorithm:
LOCK (object1);
LOCK (object1);
if (object1->pointer)
_unref (object1->pointer);
object1->pointer = _ref (object2);
UNLOCK (object1);
After releasing the lock on the first object is is not sure that
object2 is still reffed from object1.
+---------+ +---------+
*--->| object1 | *--->| object2 |
| *--------->| |
| 1| | 2|
+---------+ +---------+
c) using the single-reffed relationship
The only way to access object2 is by holding a ref to it or by
getting the reference from object1.
Reading the object pointed to by object1 can be done like this:
LOCK (object1);
object2 = object1->pointer;
_ref (object2);
UNLOCK (object1);
.. use object2 ...
_unref (object2);
Depending on the type of the object, modifications can be done either
with copy-on-write or directly into the object.
Copy on write can practically only be done like this:
LOCK (object1);
object2 = object1->pointer;
object2 = _copy_on_write (object2);
... make modifications to object2 ...
UNLOCK (object1);
Releasing the lock has only a very small window where the copy_on_write
actually does not perform a copy:
LOCK (object1);
object2 = object1->pointer;
_ref (object2);
UNLOCK (object1);
.. object2 now has at least 2 refcounts making the next
copy-on-write make a real copy, unless some other thread
writes another object2 to object1 here ...
object2 = _copy_on_write (object2);
.. make modifications to object2 ...
LOCK (object1);
if (object1->pointer != object2) {
if (object1->pointer)
_unref (object1->pointer);
object1->pointer = _ref (object2);
UNLOCK (object1);
object1->pointer = gst_object_ref (object2);
}
UNLOCK (object1);
After releasing the lock on the first object is is not sure that
object2 is still reffed from object1.
d) destroying the single-reffed relationship
The folowing algorithm removes the single-reffed link between
object1 and object2.
+---------+ +---------+
*--->| object1 | *--->| object2 |
| *--------->| |
| 1| | 2|
+---------+ +---------+
c) using the single-reffed relationship
The only way to access object2 is by holding a ref to it or by
getting the reference from object1.
Reading the object pointed to by object1 can be done like this:
LOCK (object1);
object2 = object1->pointer;
_ref (object2);
UNLOCK (object1);
.. use object2 ...
_unref (object2);
Depending on the type of the object, modifications can be done either
with copy-on-write or directly into the object.
Copy on write can practically only be done like this:
LOCK (object1);
object2 = object1->pointer;
object2 = _copy_on_write (object2);
... make modifications to object2 ...
UNLOCK (object1);
Releasing the lock has only a very small window where the copy_on_write
actually does not perform a copy:
LOCK (object1);
object2 = object1->pointer;
_ref (object2);
UNLOCK (object1);
.. object2 now has at least 2 refcounts making the next
copy-on-write make a real copy, unless some other thread
writes another object2 to object1 here ...
object2 = _copy_on_write (object2);
.. make modifications to object2 ...
LOCK (object1);
if (object1->pointer != object2) {
if (object1->pointer)
_unref (object1->pointer);
object1->pointer = gst_object_ref (object2);
}
UNLOCK (object1);
d) destroying the single-reffed relationship
The folowing algorithm removes the single-reffed link between
object1 and object2.
LOCK (object1);
_unref (object1->pointer);
object1->pointer = NULL;
UNLOCK (object1);
Which yields the following initial state again:
+---------+ +---------+
*--->| object1 | *--->| object2 |
| * | | |
| 1| | 1|
+---------+ +---------+
3) unreffed relation
+---------+ +---------+
*--->| object1 | *--->| object2 |
| *--------->| |
| 1|<---------* 1|
+---------+ +---------+
- properties
- two objects have references to eachother
- both objects can only have 1 reference to another object.
- reference fields protected with LOCK
- the references held by each object are NOT reflected in the
refcount of the other object.
- no object has ownership of the other.
- typically each object is owned by a different parent.
- creation/destruction requires two nested locks and no refcounts.
- usage
- This type of link is used when the link is less important than
the existance of the objects, If one of the objects is disposed, so
is the link.
LOCK (object1);
_unref (object1->pointer);
object1->pointer = NULL;
UNLOCK (object1);
GstRealPad <-> GstRealPad (srcpad lock taken first)
Which yields the following initial state again:
- lifecycle
+---------+ +---------+
*--->| object1 | *--->| object2 |
| * | | |
| 1| | 1|
+---------+ +---------+
a) Two objects exist unlinked.
+---------+ +---------+
*--->| object1 | *--->| object2 |
| * | | |
| 1| | * 1|
+---------+ +---------+
b) establishing the unreffed relationship
Since we need to take two locks, the order in which these locks are
taken is very important or we might cause deadlocks. This lock order
must be defined for all unreffed relations. In these examples we always
lock object1 first and then object2.
LOCK (object1);
LOCK (object2);
object2->refpointer = object1;
object1->refpointer = object2;
UNLOCK (object2);
UNLOCK (object1);
c) using the unreffed relationship
Reading requires taking one of the locks and reading the corresponing
object. Again we need to ref the object before releasing the lock.
LOCK (object1);
object2 = _ref (object1->refpointer);
UNLOCK (object1);
.. use object2 ..
_unref (object2);
d) destroying the unreffed relationship
Because of the lock order we need to be careful when destroying this
Relation.
unreffed relation
~~~~~~~~~~~~~~~~~
+---------+ +---------+
*--->| object1 | *--->| object2 |
| *--------->| |
| 1|<---------* 1|
+---------+ +---------+
When only a reference to object1 is held:
- properties
LOCK (object1);
LOCK (object2);
- two objects have references to eachother
- both objects can only have 1 reference to another object.
- reference fields protected with LOCK
- the references held by each object are NOT reflected in the
refcount of the other object.
- no object has ownership of the other.
- typically each object is owned by a different parent.
- creation/destruction requires two nested locks and no refcounts.
- usage
- This type of link is used when the link is less important than
the existance of the objects, If one of the objects is disposed, so
is the link.
GstRealPad <-> GstRealPad (srcpad lock taken first)
- lifecycle
a) Two objects exist unlinked.
+---------+ +---------+
*--->| object1 | *--->| object2 |
| * | | |
| 1| | * 1|
+---------+ +---------+
b) establishing the unreffed relationship
Since we need to take two locks, the order in which these locks are
taken is very important or we might cause deadlocks. This lock order
must be defined for all unreffed relations. In these examples we always
lock object1 first and then object2.
LOCK (object1);
LOCK (object2);
object2->refpointer = object1;
object1->refpointer = object2;
UNLOCK (object2);
UNLOCK (object1);
c) using the unreffed relationship
Reading requires taking one of the locks and reading the corresponing
object. Again we need to ref the object before releasing the lock.
LOCK (object1);
object2 = _ref (object1->refpointer);
UNLOCK (object1);
.. use object2 ..
_unref (object2);
d) destroying the unreffed relationship
Because of the lock order we need to be careful when destroying this
Relation.
When only a reference to object1 is held:
LOCK (object1);
LOCK (object2);
object1->refpointer->refpointer = NULL;
object1->refpointer = NULL;
UNLOCK (object2);
UNLOCK (object1);
When only a reference to object2 is held we need to get a handle to the
other object fist so that we can lock it first. There is a window where
we need to release all locks and the relation could be invalid. To solve
this we check the relation after grabbing both locks and retry if the
relation changed.
retry:
LOCK (object2);
object1 = _ref (object2->refpointer);
UNLOCK (object2);
.. things can change here ..
LOCK (object1);
LOCK (object2);
if (object1 == object2->refpointer) {
/* relation unchanged */
object1->refpointer->refpointer = NULL;
object1->refpointer = NULL;
}
else {
/* relation changed.. retry */
UNLOCK (object2);
UNLOCK (object1);
_unref (object1);
goto retry;
}
UNLOCK (object2);
UNLOCK (object1);
_unref (object1);
When only a reference to object2 is held we need to get a handle to the
other object fist so that we can lock it first. There is a window where
we need to release all locks and the relation could be invalid. To solve
this we check the relation after grabbing both locks and retry if the
relation changed.
When references are held to both objects. Note that it is not possible to
get references to both objects with the locks released since when the
references are taken and the locks are released, a concurrent update might
have changed the link, making the references not point to linked objects.
retry:
LOCK (object2);
object1 = _ref (object2->refpointer);
UNLOCK (object2);
.. things can change here ..
LOCK (object1);
LOCK (object2);
if (object1 == object2->refpointer) {
/* relation unchanged */
object1->refpointer->refpointer = NULL;
object1->refpointer = NULL;
}
else {
/* relation changed.. retry */
UNLOCK (object2);
UNLOCK (object1);
_unref (object1);
goto retry;
}
UNLOCK (object2);
UNLOCK (object1);
_unref (object1);
When references are held to both objects. Note that it is not possible to
get references to both objects with the locks released since when the
references are taken and the locks are released, a concurrent update might
have changed the link, making the references not point to linked objects.
LOCK (object1);
LOCK (object2);
if (object1->refpointer == object2) {
object2->refpointer = NULL;
object1->refpointer = NULL;
}
else {
.. objects are not linked ..
}
UNLOCK (object2);
UNLOCK (object1);
LOCK (object1);
LOCK (object2);
if (object1->refpointer == object2) {
object2->refpointer = NULL;
object1->refpointer = NULL;
}
else {
.. objects are not linked ..
}
UNLOCK (object2);
UNLOCK (object1);
4) double-reffed relation
double-reffed relation
~~~~~~~~~~~~~~~~~~~~~~
+---------+ +---------+
*--->| object1 | *--->| object2 |
| *--------->| |
| 2|<---------* 2|
+---------+ +---------+
- properties
- two objects have references to eachother
- reference fields protected with LOCK
- the references held by each object are reflected in the
refcount of the other object.
- no object has ownership of the other.
- typically each object is owned by a different parent.
- creation/destruction requires two locks and two refcounts.
- usage
Not used in GStreamer.
- lifecycle
+---------+ +---------+
*--->| object1 | *--->| object2 |
| *--------->| |
| 2|<---------* 2|
+---------+ +---------+
- properties
- two objects have references to eachother
- reference fields protected with LOCK
- the references held by each object are reflected in the
refcount of the other object.
- no object has ownership of the other.
- typically each object is owned by a different parent.
- creation/destruction requires two locks and two refcounts.
- usage
Not used in GStreamer.
- lifecycle

261
docs/design/part-scheduling.txt
View file

@ -5,7 +5,7 @@ The scheduling in GStreamer is based on pads actively pushing (producing) data o
pad pulling in data (consuming) from other pads.
Pushing
-------
~~~~~~~
A pad can produce data and push it to the next pad. A pad that behaves this way
exposes a loop function that will be called repeadedly until it returns false.
@ -20,7 +20,7 @@ This method of producing data is called the streaming mode since the producer
produces a constant stream of data.
Pulling
-------
~~~~~~~
Pads that operate in pulling mode can only pull data from a pad that exposes the
pull_range function. In this case, the sink pad exposes a loop function that will be
@ -31,7 +31,7 @@ push function to push the result to the peer sinkpad.
Deciding the scheduling mode
----------------------------
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
When tha pad is activated, the _activate() function is called. The pad can then
choose to activate itself in push or pull mode depending on upstream
@ -41,162 +41,163 @@ The GStreamer core will by default activate pads in push mode when there is no
activate function for the pad.
The chain function
------------------
~~~~~~~~~~~~~~~~~~
The chain function will be called when a upstream element perform a _push() on the pad.
The upstream element can be another chain based element or a pushing source.
The getrange function
---------------------
~~~~~~~~~~~~~~~~~~~~~
The getrange function is called when a peer pad perform a _pull_range() on the pad. This
downstream pad can be a pulling element or another _pull_range() based element.
Plug-in techniques
------------------
~~~~~~~~~~~~~~~~~~
Multi-sink elements
-------------------
^^^^^^^^^^^^^^^^^^^
Elements with multiple sinks can either expose a loop function on each of the pads to
actively pull_range data or they can expose a chain function on each pad.
Implementing a chain function is usually easy and allows for all possible scheduling
methods.
Pad select
----------
If the chain based sink wants to wait for one of the pads to receive a buffer, just
implement the action to perform in the chain function. Be aware that the action could
be performed in different threads and possibly simultaneously so grab the STREAM_LOCK.
Collect pads
------------
If the chain based sink pads all require one buffer before the element can operate on
the data, collect all the buffers in the chain function and perform the action when
all chainpads received the buffer.
In this case you probably also don't want to accept more data on a pad that has a buffer
queued. This can easily be done with the following code snippet:
static GstFlowReturn _chain (GstPad *pad, GstBuffer *buffer)
{
LOCK (mylock);
while (pad->store != NULL) {
WAIT (mycond, mylock);
}
pad->store = buffer;
SIGNAL (mycond);
UNLOCK (mylock);
return GST_FLOW_OK;
}
static void _pull (GstPad *pad, GstBuffer **buffer)
{
LOCK (mylock);
while (pad->store == NULL) {
WAIT (mycond, mylock);
}
**buffer = pad->store;
pad->store = NULL;
SIGNAL (mycond);
UNLOCK (mylock);
}
Elements with multiple sinks can either expose a loop function on each of the pads to
actively pull_range data or they can expose a chain function on each pad.
Implementing a chain function is usually easy and allows for all possible scheduling
methods.
Pad select
----------
If the chain based sink wants to wait for one of the pads to receive a buffer, just
implement the action to perform in the chain function. Be aware that the action could
be performed in different threads and possibly simultaneously so grab the STREAM_LOCK.
Collect pads
------------
If the chain based sink pads all require one buffer before the element can operate on
the data, collect all the buffers in the chain function and perform the action when
all chainpads received the buffer.
In this case you probably also don't want to accept more data on a pad that has a buffer
queued. This can easily be done with the following code snippet:
static GstFlowReturn _chain (GstPad *pad, GstBuffer *buffer)
{
LOCK (mylock);
while (pad->store != NULL) {
WAIT (mycond, mylock);
}
pad->store = buffer;
SIGNAL (mycond);
UNLOCK (mylock);
return GST_FLOW_OK;
}
static void _pull (GstPad *pad, GstBuffer **buffer)
{
LOCK (mylock);
while (pad->store == NULL) {
WAIT (mycond, mylock);
}
**buffer = pad->store;
pad->store = NULL;
SIGNAL (mycond);
UNLOCK (mylock);
}
Cases
-----
Inside the braces below the pads is stated what function the
pad support:
~~~~~
l: exposes a loop function, so it can act as a pushing source.
g: exposes a getrange function
c: exposes a chain function
Inside the braces below the pads is stated what function the
pad support:
following scheduling decisions are made based on the scheduling
methods exposed by the pads:
l: exposes a loop function, so it can act as a pushing source.
g: exposes a getrange function
c: exposes a chain function
(g) - (l): sinkpad will pull data from src
(l) - (c): srcpad actively pushes data to sinkpad
() - (c): srcpad will push data to sinkpad.
following scheduling decisions are made based on the scheduling
methods exposed by the pads:
() - () : not schedulable.
() - (l): not schedulable.
(g) - () : not schedulable.
(g) - (c): not schedulable.
(l) - () : not schedulable.
(l) - (l): not schedulable
(g) - (l): sinkpad will pull data from src
(l) - (c): srcpad actively pushes data to sinkpad
() - (c): srcpad will push data to sinkpad.
() - (g): impossible
(g) - (g): impossible.
(l) - (g): impossible
(c) - () : impossible
(c) - (g): impossible
(c) - (l): impossible
(c) - (c): impossible
() - () : not schedulable.
() - (l): not schedulable.
(g) - () : not schedulable.
(g) - (c): not schedulable.
(l) - () : not schedulable.
(l) - (l): not schedulable
+---------+ +------------+ +-----------+
| filesrc | | mp3decoder | | audiosink |
| src--sink src--sink |
+---------+ +------------+ +-----------+
(l-g) (c) () (c)
() - (g): impossible
(g) - (g): impossible.
(l) - (g): impossible
(c) - () : impossible
(c) - (g): impossible
(c) - (l): impossible
(c) - (c): impossible
When activating the pads:
+---------+ +------------+ +-----------+
| filesrc | | mp3decoder | | audiosink |
| src--sink src--sink |
+---------+ +------------+ +-----------+
(l-g) (c) () (c)
* audiosink has a chain function and the peer pad has no
loop function, no scheduling is done.
* mp3decoder and filesrc expose an (l) - (c) connection,
a thread is created to call the srcpad loop function.
When activating the pads:
+---------+ +------------+ +----------+
| filesrc | | avidemuxer | | fakesink |
| src--sink src--sink |
+---------+ +------------+ +----------+
(l-g) (l) () (c)
* fakesink has a chain function and the peer pad has no
loop function, no scheduling is done.
* avidemuxer and filesrc expose an (g) - (l) connection,
a thread is created to call the sinkpad loop function.
* audiosink has a chain function and the peer pad has no
loop function, no scheduling is done.
* mp3decoder and filesrc expose an (l) - (c) connection,
a thread is created to call the srcpad loop function.
+---------+ +----------+ +------------+ +----------+
| filesrc | | identity | | avidemuxer | | fakesink |
| src--sink src--sink src--sink |
+---------+ +----------+ +------------+ +----------+
(l-g) (c) () (l) () (c)
+---------+ +------------+ +----------+
| filesrc | | avidemuxer | | fakesink |
| src--sink src--sink |
+---------+ +------------+ +----------+
(l-g) (l) () (c)
* fakesink has a chain function and the peer pad has no
loop function, no scheduling is done.
* avidemuxer and filesrc expose an (g) - (l) connection,
a thread is created to call the sinkpad loop function.
* fakesink has a chain function and the peer pad has no
loop function, no scheduling is done.
* avidemuxer and identity expose no schedulable connection so
this pipeline is not schedulable.
+---------+ +----------+ +------------+ +----------+
| filesrc | | identity | | avidemuxer | | fakesink |
| src--sink src--sink src--sink |
+---------+ +----------+ +------------+ +----------+
(l-g) (c) () (l) () (c)
+---------+ +----------+ +------------+ +----------+
| filesrc | | identity | | avidemuxer | | fakesink |
| src--sink src--sink src--sink |
+---------+ +----------+ +------------+ +----------+
(l-g) (c-l) (g) (l) () (c)
* fakesink has a chain function and the peer pad has no
loop function, no scheduling is done.
* avidemuxer and identity expose no schedulable connection so
this pipeline is not schedulable.
* fakesink has a chain function and the peer pad has no
loop function, no scheduling is done.
* avidemuxer and identity expose an (g) - (l) connection,
a thread is created to call the sinkpad loop function.
* identity knows the srcpad is getrange based and uses the
thread from avidemux to getrange data from filesrc.
+---------+ +----------+ +------------+ +----------+
| filesrc | | identity | | avidemuxer | | fakesink |
| src--sink src--sink src--sink |
+---------+ +----------+ +------------+ +----------+
(l-g) (c-l) (g) (l) () (c)
+---------+ +----------+ +------------+ +----------+
| filesrc | | identity | | oggdemuxer | | fakesink |
| src--sink src--sink src--sink |
+---------+ +----------+ +------------+ +----------+
(l-g) (c) () (l-c) () (c)
* fakesink has a chain function and the peer pad has no
loop function, no scheduling is done.
* avidemuxer and identity expose an (g) - (l) connection,
a thread is created to call the sinkpad loop function.
* identity knows the srcpad is getrange based and uses the
thread from avidemux to getrange data from filesrc.
* fakesink has a chain function and the peer pad has no
loop function, no scheduling is done.
* oggdemuxer and identity expose an () - (l-c) connection,
oggdemux has to operate in chain mode.
* identity chan only work chain based and so filesrc creates
a thread to push data to identity.
+---------+ +----------+ +------------+ +----------+
| filesrc | | identity | | oggdemuxer | | fakesink |
| src--sink src--sink src--sink |
+---------+ +----------+ +------------+ +----------+
(l-g) (c) () (l-c) () (c)
* fakesink has a chain function and the peer pad has no
loop function, no scheduling is done.
* oggdemuxer and identity expose an () - (l-c) connection,
oggdemux has to operate in chain mode.
* identity chan only work chain based and so filesrc creates
a thread to push data to identity.

14
docs/design/part-seeking.txt
View file

@ -58,7 +58,7 @@ more important than accurately producing all frames.
Seeking in push based elements
------------------------------
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@ -66,19 +66,21 @@ Seeking in push based elements
Generating seeking events
-------------------------
~~~~~~~~~~~~~~~~~~~~~~~~~
A seek event is created with gst_event_new_seek ().
Seeking variants
~~~~~~~~~~~~~~~~
The different kinds of seeking methods and their internal workings are
described below.
FLUSH seeking
-------------
^^^^^^^^^^^^^
This is the most common way of performing a seek in a playback application.
The application issues a seek on the pipeline and the new media is immediatly
@ -86,7 +88,7 @@ played after the seek calls returns.
seeking without FLUSH
---------------------
^^^^^^^^^^^^^^^^^^^^^
This seek type is typically performed after issuing segment seeks to finish
the playback of the pipeline.
@ -97,13 +99,13 @@ sinks.
segment seeking with FLUSH
--------------------------
^^^^^^^^^^^^^^^^^^^^^^^^^^
This seek is typically performed when starting seamless looping.
segment seeking without FLUSH
-----------------------------
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
This seek is typically performed when continuing seamless looping.

6
docs/design/part-segments.txt
View file

@ -29,7 +29,7 @@ outside of the segment and that might therefore be discarded or clipped.
Use case: FLUSHING seek
-----------------------
~~~~~~~~~~~~~~~~~~~~~~~
ex.
@ -95,13 +95,13 @@ Use case: FLUSHING seek
Use case: live stream
---------------------
~~~~~~~~~~~~~~~~~~~~~
Use case: segment looping
-------------------------
~~~~~~~~~~~~~~~~~~~~~~~~~
Consider the case of a wav file with raw audio.

9
docs/design/part-sparsestreams.txt
View file

@ -2,7 +2,8 @@ DRAFT Sparse Streams
--------------------
Introduction
------------
~~~~~~~~~~~~
In 0.8, there was some support for Sparse Streams through the use of
FILLER events. These were used to mark gaps between buffers so that downstream
elements could know not to expect any more data for that gap.
@ -11,7 +12,8 @@ In 0.10, segment information conveyed through NEWSEGMENT events can be used
for the same purpose.
Use cases
---------
~~~~~~~~~
1) Sub-title streams
Sub-title information from muxed formats such as Matroska or MPEG consist of
irregular buffers spaced far apart compared to the other streams
@ -37,7 +39,8 @@ Use cases
application that uses noise-gating (to save bandwith).
Details
-------
~~~~~~~
1) Sub-title streams
The main requirement here is to avoid stalling the pipeline between sub-title
packets, and is effectively updating the minimum-timestamp for that stream.

6
docs/design/part-standards.txt
View file

@ -6,7 +6,7 @@ collection has many potential pitfalls as far as the pointers go. Therefore,
some standards must be adhered to as far as who owns what.
Strings
-------
~~~~~~~
Arguments passed into a function are owned by the caller, and the function
will make a copy of the string for its own internal use. The string should
@ -21,7 +21,7 @@ original and should be freed after usage by the caller.
Objects
-------
~~~~~~~
Objects passed into a function are owned by the caller, any additional
reference held to the object after leaving the function should increase the
@ -40,7 +40,7 @@ called should _free() or _unref() the object after usage.
Iterators
---------
~~~~~~~~~
When retrieving multiple objects from an object an iterator should be used.
The iterator allows you to access the objects one after another while making

28
docs/design/part-states.txt
View file

@ -1,5 +1,5 @@
States
======
------
Both elements and pads can be in different states. The states of the pads are
linked to the state of the element so the design of the states is mainly
@ -10,7 +10,7 @@ is initially instantiated, it is in the NULL state.
State definitions
-----------------
~~~~~~~~~~~~~~~~~
- NULL: This is the initial state of an element.
- READY: The element should be prepared to go to PAUSED.
@ -24,7 +24,7 @@ a downwards state change.
State transitions
-----------------
~~~~~~~~~~~~~~~~~
the following state changes are possible:
@ -89,7 +89,7 @@ the following state changes are possible:
State variables
---------------
~~~~~~~~~~~~~~~
An element has 4 state variables that are protected with the object LOCK:
@ -111,7 +111,7 @@ _set_state(), called the STATE_LOCK.
Setting state on elements
-------------------------
~~~~~~~~~~~~~~~~~~~~~~~~~
The state of an element can be changed with _element_set_state(). When chaning
the state of an element all intermediate states will also be set on the element
@ -147,7 +147,7 @@ is immediatly returned to the caller.
Getting state of elements
-------------------------
~~~~~~~~~~~~~~~~~~~~~~~~~
The _get_state() function takes 3 arguments, two pointers that will hold the
current and pending state and one GstClockTime that holds a timeout value. The
@ -184,7 +184,7 @@ function returns a GstElementStateReturn.
States in GstBin
----------------
~~~~~~~~~~~~~~~~
A GstBin manages the state of its children. It does this by propagating the state
changes performed on it to all of its children. The _set_state() function on a
@ -221,15 +221,15 @@ current state fields when it receives state messages from the children.
Implementing states in elements
-------------------------------
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
READY
-----
^^^^^
upward state change
-------------------
~~~~~~~~~~~~~~~~~~~
Upward state changes always return ASYNC either if the STATE_PENDING is
reached or not.
@ -262,7 +262,7 @@ Bin:
to STATE_PENDING
downward state change
----------------------
~~~~~~~~~~~~~~~~~~~~~
Downward state changes only return ASYNC if the final state is ASYNC.
This is to make sure that it's not needed to wait for an element to
@ -296,7 +296,7 @@ Bin:
Locking overview (element)
--------------------------
~~~~~~~~~~~~~~~~~~~~~~~~~~
* Element commiting SUCCESS
@ -376,8 +376,8 @@ Locking overview (element)
| ...
STREAM_UNLOCK
*********************************************
*********************************************
Remarks
~~~~~~~
set_state cannot be called from multiple threads at the same time. The STATE_LOCK
prevents this.

6
docs/design/part-stream-status.txt
View file

@ -26,7 +26,7 @@ We allow for the following scenarios:
Use cases
---------
~~~~~~~~~
* boost the priority of the udp receiver streaming thread
@ -56,7 +56,7 @@ Use cases
Messages
--------
~~~~~~~~
The existing STREAM_STATUS message will be further defined and implemented in
(selected) elements. The following fields will be contained in the message:
@ -101,7 +101,7 @@ Messages
Events
------
~~~~~~

4
docs/design/part-streams.txt
View file

@ -6,7 +6,7 @@ Streams
Stream objects
--------------
~~~~~~~~~~~~~~
The following objects are to be expected in the streaming thread:
@ -21,7 +21,7 @@ and live sources.
Typical stream
--------------
~~~~~~~~~~~~~~
A typical stream starts with a newsegment event that marks the
buffer timestamp range. After that buffers are send one after the

10
docs/design/part-synchronisation.txt
View file

@ -12,7 +12,7 @@ Synchronisation in a GstPipeline is achieved using the following 3 components:
A GstClock
----------
~~~~~~~~~~
This object provides a counter that represents the current time in nanoseconds.
This value is called the absolute_time.
@ -33,7 +33,7 @@ A GstClock always counts time upwards and does not necessarily start at 0.
Running time
------------
~~~~~~~~~~~~
After a pipeline selected a clock it will maintain the running_time based on the
selected clock. This running_time represents the total time spent in the PLAYING
@ -66,7 +66,7 @@ This value is monotonically increasing at the rate of the clock.
Timestamps
----------
~~~~~~~~~~
The GstBuffer timestamps and the preceeding NEW_SEGMENT event (See
part-streams.txt) define a transformation of the buffer timestamps to
@ -112,7 +112,7 @@ NS.stop and NS.accum == 0).
Synchronisation
---------------
~~~~~~~~~~~~~~~
As we have seen, we can get a running_time:
@ -161,7 +161,7 @@ synchronized buffers have the same timestamps.
Stream time
-----------
~~~~~~~~~~~
The stream time is also known as the position in the stream and is a value
between 0 and the total duration of the media file.

89
docs/design/part-trickmodes.txt
View file

@ -1,5 +1,5 @@
Trickmodes
==========
----------
GStreamer provides API for performing various trickmode playback. This includes:
@ -18,7 +18,7 @@ Seeking can both be done in a playback pipeline and a transcoding pipeline.
General seeking overview
------------------------
~~~~~~~~~~~~~~~~~~~~~~~~~
Consider a typical playback pipeline:
@ -35,56 +35,60 @@ position 0 and stopping at the total duration of the file.
When performing a seek, the following steps have to be taken by the application:
1) Create a seek event:
Create a seek event
^^^^^^^^^^^^^^^^^^^
The seek event contains:
The seek event contains:
- various flags describing:
- where to seek to (KEY_UNIT)
- how accurate the seek should be (ACCURATE)
- how to perform the seek (FLUSH)
- what to do when the stop position is reached (SEGMENT).
- extra playback options (SKIP)
- a format to seek in, this can be time, bytes, units (frames, samples), ...
- a playback rate, 1.0 is normal playback speed, positive values bigger than 1.0
mean fast playback. negative values mean reverse playback. A playback speed of
0.0 is not allowed (but is equivalent to PAUSING the pipeline).
- a start position, this value has to be between 0 and the total duration of the
file. It can also be relative to the previously configured start value.
- a stop position, this value has to be between 0 and the total duration. It can
also be relative to the previously configured stop value.
- various flags describing:
- where to seek to (KEY_UNIT)
- how accurate the seek should be (ACCURATE)
- how to perform the seek (FLUSH)
- what to do when the stop position is reached (SEGMENT).
- extra playback options (SKIP)
- a format to seek in, this can be time, bytes, units (frames, samples), ...
- a playback rate, 1.0 is normal playback speed, positive values bigger than 1.0
mean fast playback. negative values mean reverse playback. A playback speed of
0.0 is not allowed (but is equivalent to PAUSING the pipeline).
- a start position, this value has to be between 0 and the total duration of the
file. It can also be relative to the previously configured start value.
- a stop position, this value has to be between 0 and the total duration. It can
also be relative to the previously configured stop value.
See also gst_event_new_seek().
See also gst_event_new_seek().
2) Send the seek event to the pipeline with gst_element_send_event()
Send the seek event
^^^^^^^^^^^^^^^^^^^
By default the pipeline will send the event to all sink elements.
By default an element will forward the event upstream on all sinkpads.
Elements can modify the format of the seek event. The most common format is
GST_FORMAT_TIME.
Send the new seek event to the pipeline with gst_element_send_event().
One element will actually perform the seek, this is usually the demuxer or
source element. For more information on how to perform the different seek
types see part-seeking.txt.
By default the pipeline will send the event to all sink elements.
By default an element will forward the event upstream on all sinkpads.
Elements can modify the format of the seek event. The most common format is
GST_FORMAT_TIME.
For client side trickmode a NEW_SEGMENT event will be sent downstream with
the new rate and start/stop positions. All elements prepare themselves to
handle the rate (see below). The applied rate of the NEW_SEGMENT event will
be set to 1.0 to indicate that no rate adjustment has been done.
One element will actually perform the seek, this is usually the demuxer or
source element. For more information on how to perform the different seek
types see part-seeking.txt.
for server side trick mode a NEW_SEGMENT event is sent downstream with a
rate of 1.0 and the start/stop positions. The elements will configure themselves
for normal playback speed since the server will perform the rate conversions.
The applied rate will be set to the rate that will be applied by the server. This
is done to insure that the position reporting performed in the sink is aware
of the trick mode.
For client side trickmode a NEW_SEGMENT event will be sent downstream with
the new rate and start/stop positions. All elements prepare themselves to
handle the rate (see below). The applied rate of the NEW_SEGMENT event will
be set to 1.0 to indicate that no rate adjustment has been done.
When the seek succeeds, the _send_event() function will return TRUE.
for server side trick mode a NEW_SEGMENT event is sent downstream with a
rate of 1.0 and the start/stop positions. The elements will configure themselves
for normal playback speed since the server will perform the rate conversions.
The applied rate will be set to the rate that will be applied by the server. This
is done to insure that the position reporting performed in the sink is aware
of the trick mode.
When the seek succeeds, the _send_event() function will return TRUE.
Server side trickmode
---------------------
~~~~~~~~~~~~~~~~~~~~~
The source element operates in push mode. It can reopen a server connection requesting
a new byte or time position and a new playback speed. The capabilities can be queried
@ -131,7 +135,7 @@ playback speed or direction.
client side forward trickmodes
------------------------------
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The seek happens as stated above. a NEW_SEGMENT event is sent downstream with a rate
different from 1.0. Plugins receiving the NEW_SEGMENT can decide to perform the
@ -154,7 +158,7 @@ case, the demuxer would scale the timestamps and would set an applied rate of S.
client side backwards trickmode
-------------------------------
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
For backwards playback the following rules apply:
@ -202,7 +206,8 @@ For plugins the following rules apply:
In SKIP mode, the same algorithm as for forward SKIP mode can be used.
Notes:
Notes
~~~~~
- The clock/running_time keeps running forward.
- backwards playback potentially uses a lot of memory as frames and undecoded