mirrors/gstreamer
mirrors/gstreamer
1
1
Fork 0
mirror of https://gitlab.freedesktop.org/gstreamer/gstreamer.git synced 2024-11-08 18:39:54 +00:00
Code Issues 1 Projects Releases Wiki Activity

docs/design/: Many doc updates.

Browse source 
Original commit message from CVS:
* docs/design/part-TODO.txt:
* docs/design/part-clocks.txt:
* docs/design/part-events.txt:
* docs/design/part-gstbin.txt:
* docs/design/part-gstelement.txt:
* docs/design/part-gstpipeline.txt:
* docs/design/part-live-source.txt:
* docs/design/part-messages.txt:
* docs/design/part-overview.txt:
* docs/design/part-states.txt:
Many doc updates.
This commit is contained in:
Wim Taymans 2005-10-08 16:49:15 +00:00
parent bed8c238f0
commit 5b4e6c33c8
11 changed files with 316 additions and 90 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

14
ChangeLog
View file

@ -1,3 +1,17 @@
2005-10-08 Wim Taymans <wim@fluendo.com>
* docs/design/part-TODO.txt:
* docs/design/part-clocks.txt:
* docs/design/part-events.txt:
* docs/design/part-gstbin.txt:
* docs/design/part-gstelement.txt:
* docs/design/part-gstpipeline.txt:
* docs/design/part-live-source.txt:
* docs/design/part-messages.txt:
* docs/design/part-overview.txt:
* docs/design/part-states.txt:
Many doc updates.
2005-10-08 Wim Taymans <wim@fluendo.com>
* gst/gstevent.c:

13
docs/design/part-TODO.txt
View file

@ -10,10 +10,19 @@
- unlinking pads in the PAUSED state needs to make sure the stream thread is not
executing code. Can this be done with a flush to unlock all downstream chain
functions?
functions? Do we do this automatically or let the app handle this?
- implement clock selection as explained in part-gstpipeline.txt
- implement clock selection as explained in part-gstpipeline.txt.
- when a pipeline with a live source goes to PAUSED again, a sample is prerolled
in the sinks. This sample should be discarded, possibly with a flush event
started from the source.
- implement latency calculation for live sources.
- implement master/slave clocks.
- implement QOS.
- implement BUFFERSIZE.

15
docs/design/part-clocks.txt
View file

@ -5,8 +5,8 @@ To synchronize the different elements, the GstPipeline is responsible for
selecting and distributing a global GstClock for all the elements in it.
This selection happens whenever an element is added or removed from the
pipeline. Whever the clock changes in a pipeline, a message is posted on
the bus signaling the new clock to the application.
pipeline. Whever the clock changes in a pipeline, a NEW_CLOCK message is
posted on the bus signaling the new clock to the application.
The GstClock returns a monotonically increasing time with the method
_get_time(). Its accuracy and base time depends on the specific clock
@ -39,22 +39,22 @@ GstClock absolute time.
Timestamps
----------
The combination of the last DISCONT event and the buffer timestamps
The combination of the last NEWSEGMENT event and the buffer timestamps
express the presentation stream time of the buffer. The stream time
of a buffer is calculated as follows:
ST = TS - DT where: TS = buffer timestamp
DT = DISCONT timestamp
DT = NEWSEGMENT timestamp
ST = buffer stream time
The reason for not making the buffer times express the stream time directly
is for the following reasons:
- demuxers are easier if they can just copy the timestamps as encoded in
the file. The initial discont event would contain the lowest timestamp in
the file. The initial NEWSEGMENT event would contain the lowest timestamp in
the stream which makes the stream time start from 0.
- pipelines requiring retimestamping of buffers can efficiently adjust
the timestamp in the discont events and have all buffers retimestamped
the timestamp in the NEWSEGMENT events and have all buffers retimestamped
automatically.
- resync after different kinds of seeks is easier.
@ -110,7 +110,8 @@ invalid.
These clock operations do not operate on the stream time, so the callbacks
will also occur when not in PLAYING state as if the clock just keeps on
running.
running. Some clocks however do not progress when the element that provided
the clock is not PLAYING.
Clock implementations

47
docs/design/part-events.txt
View file

@ -16,6 +16,7 @@ Different types of events exist to implement various functionalities.
GST_EVENT_NEWSEGMENT: A new group of buffers with common start time
GST_EVENT_TAG: Stream metadata.
GST_EVENT_FILLER: Filler for sparse data streams
GST_EVENT_BUFFERSIZE: Buffer size requirements
GST_EVENT_QOS: A notification of the quality of service of the stream
GST_EVENT_SEEK: A seek should be performed to a new position in the stream
GST_EVENT_NAVIGATION: A navigation event.
@ -64,7 +65,7 @@ An EOS event sent on a srcpad returns GST_FLOW_UNEXPECTED.
The downstream element should forward the EOS event to its downstream peer
elements. This way the event will eventually reach the renderers which should
then post an EOS message on the bus.
then post an EOS message on the bus when in PLAYING.
An element might want to flush its internally queued data before forwarding
the EOS event downstream. This flushing can be done in the same thread as
@ -85,13 +86,15 @@ sending data.
An element that sends EOS on a pad should stop sending data on that pad. Source
elements typically pause() their task for that purpose.
By default, the pipeline collects all EOS events from all the sinks before
passing the EOS message to the application.
By default, a GstBin collects all EOS events from all its sinks before
posting the EOS message to its parent.
The EOS is only posted on the bus by the sink elements in the PLAYING state. If
the EOS event is received in the PAUSED state, it is queued until the element
goes to PLAYING.
A seek event on an element flushes all pending EOS messages.
NEWSEGMENT
-------------
@ -123,7 +126,7 @@ The NEWSEGMENT event can be send from both the application and the streaming
thread and should be serialized with the buffers.
Buffers should be clipped within the range indicated by the newsegment event
start and stop values. Sinks are allowed to drop buffers with timestamps out
start and stop values. Sinks must to drop buffers with timestamps out
of the indicated newsegment range.
If a newsegment arrives at an element not preceeded by a flush event, the
@ -131,6 +134,34 @@ streamtime of the pipeline will not be reset to 0 so any element that syncs
to the clock must use the stop times of the previous newsegment events to
make the buffer timestamps increasing.
TAG
---
The tag event is sent downstream when an element has discovered metadata
tags in a media file. Encoders can use this event to adjust their tagging
system. A tag is serialized with buffers.
FILLER
------
BUFFERSIZE
----------
An element can suggest a buffersize for downstream elements. This is
typically done by elements that produce data on multiple source pads
such as demuxers.
QOS
---
A QOS, or quality of service message, is generated in an element to report
to the upstream elements about the current quality of the stream. This
is typically done by the sinks that measure the amount of framedrops they
have.
SEEK
----
@ -194,11 +225,3 @@ of a navigation event such as a mouse movement or button click.
Navigation events travel upstream.
TAG
---
The tag event is sent downstream when an element has discovered metadata
tags in a media file. Encoders can use this event to adjust their tagging
system. A tag is serialized with buffers.

21
docs/design/part-gstbin.txt
View file

@ -36,7 +36,6 @@ operations on itself to all of its children. This includes:
- state changes
- index get/set
- clock gst/set
- scheduler set/get
The state change distribution is the most complex and is explained in
part-states.txt.
@ -50,7 +49,12 @@ messages from children. The bus for receiving messages from children is
distinct from the bin's own externally-visible GstBus.
Messages received from children are forwarded intact onto the bin's
external message bus, except for EOS which is handled specially.
external message bus, except for EOS and SEGMENT_START/DONE which are
handled specially.
STATE_CHANGED messages received from the children are used to trigger a
recalculation of the current state of the bin, as described in
part-states.txt.
The application can retrieve the external GstBus and integrate it in the
mainloop or it can just _pop() messages off in its own thread.
@ -75,10 +79,21 @@ The list of queued EOS messages is cleared when the bin goes to PAUSED
again. This means that all elements should repost the EOS message when going
to PLAYING again.
SEGMENT_START/DONE
------------------
* not implemented.
A bin collects SEGMENT_START messages but does not post them to the application.
It counts the number of SEGMENT_START messages and posts a SEGMENT_STOP message
to the application when an equal number of SEGMENT_STOP messages where received.
Subclassing
-----------
Subclasses of GstBin are free to implement their own add/remove implementations.
It is a good idea to update the GList of children so that the _iterate() functions
can still be used.
can still be used if the custom bin allows acces to its children.

7
docs/design/part-gstelement.txt
View file

@ -70,9 +70,4 @@ Ghost Pads
State
-----
Elements use state to determine what the are capable of doing at any given moment. The states are defined as follows:
NULL No state is held for the element
READY Devices are open
PLAYING
PAUSED
An element has a state. More info in part-states.txt.

18
docs/design/part-gstpipeline.txt
View file

@ -18,15 +18,21 @@ State changes
In addition to the normal state change procedure of its parent class
GstBin, the pipeline performs the following actions during a state change:
- NULL -> READY:
- set the bus to non-flushing
- READY -> PAUSED:
- Select and set a clock.
- reset the stream time to 0
- PAUSED -> PLAYING:
- calculate the stream time.
- Select and set a clock.
- calculate base time using the stream time.
The GstPipeline will also wait for any async state change to complete before
proceeding to the next state change. This is usefull for the application because
it does not have to deal with ASYNC state changes then.
- PAUSED -> PLAYING:
- calculate the stream time when the pipeline was stopped.
- READY -> NULL:
- set the bus to flushing
Clock selection
@ -65,7 +71,7 @@ When performing a seek on the pipeline element using gst_element_send_event(),
the pipeline performs the following actions:
- record the current state of the pipeline.
- set the pipeline to paused
- set the pipeline to paused if a FLUSHING seek is requested
- send the seek event to all sinks
- when a FLUSH seek is done, the stream_time is set 0 again.
- restore old state of the pipeline.

2
docs/design/part-live-source.txt
View file

@ -20,7 +20,7 @@ the get_state() function would block on the sinks.
A gstbin therefore always performs a zero timeout get_state() on its
elements to discover the NO_PREROLL (and ERROR) elements before performing
a blocking wait on all elements.
a blocking wait.
Scheduling

21
docs/design/part-messages.txt
View file

@ -48,13 +48,22 @@ Message types
GST_MESSAGE_STATE_CHANGED:
An element changed state in the pipeline. The message carries the old an new
state of the element.
An element changed state in the pipeline. The message carries the old, new
and pending state of the element.
GST_MESSAGE_STEP_DONE:
An element stepping frames has finished.
GST_MESSAGE_CLOCK_PROVIDE:
An element notifies it capability of providing a clock for the pipeline.
GST_MESSAGE_CLOCK_LOST:
The current clock as selected by the pipeline became unusable. The pipeline
will select a new clock on the next PLAYING state change.
GST_MESSAGE_NEW_CLOCK:
A new clock was selected for the pipeline.
@ -71,12 +80,16 @@ Message types
GST_MESSAGE_APPLICATION:
An element posted an application specific message.
The application posted a message.
GST_MESSAGE_ELEMENT:
Element-specific message, see the specific element's documentation
GST_MESSAGE_SEGMENT_START:
An element started playback of a new segment. This message is not forwarded
the the application but is used internally to scheduler SEGMENT_DONE messages.
the the application but is used internally to schedule SEGMENT_DONE messages.
GST_MESSAGE_SEGMENT_DONE:

37
docs/design/part-overview.txt
View file

@ -19,7 +19,7 @@ Introduction
is an object that performs some action on a multimedia stream such as:
- read a file
- decode or encoder between formats
- decode or encode between formats
- capture from a hardware device
- render to a hardware device
- mix or multiplex multiple streams
@ -283,7 +283,7 @@ Pipeline clock
of one GST_SECOND per second. Clock values are expressed in nanoseconds.
Elements use the clock time to synchronized the playback of data.
Before the pipeline is set to PAUSED, the pipeline asks each element if they can
Before the pipeline is set to PLAYING, the pipeline asks each element if they can
provide a clock. The clock is selected in the following order:
- If the application selected a clock, use that one.
@ -316,13 +316,14 @@ Pipeline states
All intermediate states are traversed for each element resulting in the following
chain of state changes:
alsasink to READY: the audio device is opened
mp3dec to READY: the decoding library is initialized
filesrc to READY: the file is opened
alsasink to PAUSED: alsasink is a sink and returns ASYNC because it did not receive
data yet.
mp3dec to PAUSED: nothing happens
filesrc to PAUSED: a thread is started to push data to mp3dec
alsasink to READY: the audio device is probed
mp3dec to READY: nothing happens.
filesrc to READY: the file is probed
alsasink to PAUSED: the audio device is opened. alsasink is a sink and returns
ASYNC because it did not receive data yet.
mp3dec to PAUSED: the decoding library is initialized
filesrc to PAUSED: the file is opened and a thread is started to push data to
mp3dec
At this point data flows from filesrc to mp3dec and alsasink. Since mp3dec is PAUSED,
it accepts the data from filesrc on the sinkpad and starts decoding the compressed
@ -336,7 +337,7 @@ Pipeline states
Alsasink then receives the buffer, inspects the caps and reconfigures itself to process
the buffer. Since it received the first buffer of samples, it completes the state change
to the PAUSED state. At this point the pipeline is prerolled and all elements have
samples.
samples. Alsasink is now also capable of providing a clock to the pipeline.
Since alsasink is now in the PAUSED state it blocks while receiving the first buffer. This
effectively blocks both mp3dec and filesrc in their gst_pad_push().
@ -344,9 +345,9 @@ Pipeline states
Since all elements now return SUCCESS from the gst_element_get_state() function,
the pipeline can be put in the PLAYING state.
Before going to PLAYING, the pipeline samples the current time of the clock. This is
the base time. It then distributes this time to all elements. Elements can then
synchronize against the clock using the buffer timestamp+base time.
Before going to PLAYING, the pipeline select a clock and samples the current time of
the clock. This is the base time. It then distributes this time to all elements.
Elements can then synchronize against the clock using the buffer timestamp+base time.
The following chain of state changes then takes place:
@ -402,8 +403,8 @@ Pipeline EOS
Pipeline READY
--------------
When a running pipeline is set from the RUNNING to READY the following actions
occur in the pipeline:
When a running pipeline is set from the PLAYING to READY state, the following
actions occur in the pipeline:
alsasink to PAUSED: alsasink blocks and completes the state change on the
next sample. If the element was EOS, it does not wait for
@ -488,8 +489,8 @@ Pipeline seeking
Since the pipeline is still PAUSED, this will preroll the next media sample in the
sinks.
The last step in the seek operation is then to adjust the media time of the pipeline
to the new position and to set the pipeline back to PLAYING.
The last step in the seek operation is then to adjust the stream time of the pipeline
to 0 and to set the pipeline back to PLAYING.
The sequence of events in out mp3 playback example.
@ -511,6 +512,6 @@ Pipeline seeking
--------------------------> 4) FLUSH done event
--------------------------> 5) NEWSEGMENT event
| e) update stream time
| e) update stream time to 0
| f) PLAY pipeline

211
docs/design/part-states.txt
View file

@ -23,6 +23,64 @@ We call the sequence NULL->PLAYING an upwards state change and PLAYING->NULL
a downwards state change.
State transitions
-----------------
the following state changes are possible:
NULL -> READY
- The element must check if the resources it needs are available.
Audiosinks typically try to probe the device.
READY -> PAUSED
- The element opens the device and prepares itself for PLAYING.
- the element pads are activated in order to receive data in PAUSED.
streaming threads are started.
- some elements might need to return ASYNC and complete the state change
when they have enough information. It is a requirement for sinks to
return ASYNC and complete the state change when they receive the first
buffer or EOS event (prerol). Sinks also block the dataflow when in PAUSED.
- a pipeline resets the stream time to 0.
- live sources return NO_PREROLL and don't generate data.
PAUSED -> PLAYING
- most elements ignore this state change.
- The pipeline selects a clock and distributes this to all the children
before setting them to PLAYING. This means that it is only alowed to
synchronize on the clock in the PLAYING state.
- The pipeline uses the clock and the stream time to calculate the base time.
The base time is distributed to all children when performing the state
change.
- sink elements stop blocking on the preroll buffer or event and start
rendering the data.
- sinks can post the EOS message in the PLAYING state. It is not allowed to
post EOS when not in the PLAYING state.
- while streaming in PAUSED or PLAYING elements can create and remove
dynamic pads.
- live sources start generating data and return SUCCESS.
PLAYING -> PAUSED
- most elements ignore this state change.
- The pipeline calculates the stream time based on the last selected clock
and the base time. It stores this information to continue playback when
going back to the PLAYING state.
- sinks unblock any clock wait calls.
- sinks return ASYNC from this state change and complete the state change
when they receive a buffer or an EOS event.
- any queued EOS messages are removed since they will be reposted when going
back to the PLAYING state.
- live sources stop generating data and return NO_PREROLL.
PAUSED -> READY
- sinks unblock any waits in the preroll.
- elements unblock any waits on devices
- the element pads are deactivated so that streaming becomes impossible and
all streaming threads are stopped.
READY -> NULL
- element removes any dynamically created pads
State variables
---------------
@ -32,18 +90,17 @@ the STATE_LOCK.
The STATE_LOCK protects 3 element variables:
- STATE
- PENDING_STATE
- STATE_ERROR flag
- NO_PREROLL flag
- STATE_NEXT
- STATE_PENDING
- STATE_RETURN
The STATE always reflects the current state of the element. The PENDING_STATE
always reflects the required state of the element. The PENDING_STATE can be
VOID_PENDING if the element is in the right state. The STATE_ERROR flag
indicates that an error occured while doing the last state change.
The NO_PREROLL flag indicates that the element said it was not able to preroll
in its last state change. This flag is used in live sources.
The STATE always reflects the current state of the element.
The STATE_NEXT reflects the next state the element will go to.
The STATE_PENDING always reflects the required state of the element.
The STATE_RETURN reflects the last return value of a state change.
The STATE_NEXT and STATE_PENDING can be VOID_PENDING if the element is in
the right state.
Setting state on elements
-------------------------
@ -68,13 +125,16 @@ The _set_state() function can return 3 possible values:
will not be able to produce data in the PAUSED state.
In the case of an async state change, it is possible to proceed to the next
state before the current state change completed. After receiving an ASYNC return
value, you can use _element_get_state() to poll the status of the element.
state before the current state change completed, however, the element will only
get to this next state before completing the previous ASYNC state change.
After receiving an ASYNC return value, you can use _element_get_state() to poll
the status of the element. If the polling returns SUCCESS, the element completed
the state change to the last requested state with _set_state().
When setting the state of an element, the PENDING_STATE is set to the required
state and the STATE_ERROR flag is cleared. Then the state change function of the
element is called and the result of that function is used to update the STATE,
PENDING_STATE and STATE_ERROR flags. If the function returned ASYNC, this result
When setting the state of an element, the STATE_PENDING is set to the required
state. Then the state change function of the element is called and the result of
that function is used to update the STATE and STATE_RETURN fields, STATE_NEXT,
STATE_PENDING and STATE_RETURN fields. If the function returned ASYNC, this result
is immediatly returned to the caller.
@ -143,21 +203,10 @@ ASYNC, the function returns ASYNC as well.
If after calling the state function on all children, one of the children returned
NO_PREROLL, the function returns NO_PREROLL as well.
The current state of the bin can be retrieved with _get_state(). This function will
call the _get_state() function on all the elements.
The current state of the bin can be retrieved with _get_state().
First the bin will perform a _get_state() on all children with a 0 timeout. This
is to find any children with an ERROR/NO_PREROLL result value.
Then the bin performs the _get_state() with the requested timeout. The reason for
the 2 phases is that when an ERROR or NO_PREROLL result is found, a blocking
wait on the sinks might never return.
The _get_state() function will be called on the children with the same timout value
so the function can potentially block timeout*num_children.
The bin also updates its state variables after polling its children, this means that
the state variables of the bin are only updated after calling _get_state() on the bin.
If the bin is performing an ASYNC state change, it will automatically update its
current state fields when it receives state messages from the children.
Implementing states in elements
@ -167,6 +216,106 @@ READY
-----
upward state change
-------------------
Upward state changes always return ASYNC either if the STATE_PENDING is
reached or not.
Element:
A -> B => SUCCESS
- commit state
A -> B => ASYNC
- no commit state
- element commits state ASYNC
A -> B while ASYNC
- update STATE_PENDING state
- no commit state
- no change_state called on element
Bin:
A->B: all elements SUCCESS
- commit state
A->B: some elements ASYNC
- no commit state
- listen for commit messages on bus
- for each commit message, poll elements
- if no ASYNC elements, commit state, continue state change
to STATE_PENDING
downward state change
----------------------
Downward state changes only return ASYNC if the final state is ASYNC.
This is to make sure that it's not needed to wait for an element to
complete the preroll or other ASYNC state changes when one only wants to
shut down an element.
Element:
A -> B => SUCCESS
- commit state
A -> B => ASYNC not final state
- commit state on behalf of element
A -> B => ASYNC final state
- element will commit ASYNC
Bin:
A -> B -> SUCCESS
- commit state
A -> B -> ASYNC not final state
- commit state on behalf of element, continue state change
A -> B => ASYNC final state
- no commit state
- listen for commit messages on bus
- for each commit message, poll elements
- if no ASYNC elements, commit state
Locking overview (element)
--------------------------
set_state(element) change_state (element) stream_thread commit_state (element)
| | | |
| | | |
STATE_LOCK | | |
| | | |
|------------------------>| | |
| | | |
| | | |
| | (start_task) | |
| | | |
| | STREAM_LOCK |
| | | |
|<------------------------| | |
| ASYNC | |
STATE_UNLOCK | |
| .....sync........ STATE_LOCK |
ASYNC |----------------->|
| |
| |---> post_message(ASYNC)
| |---> if (!final) change_state (next)
| | else SIGNAL
|<-----------------|
STATE_UNLOCK
|
STREAM_UNLOCK