mirrors/gstreamer
mirrors/gstreamer
1
1
Fork 0
mirror of https://gitlab.freedesktop.org/gstreamer/gstreamer.git synced 2025-02-17 03:35:21 +00:00
Code Issues 1 Projects Releases Wiki Activity

docs/design/: Add doc about synchronisation

Browse source 
Original commit message from CVS:
* docs/design/Makefile.am:
* docs/design/part-synchronisation.txt:
Add doc about synchronisation
* docs/design/draft-latency.txt:
* docs/design/part-TODO.txt:
* docs/design/part-clocks.txt:
* docs/design/part-events.txt:
* docs/design/part-gstbus.txt:
* docs/design/part-gstpipeline.txt:
* docs/design/part-live-source.txt:
* docs/design/part-messages.txt:
* docs/design/part-overview.txt:
* docs/design/part-streams.txt:
* docs/design/part-trickmodes.txt:
Documentation updates.
This commit is contained in:
Wim Taymans 2007-03-07 17:13:17 +00:00
parent afac0e6e73
commit 88be8ba00f
14 changed files with 279 additions and 127 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

19
ChangeLog
View file

@ -1,3 +1,22 @@
2007-03-07 Wim Taymans <wim@fluendo.com>
* docs/design/Makefile.am:
* docs/design/part-synchronisation.txt:
Add doc about synchronisation
* docs/design/draft-latency.txt:
* docs/design/part-TODO.txt:
* docs/design/part-clocks.txt:
* docs/design/part-events.txt:
* docs/design/part-gstbus.txt:
* docs/design/part-gstpipeline.txt:
* docs/design/part-live-source.txt:
* docs/design/part-messages.txt:
* docs/design/part-overview.txt:
* docs/design/part-streams.txt:
* docs/design/part-trickmodes.txt:
Documentation updates.
2007-03-07 Jan Schmidt <thaytan@mad.scientist.com>
* gstreamer.doap:

1
docs/design/Makefile.am
View file

@ -39,6 +39,7 @@ EXTRA_DIST = \
part-standards.txt \
part-states.txt \
part-streams.txt \
part-synchronisation.txt \
part-TODO.txt \
part-trickmodes.txt

58
docs/design/draft-latency.txt
View file

@ -199,26 +199,21 @@ that no sink is set to PLAYING before it is prerolled.
In order to do this, the pipeline (at the GstBin level) keeps track of all
elements that require preroll (the ones that return ASYNC from the state
change). It keeps a custom GstBinState GST_MESSAGE_ELEMENT internally for
those elements.
change). These elements posted a ASYNC_START message without a matching
ASYNC_DONE message.
When the pipeline did not receive a NO_PREROLL state change return from any
element, it can forget about the GstBinState messages because the state change
to PLAYING will proceed when all elements commited their state when they are
prerolled automatically.
The pipeline will not change the state of the elements that are still doing an
ASYNC state change.
When the pipeline received a NO_PREROLL state change return from an element, it
keeps the ELEMENT messages. The pipeline will not change the state of the
elements that are still doing an ASYNC state change.
When an ASYNC element prerolls, it commits its state to PAUSED and posts an
ASYNC_DONE message. The pipeline notices this ASYNC_DONE message and matches it
with the ASYNC_START message it cached for the corresponding element.
When an ASYNC element prerolls, it commits its state to PAUSED and posts a
PREROLLED message. The pipeline notices this PREROLLED message and matches it
with the GstBinState message it cached for the corresponding element.
When all ASYNC_START messages are matched with an ASYNC_DONE message, the
pipeline proceeds with setting the elements to the final state again.
When all GstBinState messages are matched with a PREROLLED message, the
pipeline proceeds with setting the PREROLLED sinks to their pending state.
The base time of the element was already set by the pipeline when it changed the
nopreroll element to PLAYING. This operation has to be performed in the
NO_PREROLL element to PLAYING. This operation has to be performed in the
separate async state change thread (like the one currently used for going from
PAUSED->PLAYING in a non-live pipeline).
@ -235,7 +230,7 @@ Latency compensation
As an extension to the revised state changes we can perform latency calculation
and compensation before we proceed to the PLAYING state.
When the pipeline collected all PREROLLED messages it can calculate the global
When the pipeline collected all ASYNC_DONE messages it can calculate the global
latency as follows:
- perform a latency query on all sinks.
@ -262,11 +257,12 @@ ex2:
MIN (50, 40) = 40 >= 33 -> latency = 33
The latency is set on the pipeline by sending a LATENCY event to the sinks
that posted the PREROLLED message. This event configures the total latency on
the sinks. The sink forwards this LATENCY event upstream so that
intermediate elements can configure themselves as well.
in the pipeline. This event configures the total latency on the sinks. The
sink forwards this LATENCY event upstream so that intermediate elements can
configure themselves as well.
After this step, the pipeline continues setting the pending state on the sinks.
After this step, the pipeline continues setting the pending state on its
elements.
A sink adds the latency value, received in the LATENCY event, to
the times used for synchronizing against the clock. This will effectively
@ -289,23 +285,19 @@ A flush in a pipeline can happen in the following cases:
- flush preformed by an element
- after receiving a navigation event (DVD, ...)
When a playing sink is flushed by a FLUSH_START event, a LOST_PREROLL message is
posted by the element. As part of the message, the state of the element is
included. The element also goes to a pending PAUSED state.
When a playing sink is flushed by a FLUSH_START event, an ASYNC_START message is
posted by the element. As part of the message, the fact that the element got
flushes is included. The element also goes to a pending PAUSED state and has to
be set to the PLAYING state again later.
The message is kept in a GstBinState message by the parent bin. When the element
prerolls, it posts a PREROLLED message.
The ASYNC_START message is kept by the parent bin. When the element prerolls,
it posts an ASYNC_DONE message.
When all LOST_PREROLL messages are matched with a PREROLLED message, the bin
will capture a new base time from the clock and will bring all the prerolled
sinks back to PLAYING (or whatever their state was when they posted the
LOST_PREROLL message) after setting the new base time on them. It's also possible
When all ASYNC_START messages are matched with an ASYNC_DONE message, the bin
will capture a new base_time from the clock and will bring all the sinks back to
PLAYING after setting the new base time on them. It's also possible
to perform additional latency calculations and adjustments before doing this.
The difference with the NEED_PREROLL/PREROLLED and LOST_PREROLL/PREROLLED
message pair is that the latter makes the pipeline acquire a new base time for
the PREROLLED elements.
Dynamically adjusting latency
-----------------------------

3
docs/design/part-TODO.txt
View file

@ -22,9 +22,6 @@ API/ABI
- use | instead of + as divider in serialization of Flags
(gstvalue/gststructure)
- with the addition the PREROLLED message, we can probably get rid of the
STATE_DIRTY message.
IMPLEMENTATION
--------------

66
docs/design/part-clocks.txt
View file

@ -7,7 +7,7 @@ implementation but time is always expessed in nanoseconds. Since the
baseline of the clock is undefined, the clock time returned is not
meaningfull in itself, what matters are the deltas between two clock
times.
The time reported by the clock is called the absolute time.
The time reported by the clock is called the absolute_time.
Clock Selection
@ -30,68 +30,8 @@ CLOCK_LOST message is posted. The application must then set the pipeline to
PAUSED and PLAYING again in order to let the pipeline select a new clock
and distribute a new base time.
Time in GStreamer
-----------------
The absolute time is used to calculate the running time. The running time
is defined as follows:
- If the pipeline is NULL/READY, the running time is undefined.
- In PAUSED, the running time remains at the time when it was last
PAUSED. When the stream is PAUSED for the first time, the running time
is 0.
- In PLAYING, the running time is the delta between the absolute time
and the base time. The base time is defined as the absolute time minus
the running time at the time when the pipeline is set to PLAYING.
- after a flushing seek, the running time is set to 0 (see part-seeking.txt)
The running time is completely managed by the GstPipeline object using the
GstClock absolute time. It basically represents the elapsed time that the
pipeline spend in the PLAYING state.
Timestamps
----------
NOTE: this info is inaccurate, see part-synchronisation.txt for how the clock
time is used to synchronize buffer timestamps.
The combination of the last NEWSEGMENT event and the buffer timestamps
express the presentation stream time of the buffer. The stream time
of a buffer is calculated as follows:
ST = TS - DT where: TS = buffer timestamp
DT = NEWSEGMENT timestamp
ST = buffer stream time
The reason for not making the buffer times express the stream time directly
is for the following reasons:
- demuxers are easier if they can just copy the timestamps as encoded in
the file. The initial NEWSEGMENT event would contain the lowest timestamp in
the stream which makes the buffer stream time start from 0.
- pipelines requiring retimestamping of buffers can efficiently adjust
the timestamp in the NEWSEGMENT events and have all buffers retimestamped
automatically.
- resync after different kinds of seeks is easier.
If an element wants to synchronize a buffer to the clock it needs to first
calculate the buffer stream time and then bring the stream time to the
absolute clock time.
Converting a timestamp (in stream time) to absolute time is performed using
the following formula:
AT = BT + ST where: AT = absolute time
BT = base time
ST = stream time
The pipeline base time is propagated to all the element during the PAUSED
to PLAYING state change. All elements are therefore able to convert the
stream time to the absolute time. It is possible to specify an aditional
delay to the base time to compensate for the delay it takes to perform
the state change using the GstPipeline "delay" property.
The clock selection is performed as part of the state change from PAUSED to
PLAYING and is described in part-states.txt.
Clock features

2
docs/design/part-events.txt
View file

@ -242,7 +242,7 @@ contains a single GstClockTime with the required latency. The latency value is
calculated by the pipeline and distributed to all sink elements before they are
set to PLAYING. The sinks will add the configured latency value to the
timestamps of the buffer in order to delay their presentation.
See also part-latency.txt.
(See also part-latency.txt).
DRAIN

7
docs/design/part-gstbus.txt
View file

@ -7,8 +7,9 @@ a first-in first-out way from the streaming threads to the application.
Since the application typically only wants to deal with delivery of these
messages from one thread, the GstBus will marshall the messages between
different threads. This is important since the actual streaming of media
is done in another thread than the application. It is also important to not
block the streaming thread while the application deals with the message.
is done in another threads (streaming threads) than the application. It is
also important to not block the streaming threads while the application deals
with the message.
The GstBus provides support for GSource based notifications. This makes it
possible to handle the delivery in the glib mainloop. Different GSources
@ -21,7 +22,7 @@ posted message.
The bus can be polled with the gst_bus_poll() method. This methods blocks
up to the specified timeout value until one of the specified messages types
is posted on the bus. The application can then _pop() the messages from the
bus to handle them.
bus to handle them.
It is also possible to get messages from the bus without any thread
marshalling with the gst_bus_set_sync_handler() method. This makes it

19
docs/design/part-gstpipeline.txt
View file

@ -7,8 +7,10 @@ children with a clock.
A GstPipeline also provides a toplevel GstBus (see part-gstbus.txt)
The pipeline also calculates the running_time based on the selected
clock (see part-clocks.txt).
clock (see also clocks.txt and part-synchronisation.txt).
The pipeline will calculate a global latency for the elements in the pipeline.
(See also part-latency.txt).
State changes
-------------
@ -25,6 +27,7 @@ GstBin, the pipeline performs the following actions during a state change:
- PAUSED -> PLAYING:
- Select and a clock.
- calculate base_time using the running_time.
- calculate and distribute latency.
- set clock and base_time on all elements before performing the
state change.
@ -35,7 +38,7 @@ GstBin, the pipeline performs the following actions during a state change:
- set the bus to flushing (when auto-flushing is enabled)
The running_time represents the total elapsed time, measured in clock units,
that the pipeline spent in the PLAYING state.
that the pipeline spent in the PLAYING state (see part-synchronisation.txt).
Clock selection
@ -43,7 +46,7 @@ Clock selection
Since all of the children of a GstPipeline must use the same clock, the
pipeline must select a clock. This clock selection happens when the pipeline
gopes to the PLAYING state.
goes to the PLAYING state.
The default clock selection algorithm works as follows:
@ -61,7 +64,7 @@ The default clock selection algorithm works as follows:
a more upstream element.
- use GstSystemClock, this only happens when no element provides a
clock.
usable clock.
The application can influence this clock selection with two methods:
gst_pipeline_use_clock() and gst_pipeline_auto_clock().
@ -74,3 +77,11 @@ possible.
The _auto_clock() method removes the fixed clock and reactivates the auto-
matic clock selection algorithm described above.
GstBus
------
A GstPipeline provides a GstBus to the application. The bus can be retrieved
with gst_pipeline_get_bus() and can then be used to retrieve messages posted by
the elements in the pipeline (see part-gstbus.txt).

3
docs/design/part-live-source.txt
View file

@ -15,7 +15,8 @@ since such a buffer might never arrive.
Live sources return NO_PREROLL when going to the PAUSED state to inform the
bin/pipeline that this element will not be able to produce data in the
PAUSED state.
PAUSED state. NO_PREROLL should be returned for both READY->PAUSED and
PLAYING->PAUSED.
When performing a get_state() on a bin with a non-zero timeout value, the
bin must be sure that there are no live sources in the pipeline because else

12
docs/design/part-messages.txt
View file

@ -10,7 +10,7 @@ requiring an API change while allowing a wide range of different types
of messages.
Messages are posted by objects in the pipeline and are passed to the
application using the GstBus.
application using the GstBus (See also part-gstbus.txt and part-gstpipeline.txt).
Message types
@ -112,14 +112,14 @@ Message types
An element posts this message when it has detected or updated the stream duration.
GST_MESSAGE_LOST_PREROLL:
GST_MESSAGE_ASYNC_START:
Posted by sinks when they lose their preroll buffer in the PAUSED or PLAYING
state caused by a FLUSH_START event.
Posted by sinks when they start an asynchronous state change.
GST_MESSAGE_PREROLLED:
GST_MESSAGE_ASYNC_DONE:
Posted by sinks when they receive the first data buffer.
Posted by sinks when they receive the first data buffer and complete the
asynchronous state change.
GST_MESSAGE_LATENCY:

9
docs/design/part-overview.txt
View file

@ -166,6 +166,7 @@ Pipeline
- Manage running_time based on the selected clock. Running_time is the elapsed
time the pipeline spent in the PLAYING state and is used for
synchronisation.
- Manage latency in the pipeline.
- Provide means for elements to comunicate with the application by the GstBus.
- Manage the global state of the elements such as Errors and end-of-stream.
@ -365,7 +366,8 @@ Pipeline states
Before going to PLAYING, the pipeline select a clock and samples the current time of
the clock. This is the base_time. It then distributes this time to all elements.
Elements can then synchronize against the clock using the buffer timestamp+base time.
Elements can then synchronize against the clock using the buffer running_time +
base_time (See also part-synchronisation.txt).
The following chain of state changes then takes place:
@ -385,7 +387,7 @@ Pipeline status
The bus is distributed to all elements added to the pipeline. The elements use the bus
to post messages on. Various message types exist such as ERRORS, WARNINGS, EOS,
STATE_CHANGED, etc..
STATE_CHANGED, etc..
The pipeline handles EOS messages received from elements in a special way. It will
only forward the message to the application when all sink elements have posted an
@ -412,7 +414,8 @@ Pipeline EOS
The EOS event will eventually arrive in the sink element. The sink will then post
an EOS message on the bus to inform the pipeline that a particular stream has
finished. When all sinks have reported EOS, the pipeline forwards the EOS message
to the application.
to the application. The EOS message is only forwarded to the application in the
PLAYING state.
When in EOS, the pipeline remains in the PLAYING state, it is the applications
responsability to PAUSE or READY the pipeline. The application can also issue

30
docs/design/part-streams.txt
View file

@ -34,21 +34,31 @@ Typical stream
1) NEW_SEGMENT, rate, start/stop, time
- marks valid buffer timestamp range (start, stop)
- marks stream_time of buffers (time)
- marks playback rate (rate)
- marks applied rate (applied_rate)
- marks stream_time of buffers (time). This is the stream time of buffers
with a timestamp of NS.start.
- marks playback rate (rate). This is the required playback rate.
- marks applied rate (applied_rate). This is the already applied playback
rate. (See also part-trickmodes.txt)
2) N buffers
- displayable buffers are between start/stop of the NEW_SEGMENT
- displayable buffers are between start/stop of the NEW_SEGMENT. Buffers
outside the segment range should be dropped or clipped.
- display_time: (B.timestamp - NS.start) * NS.abs_rate
* used to calculate stream_time and sync_time
- running_time:
if (NS.rate > 0.0)
running_time = (B.timestamp - NS.start) / NS.abs_rate + NS.accum
else
running_time = (NS.stop - B.timestamp) / NS.abs_rate + NS.accum
- stream_time: display_time + NS.time
* current position in stream between 0 and duration
* used to synchronize against the clock (See also
part-synchronisation.txt).
- sync_time: display_time + NS.accum + base_time
* used to synchronize against the clock.
- stream_time:
stream_time = (B.timestamp - NS.start) * NS.abs_applied_rate + NS.time
* current position in stream between 0 and duration.
3) EOS
- marks the end of data, nothing is to be expected after EOS

176
docs/design/part-synchronisation.txt Normal file
View file

@ -0,0 +1,176 @@
Synchronisation
---------------
This document outlines the techniques used for doing synchronised playback of
multiple streams.
Synchronisation in a GstPipeline is achieved using the following 3 components:
- a GstClock, which is global for all elements in a GstPipeline.
- Timestamps on a GstBuffer.
- the NEW_SEGMENT event preceeding the buffers.
A GstClock
----------
This object provides a counter that represents the current time in nanoseconds.
This value is called the absolute_time.
Different sources exist for this counter:
- the systemtime (with g_get_current_time() and with microsend accuracy)
- an audio device (based on number of samples played)
- a network source based on packets received + timestamps in those packets (a
typical example is an RTP source)
- ...
In GStreamer any element can provide a GstClock object that can be used in the
pipeline. The GstPipeline object will select a clock from all the providers and
will distribute it to all other elements (see part-gstpipeline.txt).
A GstClock always counts time upwards and does not necessarily start at 0.
Running time
------------
After a pipeline selected a clock it will maintain the running_time based on the
selected clock. This running_time represents the total time spent in the PLAYING
state and is calculated as follows:
- If the pipeline is NULL/READY, the running_time is undefined.
- In PAUSED, the running_time remains at the time when it was last
PAUSED. When the stream is PAUSED for the first time, the running_time
is 0.
- In PLAYING, the running_time is the delta between the absolute_time
and the base time. The base time is defined as the absolute_time minus
the running_time at the time when the pipeline is set to PLAYING.
- after a flushing seek, the running_time is set to 0 (see part-seeking.txt).
This is accomplished by redistributing a new base_time to the elements that
got flushed.
This algorithm captures the running_time when the pipeline is set from PLAYING
to PAUSED and restores this time based on the current absolute_time when going
back to PLAYING. This allows for both clocks that progress when in the PAUSED
state (systemclock) and clocks that don't (audioclock).
The clock and pipeline now provides a running_time to all elements that want to
perform synchronisation. Indeed, the running time can be observed in each
element (during the PLAYING state) as:
running_time = absolute_time - base_time
Timestamps
----------
The GstBuffer timestamps and the preceeding NEW_SEGMENT event (See
part-streams.txt) define a transformation of the buffers to running_time as
follows:
The following notation is used:
B: GstBuffer
- B.timestamp = buffer timestamp (GST_BUFFER_TIMESTAMP)
NS: NEWSEGMENT event preceeding the buffers.
- NS.start: start field in the NEWSEGMENT event
- NS.rate: rate field of NEWSEGMENT event
- NS.abs_rate: absolute value of rate field of NEWSEGMENT event
- NS.time: time field in the NEWSEGMENT event
- NS.accum: total accumulated time of all previous segments
Valid buffers for synchronisation are those with B.timestamp between NS.start
and NS.stop. All other buffers outside this range should be dropped or clipped
to these boundaries (see also part-segments.txt).
The following transformation to running_time exist:
if (NS.rate > 0.0)
running_time = (B.timestamp - NS.start) / NS.abs_rate + NS.accum
else
running_time = (NS.stop - B.timestamp) / NS.abs_rate + NS.accum
The first displayable buffer will yield a value of 0 (since B.timestamp ==
NS.start and NS.accum == 0).
For NS.rate > 1.0, the timestamps will be scaled down to increase the playback
rate. Likewise, a rate between 0.0 and 1.0 will slow down playback.
For negative rates, timestamps are received stop NS.stop to NS.start so that the
first buffer received will be transformed into running_time of 0 (B.timestamp ==
NS.stop and NS.accum == 0).
Synchronisation
---------------
As we have seen, we can get a running_time:
- using the clock and the element's base_time with:
C.running_time = absolute_time - base_time
- using the buffer timestamp and the preceeding NEWSEGMENT event as (assuming
positive playback rate):
B.running_time = (B.timestamp - NS.start) / NS.abs_rate + NS.accum
We prefix C. and B. before the two running times to note how they were
calculated.
The task of synchronized playback is to make sure that we play be buffer with
B.running_time at the moment when the clock reaches the same C.running_time.
Thus the following must hold:
B.running_time = C.running_time
expaning:
B.running_time = absolute_time - base_time
or:
absolute_time = B.running_time + base_time
The absolute_time when a buffer with B.running_time should be played is noted
with B.sync_time. Thus:
B.sync_time = B.running_time + base_time
One then waits for the clock to reach B.sync_time before rendering the buffer in
the sink (See also part-clocks.txt).
For multiple streams this means that buffers with the same running_time are to
be displayed at the same time.
A demuxer must make sure that the NEWSEGMENT it emits on its output pads yield
the same running_time for buffers that should be played synchronized. This
usually means sending the same NEWSEGMENT on all pads and making sure that the
synchronized buffers have the same timestamps.
Stream time
-----------
The stream time is also known as the position in the stream and is a value
between 0 and the total duration of the media file.
It is the stream time that is used for:
- report the POSITION query in the pipeline
- the position used in seek queries
- the position used to synchronize controller values
Stream time is calculated using the buffer times and the preceeding NEWSEGMENT
event as follows:
stream_time = (B.timestamp - NS.start) * NS.abs_applied_rate + NS.time
For negative rates, B.timestamp will go backwards from NS.stop to NS.start,
making the stream time go backwards.
Note that the stream time is never used for synchronisation against the clock.

1
docs/design/part-trickmodes.txt
View file

@ -199,3 +199,4 @@ Notes:
- backwards playback potentially uses a lot of memory as frames and undecoded
data gets buffered.