mirrors/gstreamer
mirrors/gstreamer
1
1
Fork 0
mirror of https://gitlab.freedesktop.org/gstreamer/gstreamer.git synced 2024-11-23 10:11:08 +00:00
Code Issues 1 Projects Releases Wiki Activity

docs: improve clock chapter

Browse source
This commit is contained in:
Wim Taymans 2012-10-08 13:22:30 +02:00
parent 402f0166c1
commit d36377ba72
2 changed files with 288 additions and 70 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

328
docs/manual/advanced-clocks.xml
View file

@ -1,91 +1,295 @@
<chapter id="chapter-clocks">
<title>Clocks in GStreamer</title>
<title>Clocks and synchronization in &GStreamer;</title>
<para>
To maintain sync in pipeline playback (which is the only case where this
really matters), &GStreamer; uses <emphasis>clocks</emphasis>. Clocks
are exposed by some elements, whereas other elements are merely clock
slaves. The primary task of a clock is to represent the time progress
according to the element exposing the clock, based on its own playback
rate. If no clock provider is available in a pipeline, the system clock
is used instead.
When playing complex media, each sound and video sample must be played in a
specific order at a specific time. For this purpose, GStreamer provides a
synchronization mechanism.
</para>
<para>
&GStreamer; derives several <emphasis>time value</emphasis> from the clock
and the playback state.
It is important to note, that a <emphasis>clock-time</emphasis> is
monotonically rising, but the value itself is not meaningful.
Subtracting the <emphasis>base-time</emphasis> yields the
<emphasis>running-time</emphasis>. It is the same as the
<emphasis>stream-time</emphasis> if one plays from start to end at original
rate. The <emphasis>stream-time</emphasis> indicates the position in the
media. The <emphasis>running-time</emphasis> is (re-)set to 0 when the
pipeline starts to play and also after <emphasis>flushing</emphasis> seeks.
&GStreamer; provides support for the following use cases:
<itemizedlist>
<listitem>
<para>
Non-live sources with access faster than playback rate. This is
the case where one is reading media from a file and playing it
back in a synchronized fashion. In this case, multiple streams need
to be synchronized, like audio, video and subtitles.
</para>
</listitem>
<listitem>
<para>
Capture and synchronized muxing/mixing of media from multiple live
sources. This is a typical use case where you record audio and
video from a microphone/camera and mux it into a file for
storage.
</para>
</listitem>
<listitem>
<para>
Streaming from (slow) network streams with buffering. This is the
typical web streaming case where you access content from a streaming
server with http.
</para>
</listitem>
<listitem>
<para>
Capture from live source and and playback to live source with
configurable latency. This is used when, for example, capture from
a camera, apply an effect and display the result. It is also used
when streaming low latency content over a network with UDP.
</para>
</listitem>
<listitem>
<para>
Simultaneous live capture and playback from prerecorded content.
This is used in audio recording cases where you play a previously
recorded audio and record new samples, the purpose is to have the
new audio perfectly in sync with the previously recorded data.
</para>
</listitem>
</itemizedlist>
</para>
<para>
&GStreamer; uses a <classname>GstClock</classname> object, buffer
timestamps and a SEGMENT event to synchronize streams in a pipeline
as we will see in the next sections.
</para>
<figure float="1" id="chapter-clock-img">
<title>&GStreamer; clock and various times</title>
<mediaobject>
<imageobject>
<imagedata scale="75" fileref="images/clocks.&image;" format="&IMAGE;" />
</imageobject>
</mediaobject>
</figure>
<sect1 id="section-clock-time-types" xreflabel="Clock running-time">
<title>Clock running-time </title>
<para>
In a typical computer, there are many sources that can be used as a
time source, e.g., the system time, soundcards, CPU performance
counters, ... For this reason, there are many
<classname>GstClock</classname> implementations available in &GStreamer;.
The clock time doesn't always start from 0 or from some known value.
Some clocks start counting from some known start date, other clocks start
counting since last reboot, etc...
</para>
<para>
A <classname>GstClock</classname> returns the
<emphasis role="strong">absolute-time</emphasis>
according to that clock with <function>gst_clock_get_time ()</function>.
The absolute-time (or clock time) of a clock is monotonically increasing.
From the absolute-time is a <emphasis role="strong">running-time</emphasis>
calculated, which is simply the difference between a previous snapshot
of the absolute-time called the <emphasis role="strong">base-time</emphasis>.
So:
</para>
<para>
running-time = absolute-time - base-time
</para>
<para>
A &GStreamer; <classname>GstPipeline</classname> object maintains a
<classname>GstClock</classname> object and a base-time when it goes
to the PLAYING state. The pipeline gives a handle to the selected
<classname>GstClock</classname> to each element in the pipeline along
with selected base-time. The pipeline will select a base-time in such
a way that the running-time reflects the total time spent in the
PLAYING state. As a result, when the pipeline is PAUSED, the
running-time stands still.
</para>
<para>
Because all objects in the pipeline have the same clock and base-time,
they can thus all calculate the running-time according to the pipeline
clock.
</para>
</sect1>
<sect1 id="section-buffer-running-time" xreflabel="Buffer running-time">
<title>Buffer running-time</title>
<para>
To calculate a buffer running-time, we need a buffer timestamp and
the SEGMENT event that preceeded the buffer. First we can convert
the SEGMENT event into a <classname>GstSegment</classname> object
and then we can use the
<function>gst_segment_to_running_time ()</function> function to
perform the calculation of the buffer running-time.
</para>
<para>
Synchronization is now a matter of making sure that a buffer with a
certain running-time is played when the clock reaches the same
running-time. Usually this task is done by sink elements. Sink also
have to take into account the latency configured in the pipeline and
add this to the buffer running-time before synchronizing to the
pipeline clock.
</para>
<para>
Non-live sources timestamp buffers with a running-time starting
from 0. After a flushing seek, they will produce buffers again
from a running-time of 0.
</para>
<para>
Live sources need to timestamp buffers with a running-time matching
the pipeline running-time when the first byte of the buffer was
captured.
</para>
</sect1>
<sect1 id="section-buffer-stream-time" xreflabel="Buffer stream-time">
<title>Buffer stream-time</title>
<para>
The buffer stream-time, also known as the position in the stream,
is calculated from the buffer timestamps and the preceeding SEGMENT
event. It represents the time inside the media as a value between
0 and the total duration of the media.
</para>
<para>
The stream-time is used in:
<itemizedlist>
<listitem>
<para>
Report the current position in the stream with the POSITION
query.
</para>
</listitem>
<listitem>
<para>
The position used in the seek events and queries.
</para>
</listitem>
<listitem>
<para>
The position used to synchronize controlled values.
</para>
</listitem>
</itemizedlist>
</para>
<para>
The stream-time is never used to synchronize streams, this is only
done with the running-time.
</para>
</sect1>
<sect1 id="section-time-overview" xreflabel="Time overview">
<title>Time overview</title>
<para>
Here is an overview of the various timelines used in &GStreamer;.
</para>
<para>
The image below represents the different times in the pipeline when
playing a 100ms sample and repeating the part between 50ms and
100ms.
</para>
<figure float="1" id="chapter-clock-img">
<title>&GStreamer; clock and various times</title>
<mediaobject>
<imageobject>
<imagedata scale="75" fileref="images/clocks.&image;" format="&IMAGE;" />
</imageobject>
</mediaobject>
</figure>
<para>
You can see how the running-time of a buffer always increments
monotonically along with the clock-time. Buffers are played when their
running-time is equal to the clock-time - base-time. The stream-time
represents the position in the stream and jumps backwards when
repeating.
</para>
</sect1>
<sect1 id="section-clocks-providers">
<title>Clock providers</title>
<para>
A clock provider is an element in the pipeline that can provide
a <classname>GstClock</classname> object. The clock object needs to
report an absoulute-time that is monotonocally increasing when the
element is in the PLAYING state. It is allowed to pause the clock
while the element is PAUSED.
</para>
<para>
Clock providers exist because they play back media at some rate, and
this rate is not necessarily the same as the system clock rate. For
example, a soundcard may playback at 44,1 kHz, but that doesn't mean
that after <emphasis>exactly</emphasis> 1 second <emphasis>according
to the system clock</emphasis>, the soundcard has played back 44.100
samples. This is only true by approximation. Therefore, generally,
pipelines with an audio output use the audiosink as clock provider.
This ensures that one second of video will be played back at the same
rate as that the soundcard plays back 1 second of audio.
samples. This is only true by approximation. In fact, the audio
device has an internal clock based on the number of samples played
that we can expose.
</para>
<para>
Whenever some part of the pipeline requires to know the current clock
time, it will be requested from the clock through
<function>gst_clock_get_time ()</function>. The clock-time does not
need to start at 0. The pipeline, which contains the global clock that
all elements in the pipeline will use, in addition has a <quote>base
time</quote>, which is the clock time at the point where the
pipeline went to the PLAYING state. Each element can subtract the
<quote>base time</quote> from the clock-time to know the current
running time.
If an element with an internal clock needs to synchronize, it needs
to estimate when a time according to the pipeline clock will take
place according to the internal clock. To estimate this, it needs
to slave its clock to the pipeline clock.
</para>
<para>
The clock provider is responsible for making sure that the clock time
always represents the current media time as closely as possible; it
has to take care of things such as playback latencies, buffering in
audio-kernel modules, and so on, since all those could affect a/v sync
and thus decrease the user experience.
If the pipeline clock is exactly the internal clock of an element,
the element can skip the slaving step and directly use the pipeline
clock to schedule playback. This can be both faster and more
accurate.
Therefore, generally, elements with an internal clock like audio
input or output devices will be a clock provider for the pipeline.
</para>
<para>
When the pipeline goes to the PLAYING state, it will go over all
elements in the pipeline from sink to source and ask each element
if they can provide a clock. The last element that can provide a
clock will be used as the clock provider in the pipeline.
This algorithm prefers a clock from an audio sink in a typical
playback pipeline and a clock from source elements in a typical
capture pipeline.
</para>
<para>
There exist some bus messages to let you know about the clock and
clock providers in the pipeline. You can see what clock is selected
in the pipeline by looking at the NEW_CLOCK message on the bus.
When a clock provider is removed from the pipeline, a CLOCK_LOST
message is posted and the application should go to PAUSED and back
to PLAYING to select a new clock.
</para>
</sect1>
<sect1 id="section-clocks-slaves">
<title>Clock slaves</title>
<sect1 id="section-clocks-latency">
<title>Latency</title>
<para>
Clock slaves get assigned a clock by their containing pipeline. Their
task is to make sure that media playback follows the time progress as
represented by this clock as closely as possible. For most elements,
that will simply mean to wait until the buffer running-time is reached
before playing back their current sample.
The latency is the time it takes for a sample captured at timestamp X
to reach the sink. This time is measured against the clock in the
pipeline. For pipelines where the only elements that synchronize against
the clock are the sinks, the latency is always 0 since no other element
is delaying the buffer.
</para>
<para>
The buffer running-time is derived from the buffer timestamp and the
newsegment event preceeding the buffer. A buffer is played synchronized
with the clock when the clock's running-time has reached exactly the
buffer running-time; this can be done with the function
<function>gst_clock_id_wait ()</function>.
</para>
<para>
For more information on how to write elements that conform to this
required behaviour, see the Plugin Writer's Guide.
For pipelines with live sources, a latency is introduced, mostly because
of the way a live source works. Consider an audio source, it will start
capturing the first sample at time 0. If the source pushes buffers with
44100 samples at a time at 44100Hz it will have collected the buffer at
second 1. Since the timestamp of the buffer is 0 and the time of the
clock is now >= 1 second, the sink will drop this buffer because it is
too late. Without any latency compensation in the sink, all buffers will
be dropped.
</para>
<sect2 id="section-latency-compensation">
<title>Latency compensation</title>
<para>
Before the pipeline goes to the PLAYING state, it will, in addition to
selecting a clock and calculating a base-time, calculate the latency
in the pipeline. It does this by doing a LATENCY query on all the sinks
in the pipeline. The pipeline then selects the maximum latency in the
pipeline and configures this with a LATENCY event.
</para>
<para>
All sink elements will delay playback by the value in the LATENCY event.
Since all sinks delay with the same amount of time, they will be
relative in sync.
</para>
</sect2>
<sect2 id="section-latency-dynamic">
<title>Dynamic Latency</title>
<para>
Adding/removing elements to/from a pipeline or changing element
properties can change the latency in a pipeline. An element can
request a latency change in the pipeline by posting a LATENCY
message on the bus. The application can then decide to query and
redistribute a new latency or not. Changing the latency in a
pipeline might cause visual or audible glitches and should
therefore only be done by the application when it is allowed.
</para>
</sect2>
</sect1>
</chapter>

30
docs/pwg/advanced-clock.xml
View file

@ -73,7 +73,10 @@
<para>
Synchronization is now a matter of making sure that a buffer with a
certain running-time is played when the clock reaches the same
running-time. Usually this task is done by sink elements.
running-time. Usually this task is done by sink elements. Sink also
have to take into account the latency configured in the pipeline and
add this to the buffer running-time before synchronizing to the
pipeline clock.
</para>
</sect1>
@ -120,11 +123,11 @@
</sect2>
<sect2>
<title>Parser elements </title>
<title>Parser/Decoder/Encoder elements </title>
<para>
Parser elements must use the incomming timestamps and transfer those
to the resulting output buffers. They are allowed to interpolate or
reconstruct timestamps on missing input buffers when they can.
Parser/Decoder elements must use the incomming timestamps and transfer
those to the resulting output buffers. They are allowed to interpolate
or reconstruct timestamps on missing input buffers when they can.
</para>
</sect2>
@ -139,8 +142,18 @@
buffer timestamps.
</para>
</sect2>
<sect2>
<title>Muxer elements</title>
<para>
Muxer elements should use the incomming buffer running-time to mux the
different streams together. They should copy the incomming running-time
to the outgoing buffers.
</para>
</sect2>
<sect2> <title> Sink elements </title>
<sect2>
<title>Sink elements</title>
<para>
If the element is intended to emit samples at a specific time (real time
playing), the element should require a clock, and thus implement the
@ -148,8 +161,9 @@
</para>
<para>
The sink should then make sure that the sample with running-time is played
exactly when the pipeline clock reaches that running-time. Some elements
might use the clock API such as <function>gst_clock_id_wait()</function>
exactly when the pipeline clock reaches that running-time + latency.
Some elements might use the clock API such as
<function>gst_clock_id_wait()</function>
to perform this action. Other sinks might need to use other means of
scheduling timely playback of the data.
</para>