mirror of
https://gitlab.freedesktop.org/gstreamer/gstreamer.git
synced 2024-11-27 12:11:13 +00:00
407 lines
16 KiB
Text
407 lines
16 KiB
Text
Latency
|
|
-------
|
|
|
|
The latency is the time it takes for a sample captured at timestamp 0 to reach the
|
|
sink. This time is measured against the clock in the pipeline. For pipelines
|
|
where the only elements that synchronize against the clock are the sinks, the
|
|
latency is always 0 since no other element is delaying the buffer.
|
|
|
|
For pipelines with live sources, a latency is introduced, mostly because of the
|
|
way a live source works. Consider an audio source, it will start capturing the
|
|
first sample at time 0. If the source pushes buffers with 44100 samples at a
|
|
time at 44100Hz it will have collected the buffer at second 1.
|
|
Since the timestamp of the buffer is 0 and the time of the clock is now >= 1
|
|
second, the sink will drop this buffer because it is too late.
|
|
Without any latency compensation in the sink, all buffers will be dropped.
|
|
|
|
The situation becomes more complex in the presence of:
|
|
|
|
- 2 live sources connected to 2 live sinks with different latencies
|
|
* audio/video capture with synchronized live preview.
|
|
* added latencies due to effects (delays, resamplers...)
|
|
- 1 live source connected to 2 live sinks
|
|
* firewire DV
|
|
* RTP, with added latencies because of jitter buffers.
|
|
- mixed live source and non-live source scenarios.
|
|
* synchronized audio capture with non-live playback. (overdubs,..)
|
|
- clock slaving in the sinks due to the live sources providing their own
|
|
clocks.
|
|
|
|
To perform the needed latency corrections in the above scenarios, we must
|
|
develop an algorithm to calculate a global latency for the pipeline. The
|
|
algorithm must be extensible so that it can optimize the latency at runtime.
|
|
It must also be possible to disable or tune the algorithm based on specific
|
|
application needs (required minimal latency).
|
|
|
|
|
|
Pipelines without latency compensation
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
We show some examples to demonstrate the problem of latency in typical
|
|
capture pipelines.
|
|
|
|
- Example 1
|
|
|
|
An audio capture/playback pipeline.
|
|
|
|
asrc: audio source, provides a clock
|
|
asink audio sink, provides a clock
|
|
|
|
.--------------------------.
|
|
| pipeline |
|
|
| .------. .-------. |
|
|
| | asrc | | asink | |
|
|
| | src -> sink | |
|
|
| '------' '-------' |
|
|
'--------------------------'
|
|
|
|
NULL->READY:
|
|
asink: NULL->READY: probes device, returns SUCCESS
|
|
asrc: NULL->READY: probes device, returns SUCCESS
|
|
|
|
READY->PAUSED:
|
|
asink: READY:->PAUSED open device, returns ASYNC
|
|
asrc: READY->PAUSED: open device, returns NO_PREROLL
|
|
|
|
* Since the source is a live source, it will only produce data in the
|
|
PLAYING state. To note this fact, it returns NO_PREROLL from the state change
|
|
function.
|
|
* This sink returns ASYNC because it can only complete the state change to
|
|
PAUSED when it receives the first buffer.
|
|
|
|
At this point the pipeline is not processing data and the clock is not
|
|
running. Unless a new action is performed on the pipeline, this situation will
|
|
never change.
|
|
|
|
PAUSED->PLAYING:
|
|
asrc clock selected because it is the most upstream clock provider. asink can
|
|
only provide a clock when it received the first buffer and configured the
|
|
device with the samplerate in the caps.
|
|
|
|
asink: PAUSED:->PLAYING, sets pending state to PLAYING, returns ASYNC becaus
|
|
it is not prerolled. The sink will commit state to
|
|
PLAYING when it prerolls.
|
|
asrc: PAUSED->PLAYING: starts pushing buffers.
|
|
|
|
* since the sink is still performing a state change from READY -> PAUSED, it
|
|
remains ASYNC. The pending state will be set to PLAYING.
|
|
* The clock starts running as soon as all the elements have been set to
|
|
PLAYING.
|
|
* the source is a live source with a latency. Since it is synchronized with
|
|
the clock, it will produce a buffer with timestamp 0 and duration D after
|
|
time D, ie. it will only be able to produce the last sample of the buffer
|
|
(with timestamp D) at time D. This latency depends on the size of the
|
|
buffer.
|
|
* the sink will receive the buffer with timestamp 0 at time >= D. At this
|
|
point the buffer is too late already and might be dropped. This state of
|
|
constantly dropping data will not change unless a constant latency
|
|
correction is added to the incoming buffer timestamps.
|
|
|
|
The problem is due to the fact that the sink is set to (pending) PLAYING
|
|
without being prerolled, which only happens in live pipelines.
|
|
|
|
- Example 2
|
|
|
|
An audio/video capture/playback pipeline. We capture both audio and video and
|
|
have them played back synchronized again.
|
|
|
|
asrc: audio source, provides a clock
|
|
asink audio sink, provides a clock
|
|
vsrc: video source
|
|
vsink video sink
|
|
|
|
.--------------------------.
|
|
| pipeline |
|
|
| .------. .-------. |
|
|
| | asrc | | asink | |
|
|
| | src -> sink | |
|
|
| '------' '-------' |
|
|
| .------. .-------. |
|
|
| | vsrc | | vsink | |
|
|
| | src -> sink | |
|
|
| '------' '-------' |
|
|
'--------------------------'
|
|
|
|
The state changes happen in the same way as example 1. Both sinks end up with
|
|
pending state of PLAYING and a return value of ASYNC until they receive the
|
|
first buffer.
|
|
|
|
For audio and video to be played in sync, both sinks must compensate for the
|
|
latency of its source but must also use exactly the same latency correction.
|
|
|
|
Suppose asrc has a latency of 20ms and vsrc a latency of 33ms, the total
|
|
latency in the pipeline has to be at least 33ms. This also means that the
|
|
pipeline must have at least a 33 - 20 = 13ms buffering on the audio stream or
|
|
else the audio src will underrun while the audiosink waits for the previous
|
|
sample to play.
|
|
|
|
- Example 3
|
|
|
|
An example of the combination of a non-live (file) and a live source (vsrc)
|
|
connected to live sinks (vsink, sink).
|
|
|
|
.--------------------------.
|
|
| pipeline |
|
|
| .------. .-------. |
|
|
| | file | | sink | |
|
|
| | src -> sink | |
|
|
| '------' '-------' |
|
|
| .------. .-------. |
|
|
| | vsrc | | vsink | |
|
|
| | src -> sink | |
|
|
| '------' '-------' |
|
|
'--------------------------'
|
|
|
|
The state changes happen in the same way as example 1. Except sink will be
|
|
able to preroll (commit its state to PAUSED).
|
|
|
|
In this case sink will have no latency but vsink will. The total latency
|
|
should be that of vsink.
|
|
|
|
Note that because of the presence of a live source (vsrc), the pipeline can be
|
|
set to playing before sink is able to preroll. Without compensation for the
|
|
live source, this might lead to synchronisation problems because the latency
|
|
should be configured in the element before it can go to PLAYING.
|
|
|
|
|
|
- Example 4
|
|
|
|
An example of the combination of a non-live and a live source. The non-live
|
|
source is connected to a live sink and the live source to a non-live sink.
|
|
|
|
.--------------------------.
|
|
| pipeline |
|
|
| .------. .-------. |
|
|
| | file | | sink | |
|
|
| | src -> sink | |
|
|
| '------' '-------' |
|
|
| .------. .-------. |
|
|
| | vsrc | | files | |
|
|
| | src -> sink | |
|
|
| '------' '-------' |
|
|
'--------------------------'
|
|
|
|
The state changes happen in the same way as example 3. Sink will be
|
|
able to preroll (commit its state to PAUSED). files will not be able to
|
|
preroll.
|
|
|
|
sink will have no latency since it is not connected to a live source. files
|
|
does not do synchronisation so it does not care about latency.
|
|
|
|
The total latency in the pipeline is 0. The vsrc captures in sync with the
|
|
playback in sink.
|
|
|
|
As in example 3, sink can only be set to PLAYING after it successfully
|
|
prerolled.
|
|
|
|
|
|
State Changes
|
|
~~~~~~~~~~~~~
|
|
|
|
A Sink is never set to PLAYING before it is prerolled. In order to do this, the
|
|
pipeline (at the GstBin level) keeps track of all
|
|
elements that require preroll (the ones that return ASYNC from the state
|
|
change). These elements posted a ASYNC_START message without a matching
|
|
ASYNC_DONE message.
|
|
|
|
The pipeline will not change the state of the elements that are still doing an
|
|
ASYNC state change.
|
|
|
|
When an ASYNC element prerolls, it commits its state to PAUSED and posts an
|
|
ASYNC_DONE message. The pipeline notices this ASYNC_DONE message and matches it
|
|
with the ASYNC_START message it cached for the corresponding element.
|
|
|
|
When all ASYNC_START messages are matched with an ASYNC_DONE message, the
|
|
pipeline proceeds with setting the elements to the final state again.
|
|
|
|
The base time of the element was already set by the pipeline when it changed the
|
|
NO_PREROLL element to PLAYING. This operation has to be performed in the
|
|
separate async state change thread (like the one currently used for going from
|
|
PAUSED->PLAYING in a non-live pipeline).
|
|
|
|
|
|
Query
|
|
~~~~~
|
|
|
|
The pipeline latency is queried with the LATENCY query.
|
|
|
|
(out) "live", G_TYPE_BOOLEAN (default FALSE)
|
|
- if a live element is found upstream
|
|
|
|
(out) "min-latency", G_TYPE_UINT64 (default 0, must not be NONE)
|
|
- the minimum latency in the pipeline, meaning the minimum time
|
|
downstream elements synchronizing to the clock have to wait until
|
|
they can be sure that all data for the current running time has
|
|
been received.
|
|
|
|
Elements answering the latency query and introducing latency must
|
|
set this to the maximum time for which they will delay data, while
|
|
considering upstream's minimum latency. As such, from an element's
|
|
perspective this is *not* its own minimum latency but its own
|
|
maximum latency.
|
|
Considering upstream's minimum latency in general means that the
|
|
element's own value is added to upstream's value, as this will give
|
|
the overall minimum latency of all elements from the source to the
|
|
current element:
|
|
|
|
min_latency = upstream_min_latency + own_min_latency
|
|
|
|
(out) "max-latency", G_TYPE_UINT64 (default 0, NONE meaning infinity)
|
|
- the maximum latency in the pipeline, meaning the maximum time an
|
|
element synchronizing to the clock is allowed to wait for receiving
|
|
all data for the current running time. Waiting for a longer time
|
|
will result in data loss, overruns and underruns of buffers and in
|
|
general breaks synchronized data flow in the pipeline.
|
|
|
|
Elements answering the latency query should set this to the maximum
|
|
time for which they can buffer upstream data without blocking or
|
|
dropping further data. For an element this value will generally be
|
|
its own minimum latency, but might be bigger than that if it can
|
|
buffer more data. As such, queue elements can be used to increase
|
|
the maximum latency.
|
|
|
|
The value set in the query should again consider upstream's maximum
|
|
latency:
|
|
- If the current element has blocking buffering, i.e. it does
|
|
not drop data by itself when its internal buffer is full, it should
|
|
just add its own maximum latency (i.e. the size of its internal
|
|
buffer) to upstream's value. If upstream's maximum latency, or the
|
|
elements internal maximum latency was NONE (i.e. infinity), it will
|
|
be set to infinity.
|
|
|
|
if (upstream_max_latency == NONE || own_max_latency == NONE)
|
|
max_latency = NONE;
|
|
else
|
|
max_latency = upstream_max_latency + own_max_latency
|
|
|
|
If the element has multiple sinkpads, the minimum upstream latency is
|
|
the maximum of all live upstream minimum latencies.
|
|
|
|
- If the current element has leaky buffering, i.e. it drops data by
|
|
itself when its internal buffer is full, it should take the minimum
|
|
of its own maximum latency and upstream's. Examples for such
|
|
elements are audio sinks and sources with an internal ringbuffer,
|
|
leaky queues and in general live sources with a limited amount of
|
|
internal buffers that can be used.
|
|
|
|
max_latency = MIN (upstream_max_latency, own_max_latency)
|
|
|
|
Note: many GStreamer base classes allow subclasses to set a
|
|
minimum and maximum latency and handle the query themselves. These
|
|
base classes assume non-leaky (i.e. blocking) buffering for the
|
|
maximum latency. The base class' default query handler needs to be
|
|
overridden to correctly handle leaky buffering.
|
|
|
|
If the element has multiple sinkpads, the maximum upstream latency is
|
|
the minimum of all live upstream maximum latencies.
|
|
|
|
Event
|
|
~~~~~
|
|
|
|
The latency in the pipeline is configured with the LATENCY event, which contains
|
|
the following fields:
|
|
|
|
"latency", G_TYPE_UINT64
|
|
- the configured latency in the pipeline
|
|
|
|
|
|
Latency compensation
|
|
~~~~~~~~~~~~~~~~~~~~
|
|
|
|
Latency calculation and compensation is performed before the pipeline proceeds to
|
|
the PLAYING state.
|
|
|
|
When the pipeline collected all ASYNC_DONE messages it can calculate the global
|
|
latency as follows:
|
|
|
|
- perform a latency query on all sinks
|
|
- sources set their minimum and maximum latency
|
|
- other elements add their own values as described above
|
|
- latency = MAX (all min latencies)
|
|
- if MIN (all max latencies) < latency we have an impossible situation and we
|
|
must generate an error indicating that this pipeline cannot be played. This
|
|
usually means that there is not enough buffering in some chain of the
|
|
pipeline. A queue can be added to those chains.
|
|
|
|
The sinks gather this information with a LATENCY query upstream. Intermediate
|
|
elements pass the query upstream and add the amount of latency they add to the
|
|
result.
|
|
|
|
ex1:
|
|
sink1: [20 - 20]
|
|
sink2: [33 - 40]
|
|
|
|
MAX (20, 33) = 33
|
|
MIN (20, 40) = 20 < 33 -> impossible
|
|
|
|
ex2:
|
|
sink1: [20 - 50]
|
|
sink2: [33 - 40]
|
|
|
|
MAX (20, 33) = 33
|
|
MIN (50, 40) = 40 >= 33 -> latency = 33
|
|
|
|
The latency is set on the pipeline by sending a LATENCY event to the sinks
|
|
in the pipeline. This event configures the total latency on the sinks. The
|
|
sink forwards this LATENCY event upstream so that intermediate elements can
|
|
configure themselves as well.
|
|
|
|
After this step, the pipeline continues setting the pending state on its
|
|
elements.
|
|
|
|
A sink adds the latency value, received in the LATENCY event, to
|
|
the times used for synchronizing against the clock. This will effectively
|
|
delay the rendering of the buffer with the required latency. Since this delay is
|
|
the same for all sinks, all sinks will render data relatively synchronised.
|
|
|
|
|
|
Flushing a playing pipeline
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
We can implement resynchronisation after an uncontrolled FLUSH in (part of) a
|
|
pipeline in the same way. Indeed, when a flush is performed on
|
|
a PLAYING live element, a new base time must be distributed to this element.
|
|
|
|
A flush in a pipeline can happen in the following cases:
|
|
|
|
- flushing seek in the pipeline
|
|
- performed by the application on the pipeline
|
|
- performed by the application on an element
|
|
- flush preformed by an element
|
|
- after receiving a navigation event (DVD, ...)
|
|
|
|
When a playing sink is flushed by a FLUSH_START event, an ASYNC_START message is
|
|
posted by the element. As part of the message, the fact that the element got
|
|
flushed is included. The element also goes to a pending PAUSED state and has to
|
|
be set to the PLAYING state again later.
|
|
|
|
The ASYNC_START message is kept by the parent bin. When the element prerolls,
|
|
it posts an ASYNC_DONE message.
|
|
|
|
When all ASYNC_START messages are matched with an ASYNC_DONE message, the bin
|
|
will capture a new base_time from the clock and will bring all the sinks back to
|
|
PLAYING after setting the new base time on them. It's also possible
|
|
to perform additional latency calculations and adjustments before doing this.
|
|
|
|
|
|
Dynamically adjusting latency
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
An element that want to change the latency in the pipeline can do this by
|
|
posting a LATENCY message on the bus. This message instructs the pipeline to:
|
|
|
|
- query the latency in the pipeline (which might now have changed) with a
|
|
LATENCY query.
|
|
- redistribute a new global latency to all elements with a LATENCY event.
|
|
|
|
A use case where the latency in a pipeline can change could be a network element
|
|
that observes an increased inter packet arrival jitter or excessive packet loss
|
|
and decides to increase its internal buffering (and thus the latency). The
|
|
element must post a LATENCY message and perform the additional latency
|
|
adjustments when it receives the LATENCY event from the downstream peer element.
|
|
|
|
In a similar way can the latency be decreased when network conditions are
|
|
improving again.
|
|
|
|
Latency adjustments will introduce glitches in playback in the sinks and must
|
|
only be performed in special conditions.
|