mirror of
https://gitlab.freedesktop.org/gstreamer/gstreamer.git
synced 2024-11-13 12:51:16 +00:00
272 lines
8.7 KiB
Markdown
272 lines
8.7 KiB
Markdown
|
# Synchronisation
|
|||
|
|
|||
|
This document outlines the techniques used for doing synchronised
|
|||
|
playback of multiple streams.
|
|||
|
|
|||
|
Synchronisation in a GstPipeline is achieved using the following 3
|
|||
|
components:
|
|||
|
|
|||
|
- a GstClock, which is global for all elements in a GstPipeline.
|
|||
|
|
|||
|
- Timestamps on a GstBuffer.
|
|||
|
|
|||
|
- the SEGMENT event preceding the buffers.
|
|||
|
|
|||
|
## A GstClock
|
|||
|
|
|||
|
This object provides a counter that represents the current time in
|
|||
|
nanoseconds. This value is called the absolute\_time.
|
|||
|
|
|||
|
Different sources exist for this counter:
|
|||
|
|
|||
|
- the system time (with g\_get\_current\_time() and with microsecond
|
|||
|
accuracy)
|
|||
|
|
|||
|
- monotonic time (with g\_get\_monotonic\_time () with microsecond
|
|||
|
accuracy)
|
|||
|
|
|||
|
- an audio device (based on number of samples played)
|
|||
|
|
|||
|
- a network source based on packets received + timestamps in those
|
|||
|
packets (a typical example is an RTP source)
|
|||
|
|
|||
|
- …
|
|||
|
|
|||
|
In GStreamer any element can provide a GstClock object that can be used
|
|||
|
in the pipeline. The GstPipeline object will select a clock from all the
|
|||
|
providers and will distribute it to all other elements (see
|
|||
|
[gstpipeline](design/gstpipeline.md)).
|
|||
|
|
|||
|
A GstClock always counts time upwards and does not necessarily start at
|
|||
|
0.
|
|||
|
|
|||
|
While it is possible, it is not recommended to create a clock derived
|
|||
|
from the contents of a stream (for example, create a clock from the PCR
|
|||
|
in an mpeg-ts stream).
|
|||
|
|
|||
|
## Running time
|
|||
|
|
|||
|
After a pipeline selected a clock it will maintain the running\_time
|
|||
|
based on the selected clock. This running\_time represents the total
|
|||
|
time spent in the PLAYING state and is calculated as follows:
|
|||
|
|
|||
|
- If the pipeline is NULL/READY, the running\_time is undefined.
|
|||
|
|
|||
|
- In PAUSED, the running\_time remains at the time when it was last
|
|||
|
PAUSED. When the stream is PAUSED for the first time, the
|
|||
|
running\_time is 0.
|
|||
|
|
|||
|
- In PLAYING, the running\_time is the delta between the
|
|||
|
absolute\_time and the base time. The base time is defined as the
|
|||
|
absolute\_time minus the running\_time at the time when the pipeline
|
|||
|
is set to PLAYING.
|
|||
|
|
|||
|
- after a flushing seek, the running\_time is set to 0 (see
|
|||
|
[seeking](design/seeking.md)). This is accomplished by redistributing a new
|
|||
|
base\_time to the elements that got flushed.
|
|||
|
|
|||
|
This algorithm captures the running\_time when the pipeline is set from
|
|||
|
PLAYING to PAUSED and restores this time based on the current
|
|||
|
absolute\_time when going back to PLAYING. This allows for both clocks
|
|||
|
that progress when in the PAUSED state (systemclock) and clocks that
|
|||
|
don’t (audioclock).
|
|||
|
|
|||
|
The clock and pipeline now provide a running\_time to all elements that
|
|||
|
want to perform synchronisation. Indeed, the running time can be
|
|||
|
observed in each element (during the PLAYING state) as:
|
|||
|
|
|||
|
```
|
|||
|
C.running_time = absolute_time - base_time
|
|||
|
```
|
|||
|
|
|||
|
We note C.running\_time as the running\_time obtained by looking at the
|
|||
|
clock. This value is monotonically increasing at the rate of the clock.
|
|||
|
|
|||
|
## Timestamps
|
|||
|
|
|||
|
The GstBuffer timestamps and the preceding SEGMENT event (See
|
|||
|
[streams](design/streams.md)) define a transformation of the buffer timestamps to
|
|||
|
running\_time as follows:
|
|||
|
|
|||
|
The following notation is used:
|
|||
|
|
|||
|
**B**: GstBuffer
|
|||
|
- B.timestamp = buffer timestamp (GST_BUFFER_PTS or GST_BUFFER_DTS)
|
|||
|
|
|||
|
**S**: SEGMENT event preceding the buffers.
|
|||
|
- S.start: start field in the SEGMENT event. This is the lowest allowed
|
|||
|
timestamp.
|
|||
|
- S.stop: stop field in the SEGMENT event. This is the highers allowed
|
|||
|
timestamp.
|
|||
|
- S.rate: rate field of SEGMENT event. This is the playback rate.
|
|||
|
- S.base: a base time for the time. This is the total elapsed running_time of any
|
|||
|
previous segments.
|
|||
|
- S.offset: an offset to apply to S.start or S.stop. This is the amount that
|
|||
|
has already been elapsed in the segment.
|
|||
|
|
|||
|
Valid buffers for synchronisation are those with B.timestamp between
|
|||
|
S.start and S.stop (after applying the S.offset). All other buffers
|
|||
|
outside this range should be dropped or clipped to these boundaries (see
|
|||
|
also [segments](design/segments.md)).
|
|||
|
|
|||
|
The following transformation to running_time exist:
|
|||
|
|
|||
|
```
|
|||
|
if (S.rate > 0.0)
|
|||
|
B.running_time = (B.timestamp - (S.start + S.offset)) / ABS (S.rate) + S.base
|
|||
|
=>
|
|||
|
B.timestamp = (B.running_time - S.base) * ABS (S.rate) + S.start + S.offset
|
|||
|
else
|
|||
|
B.running_time = ((S.stop - S.offset) - B.timestamp) / ABS (S.rate) + S.base
|
|||
|
=>
|
|||
|
B.timestamp = S.stop - S.offset - ((B.running_time - S.base) * ABS (S.rate))
|
|||
|
```
|
|||
|
|
|||
|
We write B.running_time as the running_time obtained from the SEGMENT
|
|||
|
event and the buffers of that segment.
|
|||
|
|
|||
|
The first displayable buffer will yield a value of 0 (since B.timestamp
|
|||
|
== S.start and S.offset and S.base == 0).
|
|||
|
|
|||
|
For S.rate \> 1.0, the timestamps will be scaled down to increase the
|
|||
|
playback rate. Likewise, a rate between 0.0 and 1.0 will slow down
|
|||
|
playback.
|
|||
|
|
|||
|
For negative rates, timestamps are received stop S.stop to S.start so
|
|||
|
that the first buffer received will be transformed into B.running\_time
|
|||
|
of 0 (B.timestamp == S.stop and S.base == 0).
|
|||
|
|
|||
|
This makes it so that B.running\_time is always monotonically increasing
|
|||
|
starting from 0 with both positive and negative rates.
|
|||
|
|
|||
|
## Synchronisation
|
|||
|
|
|||
|
As we have seen, we can get a running\_time:
|
|||
|
|
|||
|
- using the clock and the element’s base\_time with:
|
|||
|
|
|||
|
```
|
|||
|
C.running_time = absolute_time - base_time
|
|||
|
```
|
|||
|
|
|||
|
- using the buffer timestamp and the preceding SEGMENT event as (assuming
|
|||
|
positive playback rate):
|
|||
|
|
|||
|
```
|
|||
|
B.running_time = (B.timestamp - (S.start + S.offset)) / ABS (S.rate) + S.base
|
|||
|
```
|
|||
|
|
|||
|
We prefix C. and B. before the two running times to note how they were
|
|||
|
calculated.
|
|||
|
|
|||
|
The task of synchronized playback is to make sure that we play a buffer
|
|||
|
with B.running\_time at the moment when the clock reaches the same
|
|||
|
C.running\_time.
|
|||
|
|
|||
|
Thus the following must hold:
|
|||
|
|
|||
|
```
|
|||
|
B.running_time = C.running_time
|
|||
|
```
|
|||
|
|
|||
|
expaning:
|
|||
|
|
|||
|
```
|
|||
|
B.running_time = absolute_time - base_time
|
|||
|
```
|
|||
|
|
|||
|
or:
|
|||
|
|
|||
|
```
|
|||
|
absolute_time = B.running_time + base_time
|
|||
|
```
|
|||
|
|
|||
|
The absolute\_time when a buffer with B.running\_time should be played
|
|||
|
is noted with B.sync\_time. Thus:
|
|||
|
|
|||
|
```
|
|||
|
B.sync_time = B.running_time + base_time
|
|||
|
```
|
|||
|
|
|||
|
One then waits for the clock to reach B.sync\_time before rendering the
|
|||
|
buffer in the sink (See also [clocks](design/clocks.md)).
|
|||
|
|
|||
|
For multiple streams this means that buffers with the same running\_time
|
|||
|
are to be displayed at the same time.
|
|||
|
|
|||
|
A demuxer must make sure that the SEGMENT it emits on its output pads
|
|||
|
yield the same running\_time for buffers that should be played
|
|||
|
synchronized. This usually means sending the same SEGMENT on all pads
|
|||
|
and making sure that the synchronized buffers have the same timestamps.
|
|||
|
|
|||
|
## Stream time
|
|||
|
|
|||
|
The stream time is also known as the position in the stream and is a
|
|||
|
value between 0 and the total duration of the media file.
|
|||
|
|
|||
|
It is the stream time that is used for:
|
|||
|
|
|||
|
- report the POSITION query in the pipeline
|
|||
|
|
|||
|
- the position used in seek events/queries
|
|||
|
|
|||
|
- the position used to synchronize controller values
|
|||
|
|
|||
|
Additional fields in the SEGMENT are used:
|
|||
|
|
|||
|
- S.time: time field in the SEGMENT event. This the stream-time of
|
|||
|
S.start
|
|||
|
|
|||
|
- S.applied\_rate: The rate already applied to the segment.
|
|||
|
|
|||
|
Stream time is calculated using the buffer times and the preceding
|
|||
|
SEGMENT event as follows:
|
|||
|
|
|||
|
```
|
|||
|
stream_time = (B.timestamp - S.start) * ABS (S.applied_rate) + S.time
|
|||
|
=> B.timestamp = (stream_time - S.time) / ABS(S.applied_rate) + S.start
|
|||
|
```
|
|||
|
|
|||
|
For negative rates, B.timestamp will go backwards from S.stop to
|
|||
|
S.start, making the stream time go backwards:
|
|||
|
|
|||
|
```
|
|||
|
stream_time = (S.stop - B.timestamp) * ABS(S.applied_rate) + S.time
|
|||
|
=> B.timestamp = S.stop - (stream_time - S.time) / ABS(S.applied_rate)
|
|||
|
```
|
|||
|
|
|||
|
In the PLAYING state, it is also possible to use the pipeline clock to
|
|||
|
derive the current stream\_time.
|
|||
|
|
|||
|
Give the two formulas above to match the clock times with buffer
|
|||
|
timestamps allows us to rewrite the above formula for stream\_time (and
|
|||
|
for positive rates).
|
|||
|
|
|||
|
```
|
|||
|
C.running_time = absolute_time - base_time
|
|||
|
B.running_time = (B.timestamp - (S.start + S.offset)) / ABS (S.rate) + S.base
|
|||
|
|
|||
|
=>
|
|||
|
(B.timestamp - (S.start + S.offset)) / ABS (S.rate) + S.base = absolute_time - base_time;
|
|||
|
|
|||
|
=>
|
|||
|
(B.timestamp - (S.start + S.offset)) / ABS (S.rate) = absolute_time - base_time - S.base;
|
|||
|
|
|||
|
=>
|
|||
|
(B.timestamp - (S.start + S.offset)) = (absolute_time - base_time - S.base) * ABS (S.rate)
|
|||
|
|
|||
|
=>
|
|||
|
(B.timestamp - S.start) = S.offset + (absolute_time - base_time - S.base) * ABS (S.rate)
|
|||
|
|
|||
|
filling (B.timestamp - S.start) in the above formule for stream time
|
|||
|
|
|||
|
=>
|
|||
|
stream_time = (S.offset + (absolute_time - base_time - S.base) * ABS (S.rate)) * ABS (S.applied_rate) + S.time
|
|||
|
```
|
|||
|
|
|||
|
This last formula is typically used in sinks to report the current
|
|||
|
position in an accurate and efficient way.
|
|||
|
|
|||
|
Note that the stream time is never used for synchronisation against the
|
|||
|
clock.
|