From d36377ba721dfc3a487d1400e2a4f14798e63cb7 Mon Sep 17 00:00:00 2001 From: Wim Taymans Date: Mon, 8 Oct 2012 13:22:30 +0200 Subject: [PATCH] docs: improve clock chapter --- docs/manual/advanced-clocks.xml | 328 ++++++++++++++++++++++++++------ docs/pwg/advanced-clock.xml | 30 ++- 2 files changed, 288 insertions(+), 70 deletions(-) diff --git a/docs/manual/advanced-clocks.xml b/docs/manual/advanced-clocks.xml index c23cf3cc6e..d6c8007123 100644 --- a/docs/manual/advanced-clocks.xml +++ b/docs/manual/advanced-clocks.xml @@ -1,91 +1,295 @@ - Clocks in GStreamer + Clocks and synchronization in &GStreamer; - To maintain sync in pipeline playback (which is the only case where this - really matters), &GStreamer; uses clocks. Clocks - are exposed by some elements, whereas other elements are merely clock - slaves. The primary task of a clock is to represent the time progress - according to the element exposing the clock, based on its own playback - rate. If no clock provider is available in a pipeline, the system clock - is used instead. + When playing complex media, each sound and video sample must be played in a + specific order at a specific time. For this purpose, GStreamer provides a + synchronization mechanism. - - &GStreamer; derives several time value from the clock - and the playback state. - It is important to note, that a clock-time is - monotonically rising, but the value itself is not meaningful. - Subtracting the base-time yields the - running-time. It is the same as the - stream-time if one plays from start to end at original - rate. The stream-time indicates the position in the - media. The running-time is (re-)set to 0 when the - pipeline starts to play and also after flushing seeks. + &GStreamer; provides support for the following use cases: + + + + Non-live sources with access faster than playback rate. This is + the case where one is reading media from a file and playing it + back in a synchronized fashion. In this case, multiple streams need + to be synchronized, like audio, video and subtitles. + + + + + Capture and synchronized muxing/mixing of media from multiple live + sources. This is a typical use case where you record audio and + video from a microphone/camera and mux it into a file for + storage. + + + + + Streaming from (slow) network streams with buffering. This is the + typical web streaming case where you access content from a streaming + server with http. + + + + + Capture from live source and and playback to live source with + configurable latency. This is used when, for example, capture from + a camera, apply an effect and display the result. It is also used + when streaming low latency content over a network with UDP. + + + + + Simultaneous live capture and playback from prerecorded content. + This is used in audio recording cases where you play a previously + recorded audio and record new samples, the purpose is to have the + new audio perfectly in sync with the previously recorded data. + + + + + + &GStreamer; uses a GstClock object, buffer + timestamps and a SEGMENT event to synchronize streams in a pipeline + as we will see in the next sections. -
- &GStreamer; clock and various times - - - - - -
+ + Clock running-time + + In a typical computer, there are many sources that can be used as a + time source, e.g., the system time, soundcards, CPU performance + counters, ... For this reason, there are many + GstClock implementations available in &GStreamer;. + The clock time doesn't always start from 0 or from some known value. + Some clocks start counting from some known start date, other clocks start + counting since last reboot, etc... + + + A GstClock returns the + absolute-time + according to that clock with gst_clock_get_time (). + The absolute-time (or clock time) of a clock is monotonically increasing. + From the absolute-time is a running-time + calculated, which is simply the difference between a previous snapshot + of the absolute-time called the base-time. + So: + + + running-time = absolute-time - base-time + + + A &GStreamer; GstPipeline object maintains a + GstClock object and a base-time when it goes + to the PLAYING state. The pipeline gives a handle to the selected + GstClock to each element in the pipeline along + with selected base-time. The pipeline will select a base-time in such + a way that the running-time reflects the total time spent in the + PLAYING state. As a result, when the pipeline is PAUSED, the + running-time stands still. + + + Because all objects in the pipeline have the same clock and base-time, + they can thus all calculate the running-time according to the pipeline + clock. + + + + + Buffer running-time + + To calculate a buffer running-time, we need a buffer timestamp and + the SEGMENT event that preceeded the buffer. First we can convert + the SEGMENT event into a GstSegment object + and then we can use the + gst_segment_to_running_time () function to + perform the calculation of the buffer running-time. + + + Synchronization is now a matter of making sure that a buffer with a + certain running-time is played when the clock reaches the same + running-time. Usually this task is done by sink elements. Sink also + have to take into account the latency configured in the pipeline and + add this to the buffer running-time before synchronizing to the + pipeline clock. + + + Non-live sources timestamp buffers with a running-time starting + from 0. After a flushing seek, they will produce buffers again + from a running-time of 0. + + + Live sources need to timestamp buffers with a running-time matching + the pipeline running-time when the first byte of the buffer was + captured. + + + + + Buffer stream-time + + The buffer stream-time, also known as the position in the stream, + is calculated from the buffer timestamps and the preceeding SEGMENT + event. It represents the time inside the media as a value between + 0 and the total duration of the media. + + + The stream-time is used in: + + + + Report the current position in the stream with the POSITION + query. + + + + + The position used in the seek events and queries. + + + + + The position used to synchronize controlled values. + + + + + + The stream-time is never used to synchronize streams, this is only + done with the running-time. + + + + + Time overview + + Here is an overview of the various timelines used in &GStreamer;. + + + The image below represents the different times in the pipeline when + playing a 100ms sample and repeating the part between 50ms and + 100ms. + + +
+ &GStreamer; clock and various times + + + + + +
+ + + You can see how the running-time of a buffer always increments + monotonically along with the clock-time. Buffers are played when their + running-time is equal to the clock-time - base-time. The stream-time + represents the position in the stream and jumps backwards when + repeating. + + Clock providers - + + A clock provider is an element in the pipeline that can provide + a GstClock object. The clock object needs to + report an absoulute-time that is monotonocally increasing when the + element is in the PLAYING state. It is allowed to pause the clock + while the element is PAUSED. + Clock providers exist because they play back media at some rate, and this rate is not necessarily the same as the system clock rate. For example, a soundcard may playback at 44,1 kHz, but that doesn't mean that after exactly 1 second according to the system clock, the soundcard has played back 44.100 - samples. This is only true by approximation. Therefore, generally, - pipelines with an audio output use the audiosink as clock provider. - This ensures that one second of video will be played back at the same - rate as that the soundcard plays back 1 second of audio. + samples. This is only true by approximation. In fact, the audio + device has an internal clock based on the number of samples played + that we can expose. - Whenever some part of the pipeline requires to know the current clock - time, it will be requested from the clock through - gst_clock_get_time (). The clock-time does not - need to start at 0. The pipeline, which contains the global clock that - all elements in the pipeline will use, in addition has a base - time, which is the clock time at the point where the - pipeline went to the PLAYING state. Each element can subtract the - base time from the clock-time to know the current - running time. + If an element with an internal clock needs to synchronize, it needs + to estimate when a time according to the pipeline clock will take + place according to the internal clock. To estimate this, it needs + to slave its clock to the pipeline clock. - The clock provider is responsible for making sure that the clock time - always represents the current media time as closely as possible; it - has to take care of things such as playback latencies, buffering in - audio-kernel modules, and so on, since all those could affect a/v sync - and thus decrease the user experience. + If the pipeline clock is exactly the internal clock of an element, + the element can skip the slaving step and directly use the pipeline + clock to schedule playback. This can be both faster and more + accurate. + Therefore, generally, elements with an internal clock like audio + input or output devices will be a clock provider for the pipeline. + + + When the pipeline goes to the PLAYING state, it will go over all + elements in the pipeline from sink to source and ask each element + if they can provide a clock. The last element that can provide a + clock will be used as the clock provider in the pipeline. + This algorithm prefers a clock from an audio sink in a typical + playback pipeline and a clock from source elements in a typical + capture pipeline. + + + There exist some bus messages to let you know about the clock and + clock providers in the pipeline. You can see what clock is selected + in the pipeline by looking at the NEW_CLOCK message on the bus. + When a clock provider is removed from the pipeline, a CLOCK_LOST + message is posted and the application should go to PAUSED and back + to PLAYING to select a new clock. - - Clock slaves + + Latency - Clock slaves get assigned a clock by their containing pipeline. Their - task is to make sure that media playback follows the time progress as - represented by this clock as closely as possible. For most elements, - that will simply mean to wait until the buffer running-time is reached - before playing back their current sample. + The latency is the time it takes for a sample captured at timestamp X + to reach the sink. This time is measured against the clock in the + pipeline. For pipelines where the only elements that synchronize against + the clock are the sinks, the latency is always 0 since no other element + is delaying the buffer. - The buffer running-time is derived from the buffer timestamp and the - newsegment event preceeding the buffer. A buffer is played synchronized - with the clock when the clock's running-time has reached exactly the - buffer running-time; this can be done with the function - gst_clock_id_wait (). - - - For more information on how to write elements that conform to this - required behaviour, see the Plugin Writer's Guide. + For pipelines with live sources, a latency is introduced, mostly because + of the way a live source works. Consider an audio source, it will start + capturing the first sample at time 0. If the source pushes buffers with + 44100 samples at a time at 44100Hz it will have collected the buffer at + second 1. Since the timestamp of the buffer is 0 and the time of the + clock is now >= 1 second, the sink will drop this buffer because it is + too late. Without any latency compensation in the sink, all buffers will + be dropped. + + + Latency compensation + + Before the pipeline goes to the PLAYING state, it will, in addition to + selecting a clock and calculating a base-time, calculate the latency + in the pipeline. It does this by doing a LATENCY query on all the sinks + in the pipeline. The pipeline then selects the maximum latency in the + pipeline and configures this with a LATENCY event. + + + All sink elements will delay playback by the value in the LATENCY event. + Since all sinks delay with the same amount of time, they will be + relative in sync. + + + + + Dynamic Latency + + Adding/removing elements to/from a pipeline or changing element + properties can change the latency in a pipeline. An element can + request a latency change in the pipeline by posting a LATENCY + message on the bus. The application can then decide to query and + redistribute a new latency or not. Changing the latency in a + pipeline might cause visual or audible glitches and should + therefore only be done by the application when it is allowed. + + diff --git a/docs/pwg/advanced-clock.xml b/docs/pwg/advanced-clock.xml index beb6bd7790..89c1f29e4e 100644 --- a/docs/pwg/advanced-clock.xml +++ b/docs/pwg/advanced-clock.xml @@ -73,7 +73,10 @@ Synchronization is now a matter of making sure that a buffer with a certain running-time is played when the clock reaches the same - running-time. Usually this task is done by sink elements. + running-time. Usually this task is done by sink elements. Sink also + have to take into account the latency configured in the pipeline and + add this to the buffer running-time before synchronizing to the + pipeline clock. @@ -120,11 +123,11 @@ - Parser elements + Parser/Decoder/Encoder elements - Parser elements must use the incomming timestamps and transfer those - to the resulting output buffers. They are allowed to interpolate or - reconstruct timestamps on missing input buffers when they can. + Parser/Decoder elements must use the incomming timestamps and transfer + those to the resulting output buffers. They are allowed to interpolate + or reconstruct timestamps on missing input buffers when they can. @@ -139,8 +142,18 @@ buffer timestamps. + + + Muxer elements + + Muxer elements should use the incomming buffer running-time to mux the + different streams together. They should copy the incomming running-time + to the outgoing buffers. + + - Sink elements + + Sink elements If the element is intended to emit samples at a specific time (real time playing), the element should require a clock, and thus implement the @@ -148,8 +161,9 @@ The sink should then make sure that the sample with running-time is played - exactly when the pipeline clock reaches that running-time. Some elements - might use the clock API such as gst_clock_id_wait() + exactly when the pipeline clock reaches that running-time + latency. + Some elements might use the clock API such as + gst_clock_id_wait() to perform this action. Other sinks might need to use other means of scheduling timely playback of the data.