When playing complex media, each sound and video sample must be played in a specific order at a specific time. For this purpose, GStreamer provides a synchronization mechanism. &GStreamer; provides support for the following use cases: Non-live sources with access faster than playback rate. This is the case where one is reading media from a file and playing it back in a synchronized fashion. In this case, multiple streams need to be synchronized, like audio, video and subtitles. Capture and synchronized muxing/mixing of media from multiple live sources. This is a typical use case where you record audio and video from a microphone/camera and mux it into a file for storage. Streaming from (slow) network streams with buffering. This is the typical web streaming case where you access content from a streaming server with http. Capture from live source and and playback to live source with configurable latency. This is used when, for example, capture from a camera, apply an effect and display the result. It is also used when streaming low latency content over a network with UDP. Simultaneous live capture and playback from prerecorded content. This is used in audio recording cases where you play a previously recorded audio and record new samples, the purpose is to have the new audio perfectly in sync with the previously recorded data. &GStreamer; uses a GstClock object, buffer timestamps and a SEGMENT event to synchronize streams in a pipeline as we will see in the next sections. In a typical computer, there are many sources that can be used as a time source, e.g., the system time, soundcards, CPU performance counters, ... For this reason, there are many GstClock implementations available in &GStreamer;. The clock time doesn't always start from 0 or from some known value. Some clocks start counting from some known start date, other clocks start counting since last reboot, etc... A GstClock returns the absolute-time according to that clock with gst_clock_get_time (). The absolute-time (or clock time) of a clock is monotonically increasing. From the absolute-time is a running-time calculated, which is simply the difference between a previous snapshot of the absolute-time called the base-time. So: running-time = absolute-time - base-time A &GStreamer; GstPipeline object maintains a GstClock object and a base-time when it goes to the PLAYING state. The pipeline gives a handle to the selected GstClock to each element in the pipeline along with selected base-time. The pipeline will select a base-time in such a way that the running-time reflects the total time spent in the PLAYING state. As a result, when the pipeline is PAUSED, the running-time stands still. Because all objects in the pipeline have the same clock and base-time, they can thus all calculate the running-time according to the pipeline clock. To calculate a buffer running-time, we need a buffer timestamp and the SEGMENT event that preceeded the buffer. First we can convert the SEGMENT event into a GstSegment object and then we can use the gst_segment_to_running_time () function to perform the calculation of the buffer running-time. Synchronization is now a matter of making sure that a buffer with a certain running-time is played when the clock reaches the same running-time. Usually this task is done by sink elements. Sink also have to take into account the latency configured in the pipeline and add this to the buffer running-time before synchronizing to the pipeline clock. Non-live sources timestamp buffers with a running-time starting from 0. After a flushing seek, they will produce buffers again from a running-time of 0. Live sources need to timestamp buffers with a running-time matching the pipeline running-time when the first byte of the buffer was captured. The buffer stream-time, also known as the position in the stream, is calculated from the buffer timestamps and the preceding SEGMENT event. It represents the time inside the media as a value between 0 and the total duration of the media. The stream-time is used in: Report the current position in the stream with the POSITION query. The position used in the seek events and queries. The position used to synchronize controlled values. The stream-time is never used to synchronize streams, this is only done with the running-time. Here is an overview of the various timelines used in &GStreamer;. The image below represents the different times in the pipeline when playing a 100ms sample and repeating the part between 50ms and 100ms.

You can see how the running-time of a buffer always increments monotonically along with the clock-time. Buffers are played when their running-time is equal to the clock-time - base-time. The stream-time represents the position in the stream and jumps backwards when repeating. A clock provider is an element in the pipeline that can provide a GstClock object. The clock object needs to report an absolute-time that is monotonically increasing when the element is in the PLAYING state. It is allowed to pause the clock while the element is PAUSED. Clock providers exist because they play back media at some rate, and this rate is not necessarily the same as the system clock rate. For example, a soundcard may playback at 44,1 kHz, but that doesn't mean that after exactly 1 second according to the system clock, the soundcard has played back 44.100 samples. This is only true by approximation. In fact, the audio device has an internal clock based on the number of samples played that we can expose. If an element with an internal clock needs to synchronize, it needs to estimate when a time according to the pipeline clock will take place according to the internal clock. To estimate this, it needs to slave its clock to the pipeline clock. If the pipeline clock is exactly the internal clock of an element, the element can skip the slaving step and directly use the pipeline clock to schedule playback. This can be both faster and more accurate. Therefore, generally, elements with an internal clock like audio input or output devices will be a clock provider for the pipeline. When the pipeline goes to the PLAYING state, it will go over all elements in the pipeline from sink to source and ask each element if they can provide a clock. The last element that can provide a clock will be used as the clock provider in the pipeline. This algorithm prefers a clock from an audio sink in a typical playback pipeline and a clock from source elements in a typical capture pipeline. There exist some bus messages to let you know about the clock and clock providers in the pipeline. You can see what clock is selected in the pipeline by looking at the NEW_CLOCK message on the bus. When a clock provider is removed from the pipeline, a CLOCK_LOST message is posted and the application should go to PAUSED and back to PLAYING to select a new clock. The latency is the time it takes for a sample captured at timestamp X to reach the sink. This time is measured against the clock in the pipeline. For pipelines where the only elements that synchronize against the clock are the sinks, the latency is always 0 since no other element is delaying the buffer. For pipelines with live sources, a latency is introduced, mostly because of the way a live source works. Consider an audio source, it will start capturing the first sample at time 0. If the source pushes buffers with 44100 samples at a time at 44100Hz it will have collected the buffer at second 1. Since the timestamp of the buffer is 0 and the time of the clock is now >= 1 second, the sink will drop this buffer because it is too late. Without any latency compensation in the sink, all buffers will be dropped. Before the pipeline goes to the PLAYING state, it will, in addition to selecting a clock and calculating a base-time, calculate the latency in the pipeline. It does this by doing a LATENCY query on all the sinks in the pipeline. The pipeline then selects the maximum latency in the pipeline and configures this with a LATENCY event. All sink elements will delay playback by the value in the LATENCY event. Since all sinks delay with the same amount of time, they will be relative in sync. Adding/removing elements to/from a pipeline or changing element properties can change the latency in a pipeline. An element can request a latency change in the pipeline by posting a LATENCY message on the bus. The application can then decide to query and redistribute a new latency or not. Changing the latency in a pipeline might cause visual or audible glitches and should therefore only be done by the application when it is allowed.