diff --git a/markdown/design/qos.md b/markdown/design/qos.md index 2181919e9a..65728abd0a 100644 --- a/markdown/design/qos.md +++ b/markdown/design/qos.md @@ -34,24 +34,24 @@ to reduce the framerate, for example. The QoS event is generated by an element that synchronizes against the clock. It travels upstream and contains the following fields: -* **`type`**: `GST\_TYPE\_QOS\_TYPE:` The type of the QoS event, we have the -following types and the default type is `GST\_QOS\_TYPE\_UNDERFLOW`: +* **`type`**: `GST_TYPE_QOS_TYPE:` The type of the QoS event, we have the +following types and the default type is `GST_QOS_TYPE_UNDERFLOW`: - * GST_QOS_TYPE_OVERFLOW: an element is receiving buffers too fast and can't + * `GST_QOS_TYPE_OVERFLOW`: an element is receiving buffers too fast and can't keep up processing them. Upstream should reduce the rate. - * GST_QOS_TYPE_UNDERFLOW: an element is receiving buffers too slowly + * `GST_QOS_TYPE_UNDERFLOW`: an element is receiving buffers too slowly and has to drop them because they are too late. Upstream should increase the processing rate. - * GST_QOS_TYPE_THROTTLE: the application is asking to add extra delay + * `GST_QOS_TYPE_THROTTLE`: the application is asking to add extra delay between buffers, upstream is allowed to drop buffers -* **`timestamp`**: G\_TYPE\_UINT64: The timestamp on the buffer that +* **`timestamp`**: `G_TYPE_UINT64`: The timestamp on the buffer that generated the QoS event. These timestamps are expressed in total -running\_time in the sink so that the value is ever increasing. +`running_time` in the sink so that the value is ever increasing. -* **`jitter`**: G\_TYPE\_INT64: The difference of that timestamp against the +* **`jitter`**: `G_TYPE_INT64`: The difference of that timestamp against the current clock time. Negative values mean the timestamp was on time. Positive values indicate the timestamp was late by that amount. When buffers are received in time and throttling is not enabled, the QoS @@ -59,7 +59,7 @@ type field is set to OVERFLOW. When throttling, the jitter contains the throttling delay added by the application and the type is set to THROTTLE. -* **`proportion`**: G\_TYPE\_DOUBLE: Long term prediction of the ideal rate +* **`proportion`**: `G_TYPE_DOUBLE`: Long term prediction of the ideal rate relative to normal rate to get optimal quality. The rest of this document deals with how these values can be calculated @@ -84,48 +84,48 @@ against the clock (live) or it could be posted by an upstream element that performs QoS because of QOS events received from a downstream element (\!live). -The `GST\_MESSAGE\_QOS` contains at least the following info: +The `GST_MESSAGE_QOS` contains at least the following info: -* **`live`**: G\_TYPE\_BOOLEAN: If the QoS message was dropped by a live +* **`live`**: `G_TYPE_BOOLEAN`: If the QoS message was dropped by a live element such as a sink or a live source. If the live property is FALSE, the QoS message was generated as a response to a QoS event in a non-live element. -* **`running-time`**: G\_TYPE\_UINT64: The running\_time of the buffer that +* **`running-time`**: `G_TYPE_UINT64`: The `running_time` of the buffer that generated the QoS message. -* **`stream-time`**: G\_TYPE\_UINT64: The stream\_time of the buffer that +* **`stream-time`**: `G_TYPE_UINT64`: The `stream_time` of the buffer that generated the QoS message. -* **`timestamp`**: G\_TYPE\_UINT64: The timestamp of the buffer that +* **`timestamp`**: `G_TYPE_UINT64`: The timestamp of the buffer that generated the QoS message. -* **`duration`**: G\_TYPE\_UINT64: The duration of the buffer that generated +* **`duration`**: `G_TYPE_UINT64`: The duration of the buffer that generated the QoS message. -* **`jitter`**: G\_TYPE\_INT64: The difference of the running-time against +* **`jitter`**: `G_TYPE_INT64`: The difference of the running-time against the deadline. Negative values mean the timestamp was on time. Positive values indicate the timestamp was late (and dropped) by -that amount. The deadline can be a realtime running\_time or an -estimated running\_time. +that amount. The deadline can be a realtime `running_time` or an +estimated `running_time`. -* **`proportion`**: G\_TYPE\_DOUBLE: Long term prediction of the ideal rate +* **`proportion`**: `G_TYPE_DOUBLE`: Long term prediction of the ideal rate relative to normal rate to get optimal quality. -* **`quality`**: G\_TYPE\_INT: An element dependent integer value that +* **`quality`**: `G_TYPE_INT`: An element dependent integer value that specifies the current quality level of the element. The default maximum quality is 1000000. -* **`format`**: GST\_TYPE\_FORMAT Units of the *processed* and *dropped* -fields. Video sinks and video filters will use GST\_FORMAT\_BUFFERS +* **`format`**: `GST_TYPE_FORMAT` Units of the *processed* and *dropped* +fields. Video sinks and video filters will use `GST_FORMAT_BUFFERS` (frames). Audio sinks and audio filters will likely use -GST\_FORMAT\_DEFAULT (samples). +`GST_FORMAT_DEFAULT` (samples). -* **`processed`**: G\_TYPE\_UINT64: Total number of units correctly +* **`processed`**: `G_TYPE_UINT64`: Total number of units correctly processed since the last state change to READY or a flushing operation. -* **`dropped`**: G\_TYPE\_UINT64: Total number of units dropped since the +* **`dropped`**: `G_TYPE_UINT64`: Total number of units dropped since the last state change to READY or a flushing operation. The *running-time* and *processed* fields can be used to estimate the @@ -144,7 +144,7 @@ J1 return value from the clock. The jitter J1 is simply calculated as Where CT is the clock time when the entry arrives in the sink. This value is calculated inside the clock when we perform -`gst\_clock\_id\_wait()`. +`gst_clock_id_wait()`. If the jitter is negative, the entry arrived in time and can be rendered after waiting for the clock to reach time B1 (which is also CT - J1). @@ -159,22 +159,28 @@ upstream. Using the jitter we can calculate the time when the buffer arrived in the sink: +``` T1 = B1 + J1. (1) +``` The time the buffer leaves the sink after synchronisation is measured as: +``` T2 = B1 + (J1 < 0 ? 0 : J1) (2) +``` For buffers that arrive in time (J1 \< 0) the buffer leaves after synchronisation which is exactly B1. Late buffers (J1 \>= 0) leave the -sink when they arrive, whithout any synchronisation, which is T2 = T1 = -B1 + J1. +sink when they arrive, whithout any synchronisation, which is `T2 = T1 = +B1 + J1`. Using a previous T0 and a new T1, we can calculate the time it took for upstream to generate a buffer with timestamp B1. +``` PT1 = T1 - T0 (3) +``` We call PT1 the processing time needed to generate buffer with timestamp B1. @@ -206,7 +212,7 @@ performance problems can indeed also be caused by the element itself when it receives too much data it cannot process in time. The element is then said to be overflowed. -# Short term correction +## Short term correction The timestamp and jitter serve as short term correction information for upstream elements. Indeed, given arrival time T1 as given in (1) we can @@ -269,7 +275,7 @@ Which yields a more accurate prediction for the next buffer given as: B2 = B1 + 2 * J1 + D1 (5) ``` -# Long term correction +## Long term correction The datarate used to calculate (5) for the short term prediction is based on a single observation. A more accurate datarate can be obtained @@ -285,7 +291,7 @@ Receivers of the QoS event should permanently reduce their datarate as given by the proportion member. Failure to do so will certainly lead to more dropped frames and a generally worse QoS. -# Throttling +## Throttling In throttle mode, the time distance between buffers is kept to a configurable throttle interval. This means that effectively the buffer @@ -303,7 +309,7 @@ desired throttle interval. Implementations can use the QoS Throttle type, the proportion and the jitter member to tune their implementations. -# QoS strategies +## QoS strategies Several strategies exist to reduce processing delay that might affect real time performance. @@ -325,12 +331,12 @@ real time performance. - assign more CPU(s) to critical pipeline parts -# QoS implementations +## QoS implementations Here follows a small overview of how QoS can be implemented in a range of different types of elements. -# GstBaseSink +### GstBaseSink The primary implementor of QoS is GstBaseSink. It will calculate the following values: @@ -367,12 +373,12 @@ is not noticeable for the human eye. A QoS message is posted whenever a (part of a) buffer is dropped. In throttle mode, the sink sends QoS event upstream with the timestamp -set to the running\_time of the latest buffer and the jitter set to the +set to the `running_time` of the latest buffer and the jitter set to the throttle interval. If the throttled buffer is late, the lateness is subtracted from the throttle interval in order to keep the desired throttle interval. -# GstBaseTransform +### GstBaseTransform Transform elements can entirely skip the transform based on the timestamp and jitter values of recent QoS event since these buffers will @@ -390,7 +396,7 @@ A QoS message should be posted when a frame is dropped or when the quality of the filter is reduced. The quality member in the QOS message should reflect the quality setting of the filter. -# Video Decoders +### Video Decoders A video decoder can, based on the codec in use, decide to not decode intermediate frames. A typical codec can for example skip the decoding @@ -416,7 +422,7 @@ frame every 18 frames. The quality member should be set to `1000000 * - skipping P/B frames: quality = 55555 (for I-frame spacing of 18 frames) -# Demuxers +### Demuxers Demuxers usually cannot do a lot regarding QoS except for skipping frames to the next keyframe when a lateness QoS event arrives on a @@ -429,7 +435,7 @@ Most demuxers that have multiple output pads might need to combine the QoS events on all the pads and derive an aggregated QoS event for the upstream element. -# Sources +### Sources The QoS events only apply to push based sources since pull based sources are entirely controlled by another downstream element.