mirrors/gstreamer
mirrors/gstreamer
1
1
Fork 0
mirror of https://gitlab.freedesktop.org/gstreamer/gstreamer.git synced 2025-02-22 14:06:23 +00:00
Code Issues 1 Projects Releases Wiki Activity

design: qos: fix subsection breakdown, missing markup and unneeded scaping

Browse source
This commit is contained in:
Reynaldo H. Verdejo Pinochet 2016-12-28 22:49:13 -08:00
parent ab1684cc62
commit 886911017e
1 changed files with 45 additions and 39 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

84
markdown/design/qos.md
View file

@ -34,24 +34,24 @@ to reduce the framerate, for example.
The QoS event is generated by an element that synchronizes against the The QoS event is generated by an element that synchronizes against the
clock. It travels upstream and contains the following fields: clock. It travels upstream and contains the following fields:
* **`type`**: `GST\_TYPE\_QOS\_TYPE:` The type of the QoS event, we have the * **`type`**: `GST_TYPE_QOS_TYPE:` The type of the QoS event, we have the
following types and the default type is `GST\_QOS\_TYPE\_UNDERFLOW`: following types and the default type is `GST_QOS_TYPE_UNDERFLOW`:
* GST_QOS_TYPE_OVERFLOW: an element is receiving buffers too fast and can't * `GST_QOS_TYPE_OVERFLOW`: an element is receiving buffers too fast and can't
keep up processing them. Upstream should reduce the rate. keep up processing them. Upstream should reduce the rate.
* GST_QOS_TYPE_UNDERFLOW: an element is receiving buffers too slowly * `GST_QOS_TYPE_UNDERFLOW`: an element is receiving buffers too slowly
and has to drop them because they are too late. Upstream should and has to drop them because they are too late. Upstream should
increase the processing rate. increase the processing rate.
* GST_QOS_TYPE_THROTTLE: the application is asking to add extra delay * `GST_QOS_TYPE_THROTTLE`: the application is asking to add extra delay
between buffers, upstream is allowed to drop buffers between buffers, upstream is allowed to drop buffers
* **`timestamp`**: G\_TYPE\_UINT64: The timestamp on the buffer that * **`timestamp`**: `G_TYPE_UINT64`: The timestamp on the buffer that
generated the QoS event. These timestamps are expressed in total generated the QoS event. These timestamps are expressed in total
running\_time in the sink so that the value is ever increasing. `running_time` in the sink so that the value is ever increasing.
* **`jitter`**: G\_TYPE\_INT64: The difference of that timestamp against the * **`jitter`**: `G_TYPE_INT64`: The difference of that timestamp against the
current clock time. Negative values mean the timestamp was on time. current clock time. Negative values mean the timestamp was on time.
Positive values indicate the timestamp was late by that amount. When Positive values indicate the timestamp was late by that amount. When
buffers are received in time and throttling is not enabled, the QoS buffers are received in time and throttling is not enabled, the QoS
@ -59,7 +59,7 @@ type field is set to OVERFLOW. When throttling, the jitter contains
the throttling delay added by the application and the type is set to the throttling delay added by the application and the type is set to
THROTTLE. THROTTLE.
* **`proportion`**: G\_TYPE\_DOUBLE: Long term prediction of the ideal rate * **`proportion`**: `G_TYPE_DOUBLE`: Long term prediction of the ideal rate
relative to normal rate to get optimal quality. relative to normal rate to get optimal quality.
The rest of this document deals with how these values can be calculated The rest of this document deals with how these values can be calculated
@ -84,48 +84,48 @@ against the clock (live) or it could be posted by an upstream element
that performs QoS because of QOS events received from a downstream that performs QoS because of QOS events received from a downstream
element (\!live). element (\!live).
The `GST\_MESSAGE\_QOS` contains at least the following info: The `GST_MESSAGE_QOS` contains at least the following info:
* **`live`**: G\_TYPE\_BOOLEAN: If the QoS message was dropped by a live * **`live`**: `G_TYPE_BOOLEAN`: If the QoS message was dropped by a live
element such as a sink or a live source. If the live property is element such as a sink or a live source. If the live property is
FALSE, the QoS message was generated as a response to a QoS event in FALSE, the QoS message was generated as a response to a QoS event in
a non-live element. a non-live element.
* **`running-time`**: G\_TYPE\_UINT64: The running\_time of the buffer that * **`running-time`**: `G_TYPE_UINT64`: The `running_time` of the buffer that
generated the QoS message. generated the QoS message.
* **`stream-time`**: G\_TYPE\_UINT64: The stream\_time of the buffer that * **`stream-time`**: `G_TYPE_UINT64`: The `stream_time` of the buffer that
generated the QoS message. generated the QoS message.
* **`timestamp`**: G\_TYPE\_UINT64: The timestamp of the buffer that * **`timestamp`**: `G_TYPE_UINT64`: The timestamp of the buffer that
generated the QoS message. generated the QoS message.
* **`duration`**: G\_TYPE\_UINT64: The duration of the buffer that generated * **`duration`**: `G_TYPE_UINT64`: The duration of the buffer that generated
the QoS message. the QoS message.
* **`jitter`**: G\_TYPE\_INT64: The difference of the running-time against * **`jitter`**: `G_TYPE_INT64`: The difference of the running-time against
the deadline. Negative values mean the timestamp was on time. the deadline. Negative values mean the timestamp was on time.
Positive values indicate the timestamp was late (and dropped) by Positive values indicate the timestamp was late (and dropped) by
that amount. The deadline can be a realtime running\_time or an that amount. The deadline can be a realtime `running_time` or an
estimated running\_time. estimated `running_time`.
* **`proportion`**: G\_TYPE\_DOUBLE: Long term prediction of the ideal rate * **`proportion`**: `G_TYPE_DOUBLE`: Long term prediction of the ideal rate
relative to normal rate to get optimal quality. relative to normal rate to get optimal quality.
* **`quality`**: G\_TYPE\_INT: An element dependent integer value that * **`quality`**: `G_TYPE_INT`: An element dependent integer value that
specifies the current quality level of the element. The default specifies the current quality level of the element. The default
maximum quality is 1000000. maximum quality is 1000000.
* **`format`**: GST\_TYPE\_FORMAT Units of the *processed* and *dropped* * **`format`**: `GST_TYPE_FORMAT` Units of the *processed* and *dropped*
fields. Video sinks and video filters will use GST\_FORMAT\_BUFFERS fields. Video sinks and video filters will use `GST_FORMAT_BUFFERS`
(frames). Audio sinks and audio filters will likely use (frames). Audio sinks and audio filters will likely use
GST\_FORMAT\_DEFAULT (samples). `GST_FORMAT_DEFAULT` (samples).
* **`processed`**: G\_TYPE\_UINT64: Total number of units correctly * **`processed`**: `G_TYPE_UINT64`: Total number of units correctly
processed since the last state change to READY or a flushing processed since the last state change to READY or a flushing
operation. operation.
* **`dropped`**: G\_TYPE\_UINT64: Total number of units dropped since the * **`dropped`**: `G_TYPE_UINT64`: Total number of units dropped since the
last state change to READY or a flushing operation. last state change to READY or a flushing operation.
The *running-time* and *processed* fields can be used to estimate the The *running-time* and *processed* fields can be used to estimate the
@ -144,7 +144,7 @@ J1 return value from the clock. The jitter J1 is simply calculated as
Where CT is the clock time when the entry arrives in the sink. This Where CT is the clock time when the entry arrives in the sink. This
value is calculated inside the clock when we perform value is calculated inside the clock when we perform
`gst\_clock\_id\_wait()`. `gst_clock_id_wait()`.
If the jitter is negative, the entry arrived in time and can be rendered If the jitter is negative, the entry arrived in time and can be rendered
after waiting for the clock to reach time B1 (which is also CT - J1). after waiting for the clock to reach time B1 (which is also CT - J1).
@ -159,22 +159,28 @@ upstream.
Using the jitter we can calculate the time when the buffer arrived in Using the jitter we can calculate the time when the buffer arrived in
the sink: the sink:
```
T1 = B1 + J1. (1) T1 = B1 + J1. (1)
```
The time the buffer leaves the sink after synchronisation is measured The time the buffer leaves the sink after synchronisation is measured
as: as:
```
T2 = B1 + (J1 < 0 ? 0 : J1) (2) T2 = B1 + (J1 < 0 ? 0 : J1) (2)
```
For buffers that arrive in time (J1 \< 0) the buffer leaves after For buffers that arrive in time (J1 \< 0) the buffer leaves after
synchronisation which is exactly B1. Late buffers (J1 \>= 0) leave the synchronisation which is exactly B1. Late buffers (J1 \>= 0) leave the
sink when they arrive, whithout any synchronisation, which is T2 = T1 = sink when they arrive, whithout any synchronisation, which is `T2 = T1 =
B1 + J1. B1 + J1`.
Using a previous T0 and a new T1, we can calculate the time it took for Using a previous T0 and a new T1, we can calculate the time it took for
upstream to generate a buffer with timestamp B1. upstream to generate a buffer with timestamp B1.
```
PT1 = T1 - T0 (3) PT1 = T1 - T0 (3)
```
We call PT1 the processing time needed to generate buffer with timestamp We call PT1 the processing time needed to generate buffer with timestamp
B1. B1.
@ -206,7 +212,7 @@ performance problems can indeed also be caused by the element itself
when it receives too much data it cannot process in time. The element is when it receives too much data it cannot process in time. The element is
then said to be overflowed. then said to be overflowed.
# Short term correction ## Short term correction
The timestamp and jitter serve as short term correction information for The timestamp and jitter serve as short term correction information for
upstream elements. Indeed, given arrival time T1 as given in (1) we can upstream elements. Indeed, given arrival time T1 as given in (1) we can
@ -269,7 +275,7 @@ Which yields a more accurate prediction for the next buffer given as:
B2 = B1 + 2 * J1 + D1 (5) B2 = B1 + 2 * J1 + D1 (5)
``` ```
# Long term correction ## Long term correction
The datarate used to calculate (5) for the short term prediction is The datarate used to calculate (5) for the short term prediction is
based on a single observation. A more accurate datarate can be obtained based on a single observation. A more accurate datarate can be obtained
@ -285,7 +291,7 @@ Receivers of the QoS event should permanently reduce their datarate as
given by the proportion member. Failure to do so will certainly lead to given by the proportion member. Failure to do so will certainly lead to
more dropped frames and a generally worse QoS. more dropped frames and a generally worse QoS.
# Throttling ## Throttling
In throttle mode, the time distance between buffers is kept to a In throttle mode, the time distance between buffers is kept to a
configurable throttle interval. This means that effectively the buffer configurable throttle interval. This means that effectively the buffer
@ -303,7 +309,7 @@ desired throttle interval. Implementations can use the QoS Throttle
type, the proportion and the jitter member to tune their type, the proportion and the jitter member to tune their
implementations. implementations.
# QoS strategies ## QoS strategies
Several strategies exist to reduce processing delay that might affect Several strategies exist to reduce processing delay that might affect
real time performance. real time performance.
@ -325,12 +331,12 @@ real time performance.
- assign more CPU(s) to critical pipeline parts - assign more CPU(s) to critical pipeline parts
# QoS implementations ## QoS implementations
Here follows a small overview of how QoS can be implemented in a range Here follows a small overview of how QoS can be implemented in a range
of different types of elements. of different types of elements.
# GstBaseSink ### GstBaseSink
The primary implementor of QoS is GstBaseSink. It will calculate the The primary implementor of QoS is GstBaseSink. It will calculate the
following values: following values:
@ -367,12 +373,12 @@ is not noticeable for the human eye.
A QoS message is posted whenever a (part of a) buffer is dropped. A QoS message is posted whenever a (part of a) buffer is dropped.
In throttle mode, the sink sends QoS event upstream with the timestamp In throttle mode, the sink sends QoS event upstream with the timestamp
set to the running\_time of the latest buffer and the jitter set to the set to the `running_time` of the latest buffer and the jitter set to the
throttle interval. If the throttled buffer is late, the lateness is throttle interval. If the throttled buffer is late, the lateness is
subtracted from the throttle interval in order to keep the desired subtracted from the throttle interval in order to keep the desired
throttle interval. throttle interval.
# GstBaseTransform ### GstBaseTransform
Transform elements can entirely skip the transform based on the Transform elements can entirely skip the transform based on the
timestamp and jitter values of recent QoS event since these buffers will timestamp and jitter values of recent QoS event since these buffers will
@ -390,7 +396,7 @@ A QoS message should be posted when a frame is dropped or when the
quality of the filter is reduced. The quality member in the QOS message quality of the filter is reduced. The quality member in the QOS message
should reflect the quality setting of the filter. should reflect the quality setting of the filter.
# Video Decoders ### Video Decoders
A video decoder can, based on the codec in use, decide to not decode A video decoder can, based on the codec in use, decide to not decode
intermediate frames. A typical codec can for example skip the decoding intermediate frames. A typical codec can for example skip the decoding
@ -416,7 +422,7 @@ frame every 18 frames. The quality member should be set to `1000000 *
- skipping P/B frames: quality = 55555 (for I-frame spacing of 18 - skipping P/B frames: quality = 55555 (for I-frame spacing of 18
frames) frames)
# Demuxers ### Demuxers
Demuxers usually cannot do a lot regarding QoS except for skipping Demuxers usually cannot do a lot regarding QoS except for skipping
frames to the next keyframe when a lateness QoS event arrives on a frames to the next keyframe when a lateness QoS event arrives on a
@ -429,7 +435,7 @@ Most demuxers that have multiple output pads might need to combine the
QoS events on all the pads and derive an aggregated QoS event for the QoS events on all the pads and derive an aggregated QoS event for the
upstream element. upstream element.
# Sources ### Sources
The QoS events only apply to push based sources since pull based sources The QoS events only apply to push based sources since pull based sources
are entirely controlled by another downstream element. are entirely controlled by another downstream element.