mirror of
https://gitlab.freedesktop.org/gstreamer/gstreamer.git
synced 2024-12-13 03:46:34 +00:00
436 lines
16 KiB
Text
436 lines
16 KiB
Text
Quality-of-Service
|
|
------------------
|
|
|
|
Quality of service is about measuring and adjusting the real-time
|
|
performance of a pipeline.
|
|
|
|
The real-time performance is always measured relative to the pipeline
|
|
clock and typically happens in the sinks when they synchronize buffers
|
|
against the clock.
|
|
|
|
The measurements result in QOS events that aim to adjust the datarate
|
|
in one or more upstream elements. Two types of adjustments can be
|
|
made:
|
|
|
|
- short time "emergency" corrections based on latest observation
|
|
in the sinks.
|
|
- long term rate corrections based on trends observed in the sinks.
|
|
|
|
It is also possible for the application to artificially introduce delay
|
|
between synchronized buffers, this is called throttling. It can be used
|
|
to reduce the framerate, for example.
|
|
|
|
|
|
Sources of quality problems
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
- High CPU load
|
|
- Network problems
|
|
- Other resource problems such as disk load, memory bottlenecks etc.
|
|
- application level throttling
|
|
|
|
|
|
QoS event
|
|
~~~~~~~~~
|
|
|
|
The QoS event is generated by an element that synchronizes against the clock. It
|
|
travels upstream and contains the following fields:
|
|
|
|
- type, GST_TYPE_QOS_TYPE:
|
|
The type of the QoS event, we have the following types and the default type
|
|
is GST_QOS_TYPE_UNDERFLOW:
|
|
|
|
GST_QOS_TYPE_OVERFLOW: an element is receiving buffers too fast and can't
|
|
keep up processing them. Upstream should reduce the
|
|
rate.
|
|
GST_QOS_TYPE_UNDERFLOW: an element is receiving buffers too slowly and has
|
|
to drop them because they are too late. Upstream should
|
|
increase the processing rate.
|
|
GST_QOS_TYPE_THROTTLE: the application is asking to add extra delay between
|
|
buffers, upstream is allowed to drop buffers
|
|
|
|
- timestamp, G_TYPE_UINT64:
|
|
The timestamp on the buffer that generated the QoS event. These timestamps
|
|
are expressed in total running_time in the sink so that the value is ever
|
|
increasing.
|
|
|
|
- jitter, G_TYPE_INT64:
|
|
The difference of that timestamp against the current clock time. Negative
|
|
values mean the timestamp was on time. Positive values indicate the
|
|
timestamp was late by that amount. When buffers are received in time and
|
|
throttling is not enabled, the QoS type field is set to OVERFLOW.
|
|
When throttling, the jitter contains the throttling delay added by the
|
|
application and the type is set to THROTTLE.
|
|
|
|
- proportion, G_TYPE_DOUBLE:
|
|
Long term prediction of the ideal rate relative to normal rate to get
|
|
optimal quality.
|
|
|
|
The rest of this document deals with how these values can be calculated
|
|
in a sink and how the values can be used by other elements to adjust their
|
|
operations.
|
|
|
|
|
|
QoS message
|
|
~~~~~~~~~~~
|
|
|
|
A QOS message is posted on the bus whenever an element decides to:
|
|
|
|
- drop a buffer because of QoS reasons
|
|
- change its processing strategy because of QoS reasons (quality)
|
|
|
|
It should be expected that creating and posting the QoS message is reasonably
|
|
fast and does not significantly contribute to the QoS problems. Options to
|
|
disable this feature could also be presented on elements.
|
|
|
|
This message can be posted by a sink/src that performs synchronisation against the
|
|
clock (live) or it could be posted by an upstream element that performs QoS
|
|
because of QOS events received from a downstream element (!live).
|
|
|
|
The GST_MESSAGE_QOS contains at least the following info:
|
|
|
|
- live: G_TYPE_BOOLEAN:
|
|
If the QoS message was dropped by a live element such as a sink or a live
|
|
source. If the live property is FALSE, the QoS message was generated as a
|
|
response to a QoS event in a non-live element.
|
|
|
|
- running-time, G_TYPE_UINT64:
|
|
The running_time of the buffer that generated the QoS message.
|
|
|
|
- stream-time, G_TYPE_UINT64:
|
|
The stream_time of the buffer that generated the QoS message.
|
|
|
|
- timestamp, G_TYPE_UINT64:
|
|
The timestamp of the buffer that generated the QoS message.
|
|
|
|
- duration, G_TYPE_UINT64:
|
|
The duration of the buffer that generated the QoS message.
|
|
|
|
|
|
- jitter, G_TYPE_INT64:
|
|
The difference of the running-time against the deadline. Negative
|
|
values mean the timestamp was on time. Positive values indicate the
|
|
timestamp was late (and dropped) by that amount. The deadline can be
|
|
a realtime running_time or an estimated running_time.
|
|
|
|
- proportion, G_TYPE_DOUBLE:
|
|
Long term prediction of the ideal rate relative to normal rate to get
|
|
optimal quality.
|
|
|
|
- quality, G_TYPE_INT:
|
|
An element dependent integer value that specifies the current quality
|
|
level of the element. The default maximum quality is 1000000.
|
|
|
|
|
|
- format, GST_TYPE_FORMAT
|
|
Units of the 'processed' and 'dropped' fields. Video sinks and video
|
|
filters will use GST_FORMAT_BUFFERS (frames). Audio sinks and audio filters
|
|
will likely use GST_FORMAT_DEFAULT (samples).
|
|
|
|
- processed: G_TYPE_UINT64:
|
|
Total number of units correctly processed since the last state change to
|
|
READY or a flushing operation.
|
|
|
|
- dropped: G_TYPE_UINT64:
|
|
Total number of units dropped since the last state change to READY or a
|
|
flushing operation.
|
|
|
|
The 'running-time' and 'processed' fields can be used to estimate the average
|
|
processing rate (framerate for video).
|
|
|
|
Elements might add additional fields in the message which are documented in the
|
|
relevant elements or baseclasses.
|
|
|
|
|
|
Collecting statistics
|
|
~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
A buffer with timestamp B1 arrives in the sink at time T1. The buffer
|
|
timestamp is then synchronized against the clock which yields a jitter J1
|
|
return value from the clock. The jitter J1 is simply calculated as
|
|
|
|
J1 = CT - B1
|
|
|
|
Where CT is the clock time when the entry arrives in the sink. This value
|
|
is calculated inside the clock when we perform gst_clock_id_wait().
|
|
|
|
If the jitter is negative, the entry arrived in time and can be rendered
|
|
after waiting for the clock to reach time B1 (which is also CT - J1).
|
|
|
|
If the jitter is positive however, the entry arrived too late in the sink
|
|
and should therefore be dropped. J1 is the amount of time the entry was late.
|
|
|
|
Any buffer that arrives in the sink should generate a QoS event upstream.
|
|
|
|
Using the jitter we can calculate the time when the buffer arrived in the
|
|
sink:
|
|
|
|
T1 = B1 + J1. (1)
|
|
|
|
The time the buffer leaves the sink after synchronisation is measured as:
|
|
|
|
T2 = B1 + (J1 < 0 ? 0 : J1) (2)
|
|
|
|
For buffers that arrive in time (J1 < 0) the buffer leaves after synchronisation
|
|
which is exactly B1. Late buffers (J1 >= 0) leave the sink when they arrive,
|
|
whithout any synchronisation, which is T2 = T1 = B1 + J1.
|
|
|
|
Using a previous T0 and a new T1, we can calculate the time it took for
|
|
upstream to generate a buffer with timestamp B1.
|
|
|
|
PT1 = T1 - T0 (3)
|
|
|
|
We call PT1 the processing time needed to generate buffer with timestamp B1.
|
|
|
|
Moreover, given the duration of the buffer D1, the current data rate (DR1) of
|
|
the upstream element is given as:
|
|
|
|
PT1 T1 - T0
|
|
DR1 = --- = ------- (4)
|
|
D1 D1
|
|
|
|
For values 0.0 < DR1 <= 1.0 the upstream element is producing faster than
|
|
real-time. If DR1 is exactly 1.0, the element is running at a perfect speed.
|
|
|
|
Values DR1 > 1.0 mean that the upstream element cannot produce buffers of
|
|
duration D1 in real-time. It is exactly DR1 that tells the amount of speedup
|
|
we require from upstream to regain real-time performance.
|
|
|
|
An element that is not receiving enough data is said to be underflowed.
|
|
|
|
|
|
Element measurements
|
|
~~~~~~~~~~~~~~~~~~~~
|
|
|
|
In addition to the measurements of the datarate of the upstream element, a
|
|
typical element must also measure its own performance. Global pipeline
|
|
performance problems can indeed also be caused by the element itself when it
|
|
receives too much data it cannot process in time. The element is then said to
|
|
be overflowed.
|
|
|
|
|
|
Short term correction
|
|
---------------------
|
|
|
|
The timestamp and jitter serve as short term correction information
|
|
for upstream elements. Indeed, given arrival time T1 as given in (1)
|
|
we can be certain that buffers with a timestamp B2 < T1 will be too late
|
|
in the sink.
|
|
|
|
In case of a positive jitter we can therefore send a QoS event with
|
|
a timestamp B1, jitter J1 and proportion given by (4).
|
|
|
|
This allows an upstream element to not generate any data with timestamps
|
|
B2 < T1, where the element can derive T1 as B1 + J1.
|
|
|
|
This will effectively result in frame drops.
|
|
|
|
The element can even do a better estimation of the next valid timestamp it
|
|
should output.
|
|
|
|
Indeed, given the element generated a buffer with timestamp B0 that arrived
|
|
in time in the sink but then received a QoS event stating B1 arrived J1
|
|
too late. This means generating B1 took (B1 + J1) - B0 = T1 - T0 = PT1, as
|
|
given in (3). Given the buffer B1 had a duration D1 and assuming that
|
|
generating a new buffer B2 will take the same amount of processing time,
|
|
a better estimation for B2 would then be:
|
|
|
|
B2 = T1 + D2 * DR1
|
|
|
|
expanding gives:
|
|
|
|
B2 = (B1 + J1) + D2 * (B1 + J1 - B0)
|
|
--------------
|
|
D1
|
|
|
|
assuming the durations of the frames are equal and thus D1 = D2:
|
|
|
|
B2 = (B1 + J1) + (B1 + J1 - B0)
|
|
|
|
B2 = 2 * (B1 + J1) - B0
|
|
|
|
also:
|
|
|
|
B0 = B1 - D1
|
|
|
|
so:
|
|
|
|
B2 = 2 * (B1 + J1) - (B1 - D1)
|
|
|
|
Which yields a more accurate prediction for the next buffer given as:
|
|
|
|
B2 = B1 + 2 * J1 + D1 (5)
|
|
|
|
|
|
Long term correction
|
|
--------------------
|
|
|
|
The datarate used to calculate (5) for the short term prediction is based
|
|
on a single observation. A more accurate datarate can be obtained by
|
|
creating a running average over multiple datarate observations.
|
|
|
|
This average is less susceptible to sudden changes that would only influence
|
|
the datarate for a very short period.
|
|
|
|
A running average is calculated over the observations given in (4) and is
|
|
used as the proportion member in the QoS event that is sent upstream.
|
|
|
|
Receivers of the QoS event should permanently reduce their datarate
|
|
as given by the proportion member. Failure to do so will certainly lead to
|
|
more dropped frames and a generally worse QoS.
|
|
|
|
|
|
Throttling
|
|
----------
|
|
|
|
In throttle mode, the time distance between buffers is kept to a configurable
|
|
throttle interval. This means that effectively the buffer rate is limited
|
|
to 1 buffer per throttle interval. This can be used to limit the framerate,
|
|
for example.
|
|
|
|
When an element is configured in throttling mode (this is usually only
|
|
implemented on sinks) it should produce QoS events upstream with the jitter
|
|
field set to the throttle interval. This should instruct upstream elements to
|
|
skip or drop the remaining buffers in the configured throttle interval.
|
|
|
|
The proportion field is set to the desired slowdown needed to get the
|
|
desired throttle interval. Implementations can use the QoS Throttle type,
|
|
the proportion and the jitter member to tune their implementations.
|
|
|
|
|
|
QoS strategies
|
|
--------------
|
|
|
|
Several strategies exist to reduce processing delay that might affect
|
|
real time performance.
|
|
|
|
- lowering quality
|
|
- dropping frames (reduce CPU/bandwidth usage)
|
|
- switch to a lower decoding/encoding quality (reduce algorithmic
|
|
complexity)
|
|
- switch to a lower quality source (reduce network usage)
|
|
- increasing thread priorities
|
|
- switch to real-time scheduling
|
|
- assign more CPU cycles to critial pipeline parts
|
|
- assign more CPU(s) to critical pipeline parts
|
|
|
|
|
|
QoS implementations
|
|
-------------------
|
|
|
|
Here follows a small overview of how QoS can be implemented in a range of
|
|
different types of elements.
|
|
|
|
|
|
GstBaseSink
|
|
-----------
|
|
|
|
The primary implementor of QoS is GstBaseSink. It will calculate the following
|
|
values:
|
|
|
|
- upstream running average of processing time (5) in stream time.
|
|
- running average of buffer durations.
|
|
- running average of render time (in system time)
|
|
- rendered/dropped buffers
|
|
|
|
The processing time and the average buffer durations will be used to
|
|
calculate a proportion.
|
|
|
|
The processing time in system time is compared to render time to decide if
|
|
the majority of the time is spend upstream or in the sink itself. This value
|
|
is used to decide overflow or underflow.
|
|
|
|
The number of rendered and dropped buffers is used to query stats on the sink.
|
|
|
|
A QoS event with the most current values is sent upstream for each buffer
|
|
that was received by the sink.
|
|
|
|
Normally QoS is only enabled for video pipelines. The reason being that drops
|
|
in audio are more disturbing than dropping video frames. Also video requires in
|
|
general more processing than audio.
|
|
|
|
Normally there is a threshold for when buffers get dropped in a video sink. Frames
|
|
that arrive 20 milliseconds late are still rendered as it is not noticeable for
|
|
the human eye.
|
|
|
|
A QoS message is posted whenever a (part of a) buffer is dropped.
|
|
|
|
In throttle mode, the sink sends QoS event upstream with the timestamp set to
|
|
the running_time of the latest buffer and the jitter set to the throttle interval.
|
|
If the throttled buffer is late, the lateness is subtracted from the throttle
|
|
interval in order to keep the desired throttle interval.
|
|
|
|
|
|
GstBaseTransform
|
|
----------------
|
|
|
|
Transform elements can entirely skip the transform based on the timestamp and
|
|
jitter values of recent QoS event since these buffers will certainly arrive
|
|
too late.
|
|
|
|
With any intermediate element, the element should measure its performance to
|
|
decide if it is responsible for the quality problems or any upstream/downstream
|
|
element.
|
|
|
|
some transforms can reduce the complexity of their algorithms. Depending on the
|
|
algorithm, the changes in quality may have disturbing visual or audible effect
|
|
that should be avoided.
|
|
|
|
A QoS message should be posted when a frame is dropped or when the quality
|
|
of the filter is reduced. The quality member in the QOS message should reflect
|
|
the quality setting of the filter.
|
|
|
|
|
|
Video Decoders
|
|
--------------
|
|
|
|
A video decoder can, based on the codec in use, decide to not decode intermediate
|
|
frames. A typical codec can for example skip the decoding of B-frames to reduce
|
|
the CPU usage and framerate.
|
|
|
|
If each frame is independantly decodable, any arbitrary frame can be skipped based
|
|
on the timestamp and jitter values of the latest QoS event. In addition can the
|
|
proportion member be used to permanently skip frames.
|
|
|
|
It is suggested to adjust the quality field of the QoS message with the expected
|
|
amount of dropped frames (skipping B and/or P frames). This depends on the
|
|
particular spacing of B and P frames in the stream. If the quality control would
|
|
result in half of the frames to be dropped (typical B frame skipping), the
|
|
quality field would be set to 1000000 * 1/2 = 500000. If a typical I frame spacing
|
|
of 18 frames is used, skipping B and P frames would result in 17 dropped frames
|
|
or 1 decoded frame every 18 frames. The quality member should be set to
|
|
1000000 * 1/18 = 55555.
|
|
|
|
- skipping B frames: quality = 500000
|
|
- skipping P/B frames: quality = 55555 (for I-frame spacing of 18 frames)
|
|
|
|
|
|
Demuxers
|
|
--------
|
|
|
|
Demuxers usually cannot do a lot regarding QoS except for skipping frames to the next
|
|
keyframe when a lateness QoS event arrives on a source pad.
|
|
|
|
A demuxer can however measure if the performance problems are upstream or downstream
|
|
and forward an updated QoS event upstream.
|
|
|
|
Most demuxers that have multiple output pads might need to combine the QoS
|
|
events on all the pads and derive an aggregated QoS event for the upstream element.
|
|
|
|
|
|
Sources
|
|
-------
|
|
|
|
The QoS events only apply to push based sources since pull based sources are entirely
|
|
controlled by another downstream element.
|
|
|
|
Sources can receive a overflow or underflow event that can be used to switch to
|
|
less demanding source material. In case of a network stream, a switch could be done
|
|
to a lower or higher quality stream or additional enhancement layers could be used
|
|
or ignored.
|
|
|
|
Live sources will automatically drop data when it takes too long to process the data
|
|
that the element pushes out.
|
|
|
|
Live sources should post a QoS message when data is dropped.
|
|
|