mirror of
https://gitlab.freedesktop.org/gstreamer/gstreamer.git
synced 2024-12-15 04:46:32 +00:00
308 lines
12 KiB
Text
308 lines
12 KiB
Text
Buffering
|
|
---------
|
|
|
|
This document outlines the buffering policy used in the GStreamer
|
|
core that can be used by plugins and applications.
|
|
|
|
The purpose of buffering is to accumulate enough data in a pipeline so that
|
|
playback can occur smoothly and without interruptions. It is typically done
|
|
when reading from a (slow) and non-live network source but can also be used for
|
|
live sources.
|
|
|
|
We want to be able to implement the following features:
|
|
|
|
- buffering up to a specifc amount of data, in memory, before starting playback
|
|
so that network fluctuations are minimized.
|
|
- download of the network file to a local disk with fast seeking in the
|
|
downloaded data. This is similar to the quicktime/youtube players.
|
|
- caching of semi-live streams to a local, on disk, ringbuffer with seeking in
|
|
the cached area. This is similar to tivo-like timeshifting.
|
|
- progress report about the buffering operations
|
|
- easy (backward compatible) application notification of buffering
|
|
- the possibility for the application to do more complex buffering
|
|
|
|
Some use cases:
|
|
|
|
* Stream buffering:
|
|
|
|
+---------+ +---------+ +-------+
|
|
| httpsrc | | buffer | | demux |
|
|
| src - sink src - sink ....
|
|
+---------+ +---------+ +-------+
|
|
|
|
In this case we are reading from a slow network source into a buffer element
|
|
(such as queue2).
|
|
|
|
The buffer element has a low and high watermark expressed in bytes. The
|
|
buffer uses the watermarks as follows:
|
|
|
|
- The buffer element will post BUFFERING messages until the high watermark
|
|
is hit. This instructs the application to keep the pipeline PAUSED, which
|
|
will eventually block the srcpad from pushing while data is prerolled in
|
|
the sinks.
|
|
- When the high watermark is hit, a BUFFERING message with 100% will be
|
|
posted, which instructs the application to continue playback.
|
|
- When during playback, the low watermark is hit, the queue will start posting
|
|
BUFFERING messages again, making the application PAUSE the pipeline again
|
|
until the high watermark is hit again. This is called the rebuffering
|
|
stage.
|
|
- during playback, the queue level will fluctuate between the high and the
|
|
low watermark as a way to compensate for network irregularities.
|
|
|
|
This buffering method is usable when the demuxer operates in push mode.
|
|
Seeking in the stream requires the seek to happen in the network source.
|
|
It is mostly desirable when the total duration of the file is not know, such
|
|
as in live streaming or when efficient seeking is not possible/required.
|
|
|
|
* Incremental download
|
|
|
|
+---------+ +---------+ +-------+
|
|
| httpsrc | | buffer | | demux |
|
|
| src - sink src - sink ....
|
|
+---------+ +----|----+ +-------+
|
|
V
|
|
file
|
|
|
|
In this case, we know the server is streaming a fixed length file to the
|
|
client. The application can choose to download the file on disk. The buffer
|
|
element will provide a push or pull based srcpad to the demuxer to navigate in
|
|
the downloaded file.
|
|
|
|
This mode is only suitable when the client can determine the length of the
|
|
file on the server.
|
|
|
|
In this case, buffering messages will be emited as usual when the requested
|
|
range is not within the downloaded area + buffersize. The buffering message
|
|
will also contain an indication that incremental download is being performed.
|
|
This flag can be used to let the application control the buffering in a more
|
|
intelligent way, using the BUFFERING query, for example.
|
|
|
|
The application can use the BUFFERING query to get the estimated download time
|
|
and match this time to the current/remaining playback time to control when
|
|
playback should start to have a non-interupted playback experience.
|
|
|
|
|
|
* Timeshifting
|
|
|
|
+---------+ +---------+ +-------+
|
|
| httpsrc | | buffer | | demux |
|
|
| src - sink src - sink ....
|
|
+---------+ +----|----+ +-------+
|
|
V
|
|
file-ringbuffer
|
|
|
|
In this mode, a fixed size ringbuffer is kept to download the server content.
|
|
This allows for seeking in the buffered data. Depending on the size of the
|
|
buffer one can seek further back in time.
|
|
|
|
This mode is suitable for all live streams.
|
|
|
|
As with the incremental download mode, buffering messages are emited along
|
|
with an indication that timeshifting download is in progress.
|
|
|
|
|
|
* Live buffering
|
|
|
|
In live pipelines we usually introduce some latency between the capture and
|
|
the playback elements. This latency can be introduced by a queue (such as a
|
|
jitterbuffer) or by other means (in the audiosink).
|
|
|
|
Buffering messages can be emited in those live pipelines as well and serve as
|
|
an indication to the user of the latency buffering. The application usually
|
|
does not react to these buffering messages with a state change.
|
|
|
|
|
|
Messages
|
|
~~~~~~~~
|
|
|
|
A GST_MESSAGE_BUFFERING must be posted on the bus when playback temporarily
|
|
stops to buffer and when buffering finishes. When percentage field in the
|
|
BUFFERING message is 100, buffering is done. Values less than 100 mean that
|
|
buffering is in progress.
|
|
|
|
The BUFFERING message should be intercepted and acted upon by the application.
|
|
The message contains at least one field that is sufficient for basic
|
|
functionality:
|
|
|
|
"buffer-percent", G_TYPE_INT, between 0 and 100
|
|
|
|
Several more clever ways of dealing with the buffering messages can be used when
|
|
in incremental or timeshifting download mode. For this purpose additional fields
|
|
are added to the buffering message:
|
|
|
|
"buffering-mode", GST_TYPE_BUFFERING_MODE,
|
|
enum { "stream", "download", "timeshift", "live" }
|
|
- gives the buffering mode in use. See above for an explanation of the
|
|
different modes of buffering. This field can be used to let the
|
|
application have more control over the buffering process.
|
|
|
|
"avg-in-rate", G_TYPE_INT
|
|
- gives the average input buffering speed in bytes/second. -1 is unknown.
|
|
This is the average number of bytes per second that is received on the
|
|
buffering element input (sink) pads. It is a measurement of the network
|
|
speed in most cases.
|
|
|
|
"avg-out-rate", G_TYPE_INT
|
|
- gives the average consumption speed in bytes/second. -1 is unknown.
|
|
This is the average number of bytes per second that is consumed by the
|
|
downstream element of the buffering element.
|
|
|
|
"buffering-left", G_TYPE_INT64
|
|
- gives the estimated time that bufferring will take in milliseconds.
|
|
-1 unknown.
|
|
This is measured based on the avg-in-rate and the filled level of the
|
|
queue. The application can use this hint to update the GUI about the
|
|
estimated remaining time that buffering will take.
|
|
|
|
Application
|
|
~~~~~~~~~~~
|
|
|
|
While data is buffered, the pipeline should remain in the PAUSED state. It is
|
|
also possible that more data should be buffered while the pipeline is PLAYING,
|
|
in which case the pipeline should be PAUSED until the buffering finished.
|
|
|
|
BUFFERING messages can be posted while the pipeline is prerolling. The
|
|
application should not set the pipeline to PLAYING before a BUFFERING message
|
|
with 100 percent value is received, which might only happen after the pipeline
|
|
prerolled.
|
|
|
|
An exception is made for live pipelines. The application may not change
|
|
the state of a live pipeline when a buffering message is received. Usually these
|
|
buffering messages contain the "buffering-mode" = "live".
|
|
|
|
The buffering message can also instruct the application to switch to a periodical
|
|
BUFFERING query instead to more precisely control the buffering process. The
|
|
application can, for example, choose to not act on the BUFFERING message with
|
|
100 percent fill level to resume playback but instead use the estimated download
|
|
time to resume playback to get uninterrupted playback.
|
|
|
|
|
|
Buffering Query
|
|
~~~~~~~~~~~~~~~
|
|
|
|
In addition to the BUFFERING messages posted by the buffering elements we want
|
|
to be able to query the same information from the application. We also want to
|
|
be able to present the user with information about the downloaded range in the
|
|
file so that the GUI can react on it.
|
|
|
|
In addition to all the fields present in the buffering message, the BUFFERING
|
|
query contains the following field, which indicate the available downloaded
|
|
range in a specific format and the estimated time to complete:
|
|
|
|
"busy", G_TYPE_BOOLEAN
|
|
- if buffering was busy. This flag allows the application to pause the
|
|
pipeline by using the query only.
|
|
|
|
"format", GST_TYPE_FORMAT
|
|
- the format of the "start" and "stop" values below
|
|
|
|
"start", G_TYPE_INT64, -1 unknown
|
|
- the start position of the available data
|
|
|
|
"stop", G_TYPE_INT64, -1 unknown
|
|
- the stop position of the available data
|
|
|
|
"estimated-total", G_TYPE_INT64
|
|
- gives the estimated download time in milliseconds. -1 unknown.
|
|
|
|
When the size of the downloaded file is known, this value will contain
|
|
the latest estimate of the remaining download time. This value is usualy
|
|
only filled for the "download" buffering mode. The application can use
|
|
this information to estimate the amount of remaining time to download the
|
|
complete file.
|
|
|
|
For the "download" and "timeshift" buffering-modes, the start and stop positions
|
|
specify the ranges where efficient seeking in the downloaded media is possible.
|
|
Seeking outside of these ranges might be slow or not at all possible.
|
|
|
|
For the "stream" and "live" mode the start and stop values describe the oldest
|
|
and newest item (expressed in "format") in the buffer.
|
|
|
|
|
|
Defaults
|
|
~~~~~~~~
|
|
|
|
Some defaults for common elements:
|
|
|
|
A GstBaseSrc with random access replies to the BUFFERING query with:
|
|
|
|
"buffer-percent" = 100
|
|
"buffering-mode" = "stream"
|
|
"avg-in-rate" = -1
|
|
"avg-out-rate" = -1
|
|
"buffering-left" = 0
|
|
"format" = GST_FORMAT_BYTES
|
|
"start" = 0
|
|
"stop" = the total filesize
|
|
"estimated-total" = 0
|
|
|
|
A GstBaseSrc in push mode replies to the BUFFERING query with:
|
|
|
|
"buffer-percent" = 100
|
|
"buffering-mode" = "stream"
|
|
"avg-in-rate" = -1
|
|
"avg-out-rate" = -1
|
|
"buffering-left" = 0
|
|
"format" = a valid GST_TYPE_FORMAT
|
|
"start" = current position
|
|
"stop" = current position
|
|
"estimated-total" = -1
|
|
|
|
|
|
Buffering strategies
|
|
~~~~~~~~~~~~~~~~~~~~
|
|
|
|
Buffering strategies are specific implementations based on the buffering
|
|
message and query described above.
|
|
|
|
Most strategies have to balance buffering time versus maximal playback
|
|
experience.
|
|
|
|
* simple buffering
|
|
|
|
NON-live pipelines are kept in the paused state while buffering messages with
|
|
a percent < 100% are received.
|
|
|
|
This buffering strategy relies on the buffer size and low/high watermarks of
|
|
the element. It can work with a fixed size buffer in memory or on disk.
|
|
|
|
The size of the buffer is usually expressed in a fixed amount of time units
|
|
and the estimated bitrate of the upstream source is used to convert this time
|
|
to bytes.
|
|
|
|
All GStreamer applications must implement this strategy. Failure to do so
|
|
will result in starvation at the sink.
|
|
|
|
* no-rebuffer strategy
|
|
|
|
This strategy tries to buffer as much data as possible so that playback can
|
|
continue without any further rebuffering.
|
|
|
|
This strategy is initially similar to simple buffering, the difference is in
|
|
deciding on the condition to continue playback. When a 100% buffering message
|
|
has been received, the application will not yet start the playback but it will
|
|
start a periodic buffering query, which will return the estimated amount of
|
|
buffering time left. When the estimated time left is less than the remaining
|
|
playback time, playback can continue.
|
|
|
|
This strategy requires a unlimited buffer size in memory or on disk, such as
|
|
provided by elements that implement the incremental download buffering mode.
|
|
|
|
Usually, the application can choose to start playback even before the
|
|
remaining buffer time elapsed in order to more quickly start the playback at
|
|
the expense of a possible rebuffering phase.
|
|
|
|
* Incremental rebuffering
|
|
|
|
The application implements the simple buffering strategy but with each
|
|
rebuffering phase, it increases the size of the buffer.
|
|
|
|
This strategy has quick, fixed time startup times but incrementally longer
|
|
rebuffering times if the network is slower than the media bitrate.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|