gstreamer/markdown/design/buffering.md

# Buffering

This document outlines the buffering policy used in the GStreamer core
that can be used by plugins and applications.

The purpose of buffering is to accumulate enough data in a pipeline so
that playback can occur smoothly and without interruptions. It is
typically done when reading from a (slow) non-live network source but
can also be used for live sources.

We want to be able to implement the following features:

- buffering up to a specific amount of data, in memory, before
starting playback so that network fluctuations are minimized.

- download of the network file to a local disk with fast seeking in
the downloaded data. This is similar to the quicktime/youtube
players.

- caching of semi-live streams to a local, on disk, ringbuffer with
seeking in the cached area. This is similar to tivo-like
timeshifting.

- progress report about the buffering operations

- the possibility for the application to do more complex buffering

## Some use cases

### Stream buffering

```
+---------+     +---------+     +-------+
| httpsrc |     | buffer  |     | demux |
|        src - sink      src - sink     ....
+---------+     +---------+     +-------+
```

In this case we are reading from a slow network source into a buffer element
(such as queue2).

The buffer element has a low and high watermark expressed in bytes. The
buffer uses the watermarks as follows:

- The buffer element will post `BUFFERING` messages until the high
watermark is hit. This instructs the application to keep the
pipeline `PAUSED`, which will eventually block the srcpad from
pushing while data is prerolled in the sinks.

- When the high watermark is hit, a `BUFFERING` message with 100%
will be posted, which instructs the application to continue
playback.

- When the low watermark is hit during playback, the queue will
start posting `BUFFERING` messages again, making the application
PAUSE the pipeline again until the high watermark is hit again.
This is called the rebuffering stage.

- During playback, the queue level will fluctuate between the high
and low watermarks as a way to compensate for network
irregularities.

This buffering method is usable when the demuxer operates in push mode.
Seeking in the stream requires the seek to happen in the network source.
It is mostly desirable when the total duration of the file is not known, such
as in live streaming or when efficient seeking is not possible/required.

### Incremental download

```
+---------+     +---------+     +-------+
| httpsrc |     | buffer  |     | demux |
|        src - sink      src - sink     ....
+---------+     +----|----+     +-------+
                     V
                    file
```
In this case, we know the server is streaming a fixed length file to the
client. The application can choose to download the file to disk. The buffer
element will provide a push or pull based srcpad to the demuxer to navigate in
the downloaded file.

This mode is only suitable when the client can determine the length of the
file on the server.

In this case, buffering messages will be emitted as usual when the requested
range is not within the downloaded area + buffersize. The buffering message
will also contain an indication that incremental download is being performed.
This flag can be used to let the application control the buffering in a more
intelligent way, using the `BUFFERING` query, for example.

The application can use the `BUFFERING` query to get the estimated download time
and match this time to the current/remaining playback time to control when
playback should start to have a non-interrupted playback experience.

### Timeshifting

```
+---------+     +---------+     +-------+
| httpsrc |     | buffer  |     | demux |
|        src - sink      src - sink     ....
+---------+     +----|----+     +-------+
                     V
              file-ringbuffer
```

In this mode, a fixed size ringbuffer is kept to download the server content.
This allows for seeking in the buffered data. Depending on the size of the
buffer one can seek further back in time.

This mode is suitable for all live streams.

As with the incremental download mode, buffering messages are emitted along
with an indication that timeshifting download is in progress.

### Live buffering

In live pipelines we usually introduce some latency between the capture and
the playback elements. This latency can be introduced by a queue (such as a
jitterbuffer) or by other means (in the audiosink).

Buffering messages can be emitted in those live pipelines as well and serve as
an indication to the user of the latency buffering. The application usually
does not react to these buffering messages with a state change.

## Messages

A `GST_MESSAGE_BUFFERING` must be posted on the bus when playback
temporarily stops to buffer and when buffering finishes. When the
percentage field in the `BUFFERING` message is 100, buffering is done.
Values less than 100 mean that buffering is in progress.

The `BUFFERING` message should be intercepted and acted upon by the
application. The message contains at least one field that is sufficient
for basic functionality:

* **`buffer-percent`**, `G_TYPE_INT`: between 0 and 100

Several more clever ways of dealing with the buffering messages can be
used when in incremental or timeshifting download mode. For this purpose
additional fields are added to the buffering message:

* **`buffering-mode`**, `GST_TYPE_BUFFERING_MODE`: `enum { "stream", "download",
"timeshift", "live" }`: Buffering mode in use. See above for an explanation of the different
alternatives. This field can be used to let the application have more control
over the buffering process.

* **`avg-in-rate`**, `G_TYPE_INT`: Average input buffering speed in bytes/second.
-1 is unknown. This is the average number of bytes per second that is received
on the buffering element input (sink) pads. It is a measurement of the network
speed in most cases.

* **`avg-out-rate`**, `G_TYPE_INT`: Average consumption speed in bytes/second. -1
is unknown. This is the average number of bytes per second that is consumed by
the downstream element of the buffering element.

* **`buffering-left`**, `G_TYPE_INT64`: Estimated time that buffering will take
in milliseconds. -1 is unknown. This is measured based on the avg-in-rate and
the filled level of the queue. The application can use this hint to update the
GUI about the estimated remaining time that buffering will take.

## Application

While data is buffered the pipeline should remain in the `PAUSED` state.
It is also possible that more data should be buffered while the pipeline
is `PLAYING`, in which case the pipeline should be `PAUSED` until the
buffering finishes.

`BUFFERING` messages can be posted while the pipeline is prerolling. The
application should not set the pipeline to `PLAYING` before a `BUFFERING`
message with a 100 percent value is received, which might only happen
after the pipeline prerolls.

An exception is made for live pipelines. The application may not change
the state of a live pipeline when a buffering message is received.
Usually these buffering messages contain the "buffering-mode" = "live".

The buffering message can also instruct the application to switch to a
periodical `BUFFERING` query instead, so it can more precisely control the
buffering process. The application can, for example, choose not to act
on the `BUFFERING` complete message (`buffer-percent = 100`) to resume
playback but use the estimated download time instead, resuming playback
when it has determined that it should be able to provide uninterrupted
playback.

## Buffering Query

In addition to the `BUFFERING` messages posted by the buffering elements,
we want to be able to query the same information from the application.
We also want to be able to present the user with information about the
downloaded range in the file so that the GUI can react on it.

In addition to all the fields present in the buffering message, the
`BUFFERING` query contains the following field, which indicates the
available downloaded range in a specific format and the estimated time
to complete:

* **`busy`**, `G_TYPE_BOOLEAN`: if buffering was busy. This flag allows the
application to pause the pipeline by using the query only.

* **`format`**, `GST_TYPE_FORMAT`: the format of the "start" and "stop" values
below

* **`start`**, `G_TYPE_INT64`, -1 unknown: the start position of the available
data. If there are multiple ranges, this field contains the start position of
the currently downloading range.

* **`stop`**, `G_TYPE_INT64`, -1 unknown: the stop position of the available
data. If there are multiple ranges, this field contains the stop position of
the currently downloading range.

* **`estimated-total`**, `G_TYPE_INT64`: gives the estimated download time in
milliseconds. -1 unknown. When the size of the downloaded file is known, this
value will contain the latest estimate of the remaining download time of the
currently downloading range. This value is usually only filled for the
"download" buffering mode. The application can use this information to estimate
the amount of remaining time to download till the end of the file.

* **`buffering-ranges`**, `G_TYPE_ARRAY` of `GstQueryBufferingRange`: contains
optionally the downloaded areas in the format given above. One of the ranges
contains the same start/stop position as above:

```
typedef struct
{
  gint64 start;
  gint64 stop;
} GstQueryBufferingRange;
```

For the `download` and `timeshift` buffering-modes, the start and stop
positions specify the ranges where efficient seeking in the downloaded
media is possible. Seeking outside of these ranges might be slow or not
at all possible.

For the `stream` and `live` mode the start and stop values describe the
oldest and newest item (expressed in `format`) in the buffer.

## Defaults

Some defaults for common elements:

A `GstBaseSrc` with random access replies to the `BUFFERING` query with:

```
"buffer-percent" = 100
"buffering-mode" = "stream"
"avg-in-rate" = -1
"avg-out-rate" = -1
"buffering-left" = 0
"format" = GST_FORMAT_BYTES
"start" = 0
"stop" = the total filesize
"estimated-total" = 0
"buffering-ranges" = NULL
```

A `GstBaseSrc` in push mode replies to the `BUFFERING` query with:

```
"buffer-percent" = 100
"buffering-mode" = "stream"
"avg-in-rate" = -1
"avg-out-rate" = -1
"buffering-left" = 0
"format" = a valid GST_TYPE_FORMAT
"start" = current position
"stop" = current position
"estimated-total" = -1
"buffering-ranges" = NULL
```

## Buffering strategies

Buffering strategies are specific implementations based on the buffering
message and query described above.

Most strategies have to balance buffering time versus maximal playback
experience.

### Simple buffering

NON-live pipelines are kept in the paused state while buffering messages with
a percent < 100% are received.

This buffering strategy relies on the buffer size and low/high watermarks of
the element. It can work with a fixed size buffer in memory or on disk.

The size of the buffer is usually expressed in a fixed amount of time units
and the estimated bitrate of the upstream source is used to convert this time
to bytes.

All GStreamer applications must implement this strategy. Failure to do so
will result in starvation at the sink.

### No-rebuffer strategy

This strategy tries to buffer as much data as possible so that playback can
continue without any further rebuffering.

This strategy is initially similar to simple buffering, the difference is in
deciding on the condition to continue playback. When a 100% buffering message
has been received, the application will not yet start the playback but it will
start a periodic buffering query, which will return the estimated amount of
buffering time left. When the estimated time left is less than the remaining
playback time, playback can continue.

This strategy requires a unlimited buffer size in memory or on disk, such as
provided by elements that implement the incremental download buffering mode.

Usually, the application can choose to start playback even before the
remaining buffer time elapsed in order to more quickly start the playback at
the expense of a possible rebuffering phase.

### Incremental rebuffering

The application implements the simple buffering strategy but with each
rebuffering phase, it increases the size of the buffer.

This strategy has quick, fixed time startup times but incrementally longer
rebuffering times if the network is slower than the media bitrate.