From 878e4d920469561a6863fcf0dfee9b89e173970a Mon Sep 17 00:00:00 2001 From: Wim Taymans Date: Fri, 4 Apr 2008 13:43:26 +0000 Subject: [PATCH] docs/design/draft-latency.txt: Fix typo. Original commit message from CVS: * docs/design/draft-latency.txt: Fix typo. * docs/design/part-buffering.txt: Update design docs with more buffering ideas. --- ChangeLog | 8 ++ docs/design/draft-latency.txt | 2 +- docs/design/part-buffering.txt | 216 +++++++++++++++++++++++++++++---- 3 files changed, 200 insertions(+), 26 deletions(-) diff --git a/ChangeLog b/ChangeLog index 4e227c9367..baa179d174 100644 --- a/ChangeLog +++ b/ChangeLog @@ -1,3 +1,11 @@ +2008-04-04 Wim Taymans + + * docs/design/draft-latency.txt: + Fix typo. + + * docs/design/part-buffering.txt: + Update design docs with more buffering ideas. + 2008-04-03 Tim-Philipp Müller * configure.ac: diff --git a/docs/design/draft-latency.txt b/docs/design/draft-latency.txt index 424bfef66f..ce6f97108d 100644 --- a/docs/design/draft-latency.txt +++ b/docs/design/draft-latency.txt @@ -293,7 +293,7 @@ A flush in a pipeline can happen in the following cases: When a playing sink is flushed by a FLUSH_START event, an ASYNC_START message is posted by the element. As part of the message, the fact that the element got -flushes is included. The element also goes to a pending PAUSED state and has to +flushed is included. The element also goes to a pending PAUSED state and has to be set to the PLAYING state again later. The ASYNC_START message is kept by the parent bin. When the element prerolls, diff --git a/docs/design/part-buffering.txt b/docs/design/part-buffering.txt index 10f093ef96..2fda77cf67 100644 --- a/docs/design/part-buffering.txt +++ b/docs/design/part-buffering.txt @@ -6,26 +6,109 @@ core that can be used by plugins and applications. The purpose of buffering is to accumulate enough data in a pipeline so that playback can occur smoothly and without interruptions. It is typically done -when reading from a (slow) and non-live network source. +when reading from a (slow) and non-live network source but can also be used for +live sources. + +We want to be able to implement the following features: + + - buffering up to a specifc amount of data, in memory, before starting playback + so that network fluctuations are minimized. + - download of the network file to a local disk with fast seeking in the + downloaded data. This is similar to the quicktime/youtube players. + - caching of semi-live streams to a local, on disk, ringbuffer with seeking in + the cached area. This is similar to tivo-like timeshifting. + - progress report about the buffering operations + - easy (backward compatible) application notification of buffering + - the possibility for the application to do more complex buffering Some use cases: - +---------+ +--------+ +-------+ - | httpsrc | | buffer | | demux | - | src - sink src - sink .... - +---------+ +--------+ +-------+ + * Stream buffering: + + +---------+ +---------+ +-------+ + | httpsrc | | buffer | | demux | + | src - sink src - sink .... + +---------+ +---------+ +-------+ In this case we are reading from a slow network source into a buffer element - (such as queue). + (such as queue2). The buffer element has a low and high watermark expressed in bytes. The buffer uses the watermarks as follows: - - The buffer element will not produce data on the src pad until the high - watermark is hit. While accumulating data in the buffer, progress is - reported by posting BUFFERING messages. - - The source will stop to produce data on the src pad when the low watermark - is hit. + - The buffer element will post BUFFERING messages until the high watermark + is hit. This instructs the application to keep the pipeline PAUSED, which + will eventually block the srcpad from pushing while data is prerolled in + the sinks. + - When the high watermark is hit, a BUFFERING message with 100% will be + posted, which instructs the application to continue playback. + - When during playback, the low watermark is hit, the queue will start posting + BUFFERING messages again, making the application PAUSE the pipeline again + until the high watermark is hit again. + - during playback, the queue level will fluctuate between the high and the + low watermark as a way to compensate for network irregularities. + + This buffering method is usable when the demuxer operates in push mode. + Seeking in the stream requires the seek to happen in the network source. + It is mostly desirable when the total duration of the file is not know, such + as in live streaming or when efficient seeking is not possible/required. + + * Incremental download + + +---------+ +---------+ +-------+ + | httpsrc | | buffer | | demux | + | src - sink src - sink .... + +---------+ +----|----+ +-------+ + V + file + + In this case, we know the server is streaming a fixed length file to the + client. The application can choose to download the file on disk. The buffer + element will provide a push or pull based srcpad to the demuxer to navigate in + the downloaded file. + + This mode is only suitable when the client can determine the length of the + file on the server. + + In this case, buffering messages will be emited as usual when the requested + range is not within the downloaded area + buffersize. The buffering message + will also contain an indication that incremental download is being performed. + This flag can be used to let the application control the buffering in a more + intelligent way, using the BUFFERING query, for example. + + The application can use the BUFFERING query to get the estimated download time + and match this time to the current/remaining playback time to control when + playback should start to have a non-interupted playback experience. + + + * Timeshifting + + +---------+ +---------+ +-------+ + | httpsrc | | buffer | | demux | + | src - sink src - sink .... + +---------+ +----|----+ +-------+ + V + file-ringbuffer + + In this mode, a fixed size ringbuffer is kept to download the server content. + This allows for seeking in the buffered data. Depending on the size of the + buffer one can seek further back in time. + + This mode is suitable for all live streams. + + As with the incremental download mode, buffering messages are emited along + with an indication that timeshifting download is in progress. + + + * Live buffering + + In live pipelines we usually introduce some latency between the capture and + the playback elements. This latency can be introduced by a queue (such as a + jitterbuffer) or by other means (in the audiosink). + + Buffering messages can be emited in those live pipelines as well and serve as + an indication to the user of the latency buffering. The application usually + does not react to these buffering messages with a state change. Messages @@ -37,6 +120,46 @@ BUFFERING message is 100, buffering is done. Values less than 100 mean that buffering is in progress. The BUFFERING message should be intercepted and acted upon by the application. +The message contains at least one field that is sufficient for basic +functionality: + + "buffer-percent", G_TYPE_INT, between 0 and 100 + +Several more clever ways of dealing with the buffering messages can be used when +in incremental or timeshifting download mode. For this purpose additional fields +are added to the buffering message: + + "buffering-mode", GST_TYPE_BUFFERING_MODE, + enum { "none", "stream", "download", "timeshift", "live" } + - gives the buffering mode in use. See above for an explanation of the + different modes of buffering. + + "avg-in-rate", G_TYPE_INT + - gives the average input buffering speed in bytes/second. -1 is unknown. + + This is the average number of bytes per second that is received on the + buffering element input (sink) pads. If is a measurement of the network + speed in most cases. + + "avg-out-rate", G_TYPE_INT + - gives the average consumption speed in bytes/second. -1 is unknown. + + This is the average number of bytes per second that is consumed by the + downstream element of the buffering element. + + "buffering-left", G_TYPE_INT + - gives the estimated time that bufferring will take in milliseconds. + -1 unknown. + + This is measured based on the avg-in-rate and the filled level of the + queue. The application can use this hint to update the gui while buffering. + + "estimated-time", G_TYPE_INT + - gives the estimated download time in milliseconds. -1 unknown. + + When the size of the downloaded file is known, this value will contain + the latest estimate of the remaining download time. This value is usualy + only filled for the "download" buffering mode. Application @@ -51,29 +174,72 @@ application should not set the pipeline to PLAYING before a BUFFERING message with 100 percent value is received, which might only happen after the pipeline prerolled. +An exception is made for live pipelines. The application may not change +the state of a live pipeline when a buffering message is received. Usually these +buffering messages contain the "buffering-mode" = "live". + +The buffering message can also instruct the application to switch to a periodical +BUFFERING query instead to more precisely control the buffering process. The +application can, for example, choose to not act on the BUFFERING message with +100 percent fill level to resume playback but instead use the estimated download +time to resume playback to get uninterrupted playback. + Buffering Query --------------- -It is possible to query the amount of buffering performed in the pipeline, which -is defined as the amount of data made available at the source. This amount is -expressed in some GstFormat and is usually compared to the duration or position -of the media stream in the same GstFormat. +In addition to the BUFFERING messages posted by the buffering elements we want +to be able to query the same information from the application. We also want to +be able to present the user with information about the downloaded range in the +file so that the GUI can react on it. -The buffering query should return the following information: +In addition to all the fields present in the buffering message, the BUFFERING +query contains the following field, which indicate the available downloaded +range in a specific format: - - format - - position - - duration + "format", GST_TYPE_FORMAT + - the format of the "start" and "stop" values below + + "start", G_TYPE_INT64, -1 unknown + - the start position of the available data -The format is of lesser importance, the ratio of position versus duration can be -used to calculate the percentage of available media. It should also be possible -for the application to calculate the expected time when the complete file will -be buffered. + "stop", G_TYPE_INT64, -1 unknown + - the stop position of the available data + +For the "download" and "timeshift" buffering-modes, the start and stop positions +specify the ranges where efficient seeking in the downloaded media is possible. +Seeking outside of these ranges might be slow or not at all possible. + +For the "stream" and "live" mode the start and stop values describe the oldest +and newest item (expressed in "format") in the buffer. +Defaults +-------- -Incremental download --------------------- +Some defaults for common elements: +A GstBaseSrc with random access replies to the BUFFERING query with: + + "buffer-percent" = 100 + "buffering-mode" = "none" + "avg-in-rate" = -1 + "avg-out-rate" = -1 + "buffering-left" = 0 + "estimated-time" = 0 + "format" = GST_FORMAT_BYTES + "start" = 0 + "stop" = the total filesize + +A GstBaseSrc in push mode replies to the BUFFERING query with: + + "buffer-percent" = 100 + "buffering-mode" = "none" + "avg-in-rate" = -1 + "avg-out-rate" = -1 + "buffering-left" = 0 + "estimated-time" = 0 + "format" = a valid GST_TYPE_FORMAT + "start" = current position + "stop" = current position