mirror of
https://gitlab.freedesktop.org/gstreamer/gstreamer.git
synced 2024-12-20 23:36:38 +00:00
216 lines
9 KiB
Text
216 lines
9 KiB
Text
Trickmodes
|
|
----------
|
|
|
|
GStreamer provides API for performing various trickmode playback. This includes:
|
|
|
|
- server side trickmodes
|
|
- client side fast/slow forward playback
|
|
- client side fast/slow backwards playback
|
|
|
|
Server side trickmodes mean that a source (network source) can provide a
|
|
stream with different playback speed and direction. The client does not have to
|
|
perform any special algorithms to decode this stream.
|
|
|
|
Client side trickmodes mean that the decoding client (GStreamer) performs the
|
|
needed algorithms to change the direction and speed of the media file.
|
|
|
|
Seeking can both be done in a playback pipeline and a transcoding pipeline.
|
|
|
|
|
|
General seeking overview
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
Consider a typical playback pipeline:
|
|
|
|
.---------. .------.
|
|
.-------. | decoder |->| sink |
|
|
.--------. | |-->'---------' '------'
|
|
| source |->| demux |
|
|
'--------' | |-->.---------. .------.
|
|
'-------' | decoder |->| sink |
|
|
'---------' '------'
|
|
|
|
The pipeline is initially configured to play back at speed 1.0 starting from
|
|
position 0 and stopping at the total duration of the file.
|
|
|
|
When performing a seek, the following steps have to be taken by the application:
|
|
|
|
Create a seek event
|
|
^^^^^^^^^^^^^^^^^^^
|
|
|
|
The seek event contains:
|
|
|
|
- various flags describing:
|
|
- where to seek to (KEY_UNIT)
|
|
- how accurate the seek should be (ACCURATE)
|
|
- how to perform the seek (FLUSH)
|
|
- what to do when the stop position is reached (SEGMENT).
|
|
- extra playback options (SKIP)
|
|
- a format to seek in, this can be time, bytes, units (frames, samples), ...
|
|
- a playback rate, 1.0 is normal playback speed, positive values bigger than 1.0
|
|
mean fast playback. negative values mean reverse playback. A playback speed of
|
|
0.0 is not allowed (but is equivalent to PAUSING the pipeline).
|
|
- a start position, this value has to be between 0 and the total duration of the
|
|
file. It can also be relative to the previously configured start value.
|
|
- a stop position, this value has to be between 0 and the total duration. It can
|
|
also be relative to the previously configured stop value.
|
|
|
|
See also gst_event_new_seek().
|
|
|
|
|
|
Send the seek event
|
|
^^^^^^^^^^^^^^^^^^^
|
|
|
|
Send the new seek event to the pipeline with gst_element_send_event().
|
|
|
|
By default the pipeline will send the event to all sink elements.
|
|
By default an element will forward the event upstream on all sinkpads.
|
|
Elements can modify the format of the seek event. The most common format is
|
|
GST_FORMAT_TIME.
|
|
|
|
One element will actually perform the seek, this is usually the demuxer or
|
|
source element. For more information on how to perform the different seek
|
|
types see part-seeking.txt.
|
|
|
|
For client side trickmode a SEGMENT event will be sent downstream with
|
|
the new rate and start/stop positions. All elements prepare themselves to
|
|
handle the rate (see below). The applied rate of the SEGMENT event will
|
|
be set to 1.0 to indicate that no rate adjustment has been done.
|
|
|
|
for server side trick mode a SEGMENT event is sent downstream with a
|
|
rate of 1.0 and the start/stop positions. The elements will configure themselves
|
|
for normal playback speed since the server will perform the rate conversions.
|
|
The applied rate will be set to the rate that will be applied by the server. This
|
|
is done to insure that the position reporting performed in the sink is aware
|
|
of the trick mode.
|
|
|
|
When the seek succeeds, the _send_event() function will return TRUE.
|
|
|
|
|
|
Server side trickmode
|
|
~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
The source element operates in push mode. It can reopen a server connection requesting
|
|
a new byte or time position and a new playback speed. The capabilities can be queried
|
|
from the server when the connection is opened.
|
|
|
|
We assume the source element is derived from the GstPushSrc base class. The base source
|
|
should be configured with gst_base_src_set_format (src, GST_FORMAT_TIME).
|
|
|
|
The do_seek method will be called on the push src subclass with the seek information
|
|
passed in the GstSegment argument.
|
|
|
|
The rate value in the segment should be used to reopen the connection to the server
|
|
requesting data at the new speed and possibly a new playback position.
|
|
|
|
When the server connection was successfully reopened, set the rate of the segment
|
|
to 1.0 so that the client side trickmode is not enabled. The applied rate in the
|
|
segment is set to the rate transformation done by the server.
|
|
|
|
Alternatively a combination of client side and serverside trickmode can be used, for
|
|
example if the server does not support certain rates, the client can perform rate
|
|
conversion for the remainder.
|
|
|
|
|
|
source server
|
|
do_seek | |
|
|
----------->| |
|
|
| reopen connection |
|
|
|-------------------->|
|
|
| .
|
|
| success .
|
|
|<--------------------|
|
|
modify | |
|
|
rate to 1.0 | |
|
|
| |
|
|
return | |
|
|
TRUE | |
|
|
| |
|
|
|
|
After performing the seek, the source will inform the downstream elements of the
|
|
new segment that is to be played back. Since the segment will have a rate of 1.0,
|
|
no client side trick modes are enabled. The segment will have an applied rate
|
|
different from 1.0 to indicate that the media contains data with non-standard
|
|
playback speed or direction.
|
|
|
|
|
|
client side forward trickmodes
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
The seek happens as stated above. a SEGMENT event is sent downstream with a rate
|
|
different from 1.0. Plugins receiving the SEGMENT can decide to perform the
|
|
rate conversion of the media data (retimestamp video frames, resample audio, ...).
|
|
|
|
If a plugin decides to resample or retimestamp, it should modify the SEGMENT with
|
|
a rate of 1.0 and update the applied rate so that downstream elements don't resample
|
|
again but are aware that the media has been modified.
|
|
|
|
The GStreamer base audio and video sinks will resample automatically if they receive
|
|
a SEGMENT event with a rate different from 1.0. The position reporting in the
|
|
base audio and video sinks will also depend on the applied rate of the segment
|
|
information.
|
|
|
|
When the SKIP flag is set, frames can be dropped in the elements. If S is the
|
|
speedup factor, a good algorithm for implementing frame skipping is to send audio in
|
|
chunks of Nms (usualy 300ms is good) and then skip ((S-1) * Nns) of audio data.
|
|
For the video we send only the keyframes in the (S * Nns) interval. In this
|
|
case, the demuxer would scale the timestamps and would set an applied rate of S.
|
|
|
|
|
|
client side backwards trickmode
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
For backwards playback the following rules apply:
|
|
|
|
- the rate in the SEGMENT is less than 0.0.
|
|
- the SEGMENT start position is less than the stop position, playback will
|
|
however happen from stop to start in reverse.
|
|
- the time member in the SEGMENT is set to the stream time of the start
|
|
position.
|
|
|
|
For plugins the following rules apply:
|
|
|
|
- A source plugin sends data in chunks starting from the last chunk of the
|
|
file. The actual bytes are not reversed. Each chunk that is not forward
|
|
continuous with the previous chunk is marked with a DISCONT flag.
|
|
|
|
- A demuxer accumulates the chunks. As soon as a keyframe is found, everything
|
|
starting from the keyframe up to the accumulated data is sent downstream.
|
|
Timestamps on the buffers are set starting from the stop position to start,
|
|
effectively going backwards. Chunks are marked with DISCONT when they are not
|
|
forward continuous with the previous buffer.
|
|
|
|
- A video decoder decodes and accumulates all decoded frames. If a buffer with
|
|
a DISCONT, SEGMENT or EOS is received, all accumulated frames are sent
|
|
downsteam in reverse.
|
|
|
|
- An audio decoder decodes and accumulates all decoded audio. If a buffer with
|
|
a DISCONT, SEGMENT or EOS is received, all accumulated audio is sent
|
|
downstream in reverse order. Some audio codecs need the previous
|
|
data buffer to decode the current one, in that case, the previous DISCONT
|
|
buffer needs to be combined with the last non-DISCONT buffer to generate the
|
|
last bit of output.
|
|
|
|
- A sink reverses (for audio) and retimestamps (audio, video) the buffers
|
|
before playing them back. Retimestamping occurs relative to the stop
|
|
position, making the timestamps increase again and suitable for synchronizing
|
|
against the clock.
|
|
Audio sinks also have to perform simple resampling before playing the
|
|
samples.
|
|
|
|
- for transcoding, audio and video resamplers can be used to reverse, resample
|
|
and retimestamp the buffers. Any rate adjustments performed on the media must
|
|
be added to the applied_rate and subtracted from the rate members in the
|
|
SEGMENT event.
|
|
|
|
In SKIP mode, the same algorithm as for forward SKIP mode can be used.
|
|
|
|
|
|
Notes
|
|
~~~~~
|
|
|
|
- The clock/running_time keeps running forward.
|
|
- backwards playback potentially uses a lot of memory as frames and undecoded
|
|
data gets buffered.
|
|
|
|
|