mirror of
https://gitlab.freedesktop.org/gstreamer/gstreamer.git
synced 2024-12-16 13:26:36 +00:00
250 lines
10 KiB
Text
250 lines
10 KiB
Text
Frame step
|
|
----------
|
|
|
|
This document outlines the details of the frame stepping functionality in
|
|
GStreamer.
|
|
|
|
The stepping functionality operates on the current playback segment, position
|
|
and rate as it was configured with a regular seek event. In contrast to the seek
|
|
event, it operates very closely to the sink and thus has a very low latency and
|
|
is not slowed down by queues and does not actually perform any seeking logic.
|
|
For this reason we want to include a new API instead of reusing the seek API.
|
|
|
|
The following requirements are needed:
|
|
|
|
- The ability to walk forwards and backwards in the stream.
|
|
- Arbitrary increments in any supported format (time, frames, bytes ...)
|
|
- High speed, minimal overhead. This mechanism is not more expensive than
|
|
simple playback.
|
|
- switching between forwards and backwards stepping should be fast.
|
|
- Maintain synchronisation between streams.
|
|
- Get feedback of the amount of skipped data.
|
|
- Ability to play a certain amount of data at an arbitrary speed.
|
|
|
|
We want a system where we can step frames in PAUSED as well as play short
|
|
segments of data in PLAYING.
|
|
|
|
|
|
Use Cases
|
|
~~~~~~~~~
|
|
|
|
* frame stepping in video only pipeline in PAUSED
|
|
|
|
.-----. .-------. .------. .-------.
|
|
| src | | demux | .-----. | vdec | | vsink |
|
|
| src->sink src1->|queue|->sink src->sink |
|
|
'-----' '-------' '-----' '------' '-------'
|
|
|
|
- app sets the pipeline to PAUSED to block on the preroll picture
|
|
- app seeks to required position in the stream. This can be done with a
|
|
positive or negative rate depending on the required frame stepping
|
|
direction.
|
|
- app steps frames (in GST_FORMAT_DEFAULT or GST_FORMAT_BUFFER). The
|
|
pipeline loses its PAUSED state until the required number of frames have
|
|
been skipped, it then prerolls again. This skipping is purely done in
|
|
the sink.
|
|
- sink posts STEP_DONE with amount of frames stepped and corresponding time
|
|
interval.
|
|
|
|
* frame stepping in audio/video pipeline in PAUSED
|
|
|
|
.-----. .-------. .------. .-------.
|
|
| src | | demux | .-----. | vdec | | vsink |
|
|
| src->sink src1->|queue|->sink src->sink |
|
|
'-----' | | '-----' '------' '-------'
|
|
| | .------. .-------.
|
|
| | .-----. | adec | | asink |
|
|
| src2->|queue|->sink src->sink |
|
|
'-------' '-----' '------' '-------'
|
|
|
|
|
|
- app sets the pipeline to PAUSED to block on the preroll picture
|
|
- app seeks to required position in the stream. This can be done with a
|
|
positive or negative rate depending on the required frame stepping
|
|
direction.
|
|
- app steps frames (in GST_FORMAT_DEFAULT or GST_FORMAT_BUFFER) or an amount
|
|
of time on the video sink. The pipeline loses its PAUSED state until the
|
|
required number of frames have been skipped, it then prerolls again.
|
|
This skipping is purely done in the sink.
|
|
- sink posts STEP_DONE with amount of frames stepped and corresponding time
|
|
interval.
|
|
- the app skips the same amount of time on the audiosink to align the
|
|
streams again. When huge amount of video frames are skipped, there needs
|
|
to be enough queueing in the pipeline to compensate for the accumulated
|
|
audio.
|
|
|
|
* frame stepping in audio/video pipeline in PLAYING
|
|
|
|
- app sets the pipeline to PAUSED to block on the preroll picture
|
|
- app seeks to required position in the stream. This can be done with a
|
|
positive or negative rate depending on the required frame stepping
|
|
direction.
|
|
- app configures frames steps (in GST_FORMAT_DEFAULT or GST_FORMAT_BUFFER) or
|
|
an amount of time on the sink. The step event has a flag indicating live
|
|
stepping so that the stepping will only happens in PLAYING.
|
|
- app sets pipeline to PLAYING. The pipeline continues PLAYING until it
|
|
consumed the amount of time.
|
|
- sink posts STEP_DONE with amount of frames stepped and corresponding time
|
|
interval. The sink will then wait for another step event. Since the
|
|
STEP_DONE message was emitted by the sink when it handed off the buffer to
|
|
the device, there is usually sufficient time to queue a new STEP event so
|
|
that one can seamlessly continue stepping.
|
|
|
|
|
|
events
|
|
~~~~~~
|
|
|
|
A new GST_EVENT_STEP event is introduced to start the step operation.
|
|
The step event is created with the following fields in the structure:
|
|
|
|
"format", GST_TYPE_FORMAT
|
|
The format of the step units
|
|
|
|
"amount", G_TYPE_UINT64
|
|
The amount of units to step. A 0 amount immediately completes and can be
|
|
used to cancel the current step and resume normal non-stepping behaviour
|
|
to the end of the segment.
|
|
A -1 amount steps until the end of the segment.
|
|
|
|
"rate", G_TYPE_DOUBLE
|
|
The rate at which the frames should be stepped in PLAYING mode. 1.0 is
|
|
the normal playback speed and direction of the segment, 2.0
|
|
is double speed. A speed of 0.0 is not allowed. When performing a
|
|
flushing step, the speed is not relevant. Note that we don't allow negative
|
|
rates here, use a seek with a negative rate first to reverse the playback
|
|
direction.
|
|
|
|
"flush", G_TYPE_BOOLEAN
|
|
when flushing is TRUE, the step is performed immediately:
|
|
|
|
- In the PAUSED state the pipeline loses the PAUSED state, the requested
|
|
amount of data is skipped and the pipeline prerolls again when a
|
|
non-intermediate step completes.
|
|
When the pipeline was stepping while the event is sent, the current step
|
|
operation is updated with the new amount and format. The sink will do a
|
|
best effort to comply with the new amount.
|
|
- In the PLAYING state, the pipeline loses the PLAYING state, the
|
|
requested amount of data is skipped (not rendered) from the previous STEP
|
|
request or from the position of the last PAUSED if no previous STEP
|
|
operation was performed. The pipeline goes back to the PLAYING state
|
|
when a non-intermediate step completes.
|
|
|
|
When flushing is FALSE, the step will be performed later.
|
|
|
|
- In the PAUSED state the step will be done when going to PLAYING. Any
|
|
previous step operation will be overridden with the new STEP event.
|
|
- In the PLAYING state the step operation will be performed after the
|
|
current step operation completes. If there was no previous step
|
|
operation, the step operation will be performed from the position of the
|
|
last PAUSED state.
|
|
|
|
"intermediate", G_TYPE_BOOLEAN
|
|
Signal that this step operation is an intermediate step, part of a series
|
|
of step operations. It is mostly interesting for stepping in the PAUSED state
|
|
because the sink will only perform a preroll after a non-intermediate step
|
|
operation completes. Intermediate steps are useful to flush out data from
|
|
other sinks in order to not cause excessive queueing. In the PLAYING state
|
|
the intermediate flag has no visual effect. In all states, the intermediate
|
|
flag is passed to the corresponding GST_MESSAGE_STEP_DONE.
|
|
|
|
|
|
The application will create a STEP event to start or stop the stepping
|
|
operation. Both stepping in PAUSED and PLAYING can be performed by means of
|
|
the flush flag.
|
|
|
|
The event is usually sent to the pipeline, which will typically distribute the
|
|
event to all of its sinks. For some use cases, like frame stepping on video
|
|
frames only, the event should only be sent to the video sink and upon reception
|
|
of the STEP_DONE message, one can step the other sinks to align the streams
|
|
again.
|
|
|
|
For large stepping amounts, there needs to be enough queueing in front of all
|
|
the sinks. If large steps need to be performed, they can be split up into
|
|
smaller step operations using the "intermediate" flag on the step.
|
|
|
|
Since the step event does not update the base_time of any of the elements, the
|
|
sinks should keep track of the amount of stepped data in order to remain
|
|
synchronized against the clock.
|
|
|
|
|
|
messages
|
|
~~~~~~~~
|
|
|
|
A GST_MESSAGE_STEP_START is created. It contains the following fields.
|
|
|
|
"active"
|
|
If the step was queued or activated.
|
|
|
|
"format", GST_TYPE_FORMAT
|
|
The format of the step units that queued/activated.
|
|
|
|
"amount", G_TYPE_UINT64
|
|
The amount of units that were queued/activated.
|
|
|
|
"rate", G_TYPE_DOUBLE
|
|
The rate and direction at which the frames were queued/activated.
|
|
|
|
"flush", G_TYPE_BOOLEAN
|
|
If the queued/activated frames will be flushed.
|
|
|
|
"intermediate", G_TYPE_BOOLEAN
|
|
If this is an intermediate step operation that queued/activated.
|
|
|
|
The STEP_START message is emitted 2 times:
|
|
|
|
* first when an element received the STEP event and queued it. The "active"
|
|
field will be FALSE in this case.
|
|
|
|
* second when the step operation started in the streaming thread. The "active"
|
|
field is TRUE in this case. After this message is emitted, the application
|
|
can queue a new step operation.
|
|
|
|
The purpose of this message is to find out how many elements participate in the
|
|
step operation and to queue new step operations at the earliest possible
|
|
moment.
|
|
|
|
A new GST_MESSAGE_STEP_DONE message is created. It contains the following
|
|
fields:
|
|
|
|
"format", GST_TYPE_FORMAT
|
|
The format of the step units that completed.
|
|
|
|
"amount", G_TYPE_UINT64
|
|
The amount of units that were stepped.
|
|
|
|
"rate", G_TYPE_DOUBLE
|
|
The rate and direction at which the frames were stepped.
|
|
|
|
"flush", G_TYPE_BOOLEAN
|
|
If the stepped frames were flushed.
|
|
|
|
"intermediate", G_TYPE_BOOLEAN
|
|
If this is an intermediate step operation that completed.
|
|
|
|
"duration", G_TYPE_UINT64
|
|
The total duration of the stepped units in GST_FORMAT_TIME.
|
|
|
|
"eos", G_TYPE_BOOLEAN
|
|
The step ended because of EOS.
|
|
|
|
The message is emitted by the element that performs the step operation. The
|
|
purpose is to return the duration in GST_FORMAT_TIME of the stepped media. This
|
|
especially interesting to align other stream in case of stepping frames on the
|
|
video sink element.
|
|
|
|
|
|
Direction switch
|
|
~~~~~~~~~~~~~~~~
|
|
|
|
When quickly switching between a forwards and a backwards step of, for example,
|
|
one video frame, we need either:
|
|
|
|
a) issue a new seek to change the direction from the current position.
|
|
b) cache a certain number of stepped frames and walk the cache.
|
|
|
|
option a) might be very slow.
|
|
For option b) we would ideally like to offload this caching functionality to a
|
|
separate element, which means that we need to forward the STEP event upstream.
|
|
It's unclear how this could work in a generic way. What is a demuxer supposed
|
|
to do when it received a step event? a flushing seek to what stream position?
|
|
|