mirror of
https://gitlab.freedesktop.org/gstreamer/gstreamer.git
synced 2024-12-21 07:46:38 +00:00
25067558dc
Update design doc with step-start docs. Add eos field to step done message when stepping in reverse, update the segment time field. Flush out the current step when we are flushing.
251 lines
10 KiB
Text
251 lines
10 KiB
Text
Frame step
|
|
----------
|
|
|
|
This document is a draft that lists some ideas and purely informational.
|
|
|
|
This document outlines the details of the frame stepping functionality in
|
|
GStreamer.
|
|
|
|
The stepping functionality operates on the current playback segment, position
|
|
and rate as it was configured with a regular seek event. In contrast to the seek
|
|
event, it operates very closely to the sink and thus has a very low latency and
|
|
is not slowed down by queues and does not actually perform any seeking logic.
|
|
For this reason we want to include a new API instead of reusing the seek API.
|
|
|
|
The following requirements are needed:
|
|
|
|
- The ability to walk forwards and backwards in the stream.
|
|
- Arbitrary increments in any supported format (time, frames, bytes ...)
|
|
- High speed, minimal overhead. This mechanism is not more expensive than
|
|
simple playback.
|
|
- switching between forwards and backwards stepping should be fast.
|
|
- Maintain synchronisation between streams.
|
|
- Get feedback of the amount of skipped data.
|
|
- Ability to play a certain amount of data at an arbitrary speed.
|
|
|
|
We want a system where we can step frames in PAUSED as well as play short
|
|
segments of data in PLAYING.
|
|
|
|
|
|
Use Cases
|
|
---------
|
|
|
|
* frame stepping in video only pipeline in PAUSED
|
|
|
|
.-----. .-------. .------. .-------.
|
|
| src | | demux | .-----. | vdec | | vsink |
|
|
| src->sink src1->|queue|->sink src->sink |
|
|
'-----' '-------' '-----' '------' '-------'
|
|
|
|
- app sets the pipeline to PAUSED to block on the preroll picture
|
|
- app seeks to required position in the stream. This can be done with a
|
|
positive or negative rate depending on the required frame stepping
|
|
direction.
|
|
- app steps frames (in GST_FORMAT_DEFAULT or GST_FORMAT_BUFFER). The
|
|
pipeline loses its PAUSED state until the required number of frames have
|
|
been skipped, it then prerolls again. This skipping is purely done in
|
|
the sink.
|
|
- sink posts STEP_DONE with amount of frames stepped and corresponding time
|
|
interval.
|
|
|
|
* frame stepping in audio/video pipeline in PAUSED
|
|
|
|
.-----. .-------. .------. .-------.
|
|
| src | | demux | .-----. | vdec | | vsink |
|
|
| src->sink src1->|queue|->sink src->sink |
|
|
'-----' | | '-----' '------' '-------'
|
|
| | .------. .-------.
|
|
| | .-----. | adec | | asink |
|
|
| src2->|queue|->sink src->sink |
|
|
'-------' '-----' '------' '-------'
|
|
|
|
|
|
- app sets the pipeline to PAUSED to block on the preroll picture
|
|
- app seeks to required position in the stream. This can be done with a
|
|
positive or negative rate depending on the required frame stepping
|
|
direction.
|
|
- app steps frames (in GST_FORMAT_DEFAULT or GST_FORMAT_BUFFER) or an amount
|
|
of time on the video sink. The pipeline loses its PAUSED state until the
|
|
required number of frames have been skipped, it then prerolls again.
|
|
This skipping is purely done in the sink.
|
|
- sink posts STEP_DONE with amount of frames stepped and corresponding time
|
|
interval.
|
|
- the app skips the same amount of time on the audiosink to align the
|
|
streams again. When huge amount of video frames are skipped, there needs
|
|
to be enough queueing in the pipeline to compensate for the accumulated
|
|
audio.
|
|
|
|
* frame stepping in audio/video pipeline in PLAYING
|
|
|
|
- app sets the pipeline to PAUSED to block on the preroll picture
|
|
- app seeks to required position in the stream. This can be done with a
|
|
positive or negative rate depending on the required frame stepping
|
|
direction.
|
|
- app configures frames steps (in GST_FORMAT_DEFAULT or GST_FORMAT_BUFFER) or
|
|
an amount of time on the sink. The step event has a flag indicating live
|
|
stepping so that the stepping will only happens in PLAYING.
|
|
- app sets pipeline to PLAYING. The pipeline continues PLAYING until it
|
|
consumed the amount of time.
|
|
- sink posts STEP_DONE with amount of frames stepped and corresponding time
|
|
interval. The sink will then wait for another step event. Since the
|
|
STEP_DONE message was emited by the sink when it handed off the buffer to
|
|
the device, there is usually sufficient time to queue a new STEP event so
|
|
that one can seamlessly continue stepping.
|
|
|
|
|
|
events
|
|
------
|
|
|
|
A new GST_EVENT_STEP event is introduced to start the step operation.
|
|
The step event is created with the following fields in the structure:
|
|
|
|
"format", GST_TYPE_FORMAT
|
|
The format of the step units
|
|
|
|
"amount", G_TYPE_UINT64
|
|
The amount of units to step. -1 resumes normal non-stepping behaviour to
|
|
the end of the segment.
|
|
|
|
"rate", G_TYPE_DOUBLE
|
|
The rate at which the frames should be stepped in PLAYING mode. 1.0 is
|
|
the normal playback speed and direction of the segment, 2.0
|
|
is double speed. A speed of 0.0 is not allowed. When performing a
|
|
flushing step, the speed is not relevant. Note that we don't allow negative
|
|
rates here, use a seek with a negative rate first to reverse the playback
|
|
direction.
|
|
|
|
"flush", G_TYPE_BOOLEAN
|
|
when flushing is TRUE, the step is performed immediatly:
|
|
|
|
- In the PAUSED state the pipeline loses the PAUSED state, the requested
|
|
amount of data is skipped and the pipeline prerolls again when a
|
|
non-intermediate step completes.
|
|
When the pipeline was stepping while the event is sent, the current step
|
|
operation is updated with the new amount and format. The sink will do a
|
|
best effort to comply with the new amount.
|
|
- In the PLAYING state, the pipeline loses the PLAYING state, the
|
|
requested amount of data is skipped (not rendered) from the previous STEP
|
|
request or from the position of the last PAUSED if no previous STEP
|
|
operation was performed. The pipeline goes back to the PLAYING state
|
|
when a non-intermediate step completes.
|
|
|
|
When flushing is FALSE, the step will be performed later.
|
|
|
|
- In the PAUSED state the step will be done when going to PLAYING. Any
|
|
previous step operation will be overridden with the new STEP event.
|
|
- In the PLAYING state the step operation will be performed after the
|
|
current step operation completes. If there was no previous step
|
|
operation, the step operation will be performed from the position of the
|
|
last PAUSED state.
|
|
|
|
"intermediate", G_TYPE_BOOLEAN
|
|
Signal that this step operation is an intermediate step, part of a series
|
|
of step operations. It is mostly interesting for stepping in the PAUSED state
|
|
because the sink will only perform a preroll after a non-intermediate step
|
|
operation completes. Intermediate steps are usefull to flush out data from
|
|
other sinks in order to not cause excessive queueing. In the PLAYING state
|
|
the intermediate flag has no visual effect. In all states, the intermediate
|
|
flag is passed to the corresponding GST_MESSAGE_STEP_DONE.
|
|
|
|
|
|
The application will create a STEP event to start or stop the stepping
|
|
operation. Both stepping in PAUSED and PLAYING can be performed by means of
|
|
the flush flag.
|
|
|
|
The event is usually sent to the pipeline, which will typically distribute the
|
|
event to all of its sinks. For some use cases, like frame stepping on video
|
|
frames only, the event should only be sent to the video sink and upon reception
|
|
of the STEP_DONE message, one can step the other sinks to align the streams
|
|
again.
|
|
|
|
For large stepping amounts, there needs to be enough queueing in front of all
|
|
the sinks. If large steps need to be performed, they can be split up into
|
|
smaller step operations using the "intermediate" flag on the step.
|
|
|
|
Since the step event does not update the base_time of any of the elements, the
|
|
sinks should keep track of the amount of stepped data in order to remain
|
|
synchronized against the clock.
|
|
|
|
|
|
messages
|
|
--------
|
|
|
|
A GST_MESSAGE_STEP_START is created. It contains the following fields.
|
|
|
|
"active"
|
|
If the step was queued or activated.
|
|
|
|
"format", GST_TYPE_FORMAT
|
|
The format of the step units that queued/activated.
|
|
|
|
"amount", G_TYPE_UINT64
|
|
The amount of units that were queued/activated.
|
|
|
|
"rate", G_TYPE_DOUBLE
|
|
The rate and direction at which the frames were queued/activated.
|
|
|
|
"flush", G_TYPE_BOOLEAN
|
|
If the queued/activated frames will be flushed.
|
|
|
|
"intermediate", G_TYPE_BOOLEAN
|
|
If this is an intermediate step operation that queued/activated.
|
|
|
|
The STEP_START message is emited 2 times:
|
|
|
|
* first when an element received the STEP event and queued it. The "active"
|
|
field will be FALSE in this case.
|
|
|
|
* second when the step operation started in the streaming thread. The "active"
|
|
field is TRUE in this case. After this message is emited, the application
|
|
can queue a new step operation.
|
|
|
|
The purpose of this message is to find out how many elements participate in the
|
|
step operation and to queue new step operations at the earliest possible
|
|
moment.
|
|
|
|
A new GST_MESSAGE_STEP_DONE message is created. It contains the following
|
|
fields:
|
|
|
|
"format", GST_TYPE_FORMAT
|
|
The format of the step units that completed.
|
|
|
|
"amount", G_TYPE_UINT64
|
|
The amount of units that were stepped.
|
|
|
|
"rate", G_TYPE_DOUBLE
|
|
The rate and direction at which the frames were stepped.
|
|
|
|
"flush", G_TYPE_BOOLEAN
|
|
If the stepped frames were flushed.
|
|
|
|
"intermediate", G_TYPE_BOOLEAN
|
|
If this is an intermediate step operation that completed.
|
|
|
|
"duration", G_TYPE_UINT64
|
|
The total duration of the stepped units in GST_FORMAT_TIME.
|
|
|
|
"eos", G_TYPE_BOOLEAN
|
|
The step ended because of EOS.
|
|
|
|
The message is emited by the element that performs the step operation. The
|
|
purpose is to return the duration in GST_FORMAT_TIME of the stepped media. This
|
|
especially interesting to align other stream in case of stepping frames on the
|
|
video sink element.
|
|
|
|
|
|
Direction switch
|
|
----------------
|
|
|
|
When quickly switching between a forwards and a backwards step of, for example,
|
|
one video frame, we need either:
|
|
|
|
a) issue a new seek to change the direction from the current position.
|
|
b) cache a certain number of stepped frames and walk the cache.
|
|
|
|
option a) might be very slow.
|
|
For option b) we would ideally like to offload this caching functionality to a
|
|
separate element, which means that we need to forward the STEP event upstream.
|
|
It's unclear how this could work in a generic way. What is a demuxer supposed
|
|
to do when it received a step event? a flushing seek to what stream position?
|
|
|
|
|