mirror of
https://gitlab.freedesktop.org/gstreamer/gstreamer.git
synced 2024-07-16 11:28:36 +00:00
819d436ec4
These show up in the index. Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/2222>
244 lines
10 KiB
Markdown
244 lines
10 KiB
Markdown
# Frame stepping
|
||
|
||
This document outlines the details of the frame stepping functionality
|
||
in GStreamer.
|
||
|
||
The stepping functionality operates on the current playback segment,
|
||
position and rate as it was configured with a regular seek event. In
|
||
contrast to the seek event, it operates very closely to the sink and
|
||
thus has a very low latency and is not slowed down by queues and does
|
||
not actually perform any seeking logic. For this reason we want to
|
||
include a new API instead of reusing the seek API.
|
||
|
||
The following requirements are needed:
|
||
|
||
- The ability to walk forwards and backwards in the stream.
|
||
|
||
- Arbitrary increments in any supported format (time, frames, bytes …)
|
||
|
||
- High speed, minimal overhead. This mechanism is not more expensive
|
||
than simple playback.
|
||
|
||
- switching between forwards and backwards stepping should be fast.
|
||
|
||
- Maintain synchronisation between streams.
|
||
|
||
- Get feedback of the amount of skipped data.
|
||
|
||
- Ability to play a certain amount of data at an arbitrary speed.
|
||
|
||
We want a system where we can step frames in `PAUSED` as well as play
|
||
short segments of data in `PLAYING`.
|
||
|
||
## Use Cases
|
||
|
||
### video only pipeline in PAUSED
|
||
|
||
```
|
||
.-----. .-------. .------. .-------.
|
||
| src | | demux | .-----. | vdec | | vsink |
|
||
| src->sink src1->|queue|->sink src->sink |
|
||
'-----' '-------' '-----' '------' '-------'
|
||
```
|
||
|
||
- app sets the pipeline to `PAUSED` to block on the preroll picture
|
||
|
||
- app seeks to required position in the stream. This can be done
|
||
with a positive or negative rate depending on the required frame
|
||
stepping direction.
|
||
|
||
- app steps frames (in `GST_FORMAT_DEFAULT` or `GST_FORMAT_BUFFER)`. The
|
||
pipeline loses its `PAUSED` state until the required number of frames have been
|
||
skipped, it then prerolls again. This skipping is purely done in the sink.
|
||
|
||
- sink posts `STEP_DONE` with amount of frames stepped and
|
||
corresponding time interval.
|
||
|
||
### audio/video pipeline in PAUSED
|
||
|
||
```
|
||
.-----. .-------. .------. .-------.
|
||
| src | | demux | .-----. | vdec | | vsink |
|
||
| src->sink src1->|queue|->sink src->sink |
|
||
'-----' | | '-----' '------' '-------'
|
||
| | .------. .-------.
|
||
| | .-----. | adec | | asink |
|
||
| src2->|queue|->sink src->sink |
|
||
'-------' '-----' '------' '-------'
|
||
```
|
||
|
||
- app sets the pipeline to `PAUSED` to block on the preroll picture
|
||
|
||
- app seeks to required position in the stream. This can be done
|
||
with a positive or negative rate depending on the required frame
|
||
stepping direction.
|
||
|
||
- app steps frames (in `GST_FORMAT_DEFAULT` or `GST_FORMAT_BUFFER`) or an
|
||
amount of time on the video sink. The pipeline loses its `PAUSED` state until
|
||
the required number of frames have been skipped, it then prerolls again. This
|
||
skipping is purely done in the sink.
|
||
|
||
- sink posts `STEP_DONE` with amount of frames stepped and
|
||
corresponding time interval.
|
||
|
||
- the app skips the same amount of time on the audiosink to align
|
||
the streams again. When huge amount of video frames are skipped,
|
||
there needs to be enough queueing in the pipeline to compensate
|
||
for the accumulated audio.
|
||
|
||
### audio/video pipeline in PLAYING
|
||
|
||
- app sets the pipeline to `PAUSED` to block on the preroll picture
|
||
|
||
- app seeks to required position in the stream. This can be done
|
||
with a positive or negative rate depending on the required frame
|
||
stepping direction.
|
||
|
||
- app configures frames steps (in `GST_FORMAT_DEFAULT` or
|
||
`GST_FORMAT_BUFFER` or an amount of time on the sink. The step event has
|
||
a flag indicating live stepping so that the stepping will only happens in
|
||
PLAYING.
|
||
|
||
- app sets pipeline to PLAYING. The pipeline continues PLAYING
|
||
until it consumed the amount of time.
|
||
|
||
- sink posts `STEP_DONE` with amount of frames stepped and
|
||
corresponding time interval. The sink will then wait for another
|
||
step event. Since the `STEP_DONE` message was emitted by the sink
|
||
when it handed off the buffer to the device, there is usually
|
||
sufficient time to queue a new STEP event so that one can
|
||
seamlessly continue stepping.
|
||
|
||
## events
|
||
|
||
A new `GST_EVENT_STEP` event is introduced to start the step operation.
|
||
The step event is created with the following fields in the structure:
|
||
|
||
* **`format`** `GST_TYPE_FORMAT`: The format of the step units
|
||
|
||
* **`amount`** `G_TYPE_UINT64`: The amount of units to step. A 0 amount
|
||
immediately completes and can be used to cancel the current step and resume
|
||
normal non-stepping behaviour to the end of the segment. A -1 amount steps
|
||
until the end of the segment.
|
||
|
||
* **`rate`** `G_TYPE_DOUBLE`: The rate at which the frames should be stepped in
|
||
PLAYING mode. 1.0 is the normal playback speed and direction of the segment,
|
||
2.0 is double speed. A speed of 0.0 is not allowed. When performing a flushing
|
||
step, the speed is not relevant. Note that we don't allow negative rates here,
|
||
use a seek with a negative rate first to reverse the playback direction.
|
||
|
||
* **`flush`** `G_TYPE_BOOLEAN`: when flushing is TRUE, the step is performed
|
||
immediately:
|
||
|
||
- In the `PAUSED` state the pipeline loses the `PAUSED` state, the
|
||
requested amount of data is skipped and the pipeline prerolls again
|
||
when a non-intermediate step completes. When the pipeline was
|
||
stepping while the event is sent, the current step operation is
|
||
updated with the new amount and format. The sink will do a best
|
||
effort to comply with the new amount.
|
||
|
||
- In the PLAYING state, the pipeline loses the `PLAYING` state, the
|
||
requested amount of data is skipped (not rendered) from the previous
|
||
STEP request or from the position of the last `PAUSED` if no previous
|
||
STEP operation was performed. The pipeline goes back to the `PLAYING`
|
||
state when a non-intermediate step completes.
|
||
|
||
- When flushing is FALSE, the step will be performed later.
|
||
|
||
- In the `PAUSED` state the step will be done when going to `PLAYING`. Any
|
||
previous step operation will be overridden with the new `STEP` event.
|
||
|
||
- In the `PLAYING` state the step operation will be performed after the
|
||
current step operation completes. If there was no previous step
|
||
operation, the step operation will be performed from the position of
|
||
the last `PAUSED` state.
|
||
|
||
* **`intermediate`** `G_TYPE_BOOLEAN`: Signal that this step operation is an
|
||
intermediate step, part of a series of step operations. It is mostly
|
||
interesting for stepping in the `PAUSED` state because the sink will only perform
|
||
a preroll after a non-intermediate step operation completes. Intermediate steps
|
||
are useful to flush out data from other sinks in order to not cause excessive
|
||
queueing. In the PLAYING state the intermediate flag has no visual effect. In
|
||
all states, the intermediate flag is passed to the corresponding
|
||
`GST_MESSAGE_STEP_DONE`.
|
||
|
||
The application will create a STEP event to start or stop the stepping
|
||
operation. Both stepping in `PAUSED` and `PLAYING` can be performed by means
|
||
of the flush flag.
|
||
|
||
The event is usually sent to the pipeline, which will typically
|
||
distribute the event to all of its sinks. For some use cases, like frame
|
||
stepping on video frames only, the event should only be sent to the
|
||
video sink and upon reception of the `STEP_DONE` message, one can step
|
||
the other sinks to align the streams again.
|
||
|
||
For large stepping amounts, there needs to be enough queueing in front
|
||
of all the sinks. If large steps need to be performed, they can be split
|
||
up into smaller step operations using the "intermediate" flag on the
|
||
step.
|
||
|
||
Since the step event does not update the `base_time` of any of the
|
||
elements, the sinks should keep track of the amount of stepped data in
|
||
order to remain synchronized against the clock.
|
||
|
||
## messages
|
||
|
||
A `GST_MESSAGE_STEP_START` is created. It contains the following
|
||
fields.
|
||
|
||
* **`active`**: If the step was queued or activated.
|
||
|
||
* **`format`** `GST_TYPE_FORMAT`: The format of the step units that queued/activated.
|
||
|
||
* **`amount`** `G_TYPE_UINT64`: The amount of units that were queued/activated.
|
||
|
||
* **`rate`** `G_TYPE_DOUBLE`: The rate and direction at which the frames were queued/activated.
|
||
|
||
* **`flush`** `G_TYPE_BOOLEAN`: If the queued/activated frames will be flushed.
|
||
|
||
* **`intermediate`** `G_TYPE_BOOLEAN`: If this is an intermediate step operation
|
||
that queued/activated.
|
||
|
||
The `STEP_START` message is emitted 2 times:
|
||
|
||
- first when an element received the STEP event and queued it. The
|
||
"active" field will be FALSE in this case.
|
||
|
||
- second when the step operation started in the streaming thread. The
|
||
"active" field is TRUE in this case. After this message is emitted,
|
||
the application can queue a new step operation.
|
||
|
||
The purpose of this message is to find out how many elements participate
|
||
in the step operation and to queue new step operations at the earliest
|
||
possible moment.
|
||
|
||
A new `GST_MESSAGE_STEP_DONE` message is created. It contains the
|
||
following fields:
|
||
|
||
* **`format`** `GST_TYPE_FORMAT`: The format of the step units that completed.
|
||
* **`amount`** `G_TYPE_UINT64`: The amount of units that were stepped.
|
||
* **`rate`** `G_TYPE_DOUBLE`: The rate and direction at which the frames were stepped.
|
||
* **`flush`** `G_TYPE_BOOLEAN`: If the stepped frames were flushed.
|
||
* **`intermediate`** `G_TYPE_BOOLEAN`: If this is an intermediate step operation that completed.
|
||
* **`duration`** `G_TYPE_UINT64`: The total duration of the stepped units in `GST_FORMAT_TIME`.
|
||
* **`eos`** `G_TYPE_BOOLEAN`: The step ended because of EOS.
|
||
|
||
The message is emitted by the element that performs the step operation.
|
||
The purpose is to return the duration in `GST_FORMAT_TIME` of the
|
||
stepped media. This especially interesting to align other stream in case
|
||
of stepping frames on the video sink element.
|
||
|
||
## Direction switch
|
||
|
||
When quickly switching between a forwards and a backwards step of, for
|
||
example, one video frame, we need either:
|
||
|
||
1) issue a new seek to change the direction from the current position.
|
||
2) cache a certain number of stepped frames and walk the cache.
|
||
|
||
option 1) might be very slow. For option 2) we would ideally like to
|
||
offload this caching functionality to a separate element, which means
|
||
that we need to forward the STEP event upstream. It’s unclear how this
|
||
could work in a generic way. What is a demuxer supposed to do when it
|
||
received a step event? a flushing seek to what stream position?
|