mirror of
https://gitlab.freedesktop.org/gstreamer/gstreamer.git
synced 2024-11-25 11:11:08 +00:00
c628641880
Implement handling of upstream keyframe forcing. Update the design documents too. Fixes #578656
87 lines
3.6 KiB
Text
87 lines
3.6 KiB
Text
Forcing keyframes
|
|
-----------------
|
|
|
|
Consider the following use case:
|
|
|
|
We have a pipeline that performs video and audio capture from a live source,
|
|
compresses and muxes the streams and writes the resulting data into a file.
|
|
|
|
Inside the uncompressed video data we have a specific pattern inserted at
|
|
specific moments that should trigger a switch to a new file, meaning, we close
|
|
the existing file we are writing to and start writing to a new file.
|
|
|
|
We want the new file to start with a keyframe so that one can start decoding
|
|
the file immediatly.
|
|
|
|
Components:
|
|
|
|
1) We need an element that is able to detect the pattern in the video stream.
|
|
|
|
2) We need to inform the video encoder that it should start encoding a keyframe
|
|
starting from exactly the frame with the pattern.
|
|
|
|
3) We need to inform the demuxer that it should flush out any pending data and
|
|
start creating the start of a new file with the keyframe as a first video
|
|
frame.
|
|
|
|
4) We need to inform the sink element that it should start writing to the next
|
|
file. This requires application interaction to instruct the sink of the new
|
|
filename. The application should also be free to ignore the boundary and
|
|
continue to write to the existing file. The application will typically use
|
|
an event pad probe to detect the custom event.
|
|
|
|
Implementation:
|
|
|
|
The implementation would consist of generating a GST_EVENT_CUSTOM_DOWNSTREAM
|
|
event that marks the keyframe boundary. This event is inserted into the
|
|
pipeline by the application upon a certain trigger. In the above use case this
|
|
trigger would be given by the element that detects the pattern, in the form of
|
|
an element message.
|
|
|
|
The custom event would travel further downstream to instruct encoder, muxer and
|
|
sink about the possible switch.
|
|
|
|
The information passed in the event consists of:
|
|
|
|
name: GstForceKeyUnit
|
|
(G_TYPE_UINT64)"timestamp" : the timestamp of the buffer that
|
|
triggered the event.
|
|
(G_TYPE_UINT64)"stream-time" : the stream position that triggered the
|
|
event.
|
|
(G_TYPE_UINT64)"running-time" : the running time of the stream when the
|
|
event was triggered.
|
|
.... : optional other data fields.
|
|
|
|
Note that this event is purely informational, no element is required to
|
|
perform an action but it should forward the event downstream, just like any
|
|
other event it does not handle.
|
|
|
|
Elements understanding the event should behave as follows:
|
|
|
|
1) The video encoder receives the event before the next frame. Upon reception
|
|
of the event it schedules to encode the next frame as a keyframe.
|
|
Before pushing out the encoded keyframe it must push the GstForceKeyUnit
|
|
event downstream.
|
|
|
|
2) The muxer receives the GstForceKeyUnit event and flushes out its current state,
|
|
preparing to produce data that can be used as a keyunit. Before pushing out
|
|
the new data it pushes the GstForceKeyUnit event downstream.
|
|
|
|
3) The application receives the GstForceKeyUnit on a sink padprobe of the sink
|
|
and reconfigures the sink to make it perform new actions after receiving
|
|
the next buffer.
|
|
|
|
|
|
Upstream
|
|
--------
|
|
|
|
When using RTP packets can get lost or receivers can be added at any time,
|
|
they may request a new key frame.
|
|
|
|
An downstream element sends an upstream "GstForceKeyUnit" event up the
|
|
pipeline.
|
|
|
|
When an element produces some kind of key unit in output, but has
|
|
no such concept in its input (like an encoder that takes raw frames),
|
|
it consumes the event (doesn't pass it upstream), and instead sends
|
|
a downstream GstForceKeyUnit event and a new keyframe.
|