gstreamer/markdown/design/keyframe-force.md

# Forcing keyframes

Consider the following use case:

We have a pipeline that performs video and audio capture from a live
source, compresses and muxes the streams and writes the resulting data
into a file.

Inside the uncompressed video data we have a specific pattern inserted
at specific moments that should trigger a switch to a new file, meaning,
we close the existing file we are writing to and start writing to a new
file.

We want the new file to start with a keyframe so that one can start
decoding the file immediately.

## Components

1)  We need an element that is able to detect the pattern in the video
    stream.

2)  We need to inform the video encoder that it should start encoding a
    keyframe starting from exactly the frame with the pattern.

3)  We need to inform the demuxer that it should flush out any pending
    data and start creating the start of a new file with the keyframe as
    a first video frame.

4)  We need to inform the sink element that it should start writing to
    the next file. This requires application interaction to instruct the
    sink of the new filename. The application should also be free to
    ignore the boundary and continue to write to the existing file. The
    application will typically use an event pad probe to detect the
    custom event.

## Implementation

### Downstream

The implementation would consist of generating a `GST_EVENT_CUSTOM_DOWNSTREAM`
event that marks the keyframe boundary. This event is inserted into the
pipeline by the application upon a certain trigger. In the above use case
this trigger would be given by the element that detects the pattern, in the
form of an element message.

The custom event would travel further downstream to instruct encoder,
muxer and sink about the possible switch.

The information passed in the event consists of:

**GstForceKeyUnit**

 - **"timestamp"** (`G_TYPE_UINT64`): the timestamp of the buffer that
   triggered the event.

 - **"stream-time"** (`G_TYPE_UINT64`): the stream position that triggered the event.

 - **"running-time"** (`G_TYPE_UINT64`): the running time of the stream when
   the event was triggered.

 - **"all-headers"** (`G_TYPE_BOOLEAN`): Send all headers, including
   those in the caps or those sent at the start of the stream.

 - **...**: optional other data fields.

Note that this event is purely informational, no element is required to
perform an action but it should forward the event downstream, just like
any other event it does not handle.

Elements understanding the event should behave as follows:

1)  The video encoder receives the event before the next frame. Upon
    reception of the event it schedules to encode the next frame as a
    keyframe. Before pushing out the encoded keyframe it must push the
    `GstForceKeyUnit` event downstream.

2)  The muxer receives the `GstForceKeyUnit` event and flushes out its
    current state, preparing to produce data that can be used as a
    keyunit. Before pushing out the new data it pushes the
    `GstForceKeyUnit` event downstream.

3)  The application receives the `GstForceKeyUnit` on a sink padprobe of
    the sink and reconfigures the sink to make it perform new actions
    after receiving the next buffer.

### Upstream

When using RTP packets can get lost or receivers can be added at any
time, they may request a new key frame.

An downstream element sends an upstream `GstForceKeyUnit` event up the
pipeline.

When an element produces some kind of key unit in output, but has no
such concept in its input (like an encoder that takes raw frames), it
consumes the event (doesn't pass it upstream), and instead sends a
downstream `GstForceKeyUnit` event and a new keyframe.