gstreamer/markdown/design/bufferpool.md

Bufferpool

This document details the design of how buffers are be allocated and managed in pools.

Bufferpools increase performance by reducing allocation overhead and improving possibilities to implement zero-copy memory transfer.

Together with the ALLOCATION query, elements can negotiate allocation properties and bufferpools between themselves. This also allows elements to negotiate buffer metadata between themselves.

Requirements

• Provide a GstBufferPool base class to help the efficient implementation of a list of reusable GstBuffer objects.

• Let upstream elements initiate the negotiation of a bufferpool and its configuration. Allow downstream elements provide bufferpool properties and/or a bufferpool. This includes the following properties:

  • have minimum and maximum amount of buffers with the option of preallocating buffers.

  • allocator, alignment and padding support

  • buffer metadata

  • arbitrary extra options

• Integrate with dynamic caps renegotiation.

• Notify upstream element of new bufferpool availability. This is important when a new element, that can provide a bufferpool, is dynamically linked downstream.

GstBufferPool

The bufferpool object manages a list of buffers with the same properties such as size, padding and alignment.

The bufferpool has two states: active and inactive. In the inactive state, the bufferpool can be configured with the required allocation preferences. In the active state, buffers can be retrieved from and returned to the pool.

The default implementation of the bufferpool is able to allocate buffers from any allocator with arbitrary alignment and padding/prefix.

Custom implementations of the bufferpool can override the allocation and free algorithms of the buffers from the pool. This should allow for different allocation strategies such as using shared memory or hardware mapped memory.

Negotiation

After a particular media format has been negotiated between two pads (using the CAPS event), they must agree on how to allocate buffers.

The srcpad will always take the initiative to negotiate the allocation properties. It starts with creating a GST_QUERY_ALLOCATION with the negotiated caps.

The srcpad can set the need-pool flag to TRUE in the query to optionally make the peer pad allocate a bufferpool. It should only do this if it is able to use the peer provided bufferpool.

It will then inspect the returned results and configure the returned pool or create a new pool with the returned properties when needed.

Buffers are then allocated by the srcpad from the negotiated pool and pushed to the peer pad as usual.

The allocation query can also return an allocator object when the buffers are of different sizes and can't be allocated from a pool.

Allocation query

The allocation query has the following fields:

• (in) caps, GST_TYPE_CAPS: the caps that was negotiated

• (in) need-pool, G_TYPE_BOOLEAN: if a GstBufferPool is requested

• (out) pool, G_TYPE_ARRAY of structure: an array of pool configurations:

c struct { GstBufferPool *pool; guint size; guint min_buffers; guint max_buffers; }

Use gst_query_parse_nth_allocation_pool() to get the values.

The allocator can contain multiple pool configurations. If need-pool was TRUE, the pool member might contain a GstBufferPool when the downstream element can provide one.

Size contains the size of the bufferpool's buffers and is never 0.

min_buffers and max_buffers contain the suggested min and max amount of buffers that should be managed by the pool.

The upstream element can choose to use the provided pool or make its own pool when none was provided or when the suggested pool was not acceptable.

The pool can then be configured with the suggested min and max amount of buffers or a downstream element might choose different values.

• (out) allocator, G_TYPE_ARRAY of structure: an array of allocator parameters that can be used.

    struct {
      GstAllocator *allocator;
      GstAllocationParams params;
    }

Use gst_query_parse_nth_allocation_param() to get the values.

The element performing the query can use the allocators and its parameters to allocate memory for the downstream element.

It is also possible to configure the allocator in a provided pool.

• (out) metadata, G_TYPE_ARRAY of structure: an array of metadata params that can be accepted.

    struct {
      GType api;
      GstStructure *params;
    }

Use gst_query_parse_nth_allocation_meta() to get the values.

These metadata items can be accepted by the downstream element when placed on buffers. There is also an arbitrary GstStructure associated with the metadata that contains metadata-specific options.

Some bufferpools have options to enable metadata on the buffers allocated by the pool.

Allocating from pool

Buffers are allocated from the pool of a pad:

res = gst_buffer_pool_acquire_buffer (pool, &buffer, &params);

A GstBuffer that is allocated from the pool will always be writable (have a refcount of 1) and it will also have its pool member point to the GstBufferPool that created the buffer.

Buffers are refcounted in the usual way. When the refcount of the buffer reaches 0, the buffer is automatically returned to the pool.

Since all the buffers allocated from the pool keep a reference to the pool, when nothing else is holding a refcount to the pool, it will be finalized when all the buffers from the pool are unreffed. By setting the pool to the inactive state we can drain all buffers from the pool.

When the pool is in the inactive state, gst_buffer_pool_acquire_buffer() will return GST_FLOW_FLUSHING immediately.

Extra parameters can be given to the gst_buffer_pool_acquire_buffer() method to influence the allocation decision. GST_BUFFER_POOL_FLAG_KEY_UNIT and GST_BUFFER_POOL_FLAG_DISCONT serve as hints.

When the bufferpool is configured with a maximum number of buffers, allocation will block when all buffers are outstanding until a buffer is returned to the pool. This behaviour can be changed by specifying the GST_BUFFER_POOL_FLAG_DONTWAIT flag in the parameters. With this flag set, allocation will return GST_FLOW_EOS when the pool is empty.

Renegotiation

Renegotiation of the bufferpool might need to be performed when the configuration of the pool changes. Changes can be in the buffer size (because of a caps change), alignment or number of buffers.

Downstream

When the upstream element wants to negotiate a new format, it might need to renegotiate a new bufferpool configuration with the downstream element. This can, for example, happen when the buffer size changes.

We can not just reconfigure the existing bufferpool because there might still be outstanding buffers from the pool in the pipeline. Therefore we need to create a new bufferpool for the new configuration while we let the old pool drain.

Implementations can choose to reuse the same bufferpool object and wait for the drain to finish before reconfiguring the pool.

The element that wants to renegotiate a new bufferpool uses exactly the same algorithm as when it first started. It will negotiate caps first then use the ALLOCATION query to get and configure the new pool.

upstream

When a downstream element wants to negotiate a new format, it will send a RECONFIGURE event upstream. This instructs upstream to renegotiate both the format and the bufferpool when needed.

A pipeline reconfiguration happens when new elements are added or removed from the pipeline or when the topology of the pipeline changes. Pipeline reconfiguration also triggers possible renegotiation of the bufferpool and caps.

A RECONFIGURE event tags each pad it travels on as needing reconfiguration. The next buffer allocation will then require the renegotiation or reconfiguration of a pool.

Shutting down

In push mode, a source pad is responsible for setting the pool to the inactive state when streaming stops. The inactive state will unblock any pending allocations so that the element can shut down.

In pull mode, the sink element should set the pool to the inactive state when shutting down so that the peer _get_range() function can unblock.

In the inactive state, all the buffers that are returned to the pool will automatically be freed by the pool and new allocations will fail.

Use cases

- videotestsrc ! xvimagesink

• Before videotestsrc can output a buffer, it needs to negotiate caps and a bufferpool with the downstream peer pad.

• First it will negotiate a suitable format with downstream according to the normal rules. It will send a CAPS event downstream with the negotiated configuration.

• Then it does an ALLOCATION query. It will use the returned bufferpool or configures its own bufferpool with the returned parameters. The bufferpool is initially in the inactive state.

• The ALLOCATION query lists the desired configuration of the downstream xvimagesink, which can have specific alignment and/or min/max amount of buffers.

• videotestsrc updates the configuration of the bufferpool, it will likely set the min buffers to 1 and the size of the desired buffers. It then updates the bufferpool configuration with the new properties.

• When the configuration is successfully updated, videotestsrc sets the bufferpool to the active state. This preallocates the buffers in the pool (if needed). This operation can fail when there is not enough memory available. Since the bufferpool is provided by xvimagesink, it will allocate buffers backed by an XvImage and pointing to shared memory with the X server.

• If the bufferpool is successfully activated, videotestsrc can acquire a buffer from the pool, fill in the data and push it out to xvimagesink.

• xvimagesink can know that the buffer originated from its pool by following the pool member.

• when shutting down, videotestsrc will set the pool to the inactive state, this will cause further allocations to fail and currently allocated buffers to be freed. videotestsrc will then free the pool and stop streaming.

- videotestsrc ! queue ! myvideosink

• In this second use case we have a videosink that can at most allocate 3 video buffers.

• Again videotestsrc will have to negotiate a bufferpool with the peer element. For this it will perform the ALLOCATION query which queue will proxy to its downstream peer element.

• The bufferpool returned from myvideosink will have a max_buffers set to 3. queue and videotestsrc can operate with this upper limit because none of those elements require more than that amount of buffers for temporary storage.

• Myvideosink's bufferpool will then be configured with the size of the buffers for the negotiated format and according to the padding and alignment rules. When videotestsrc sets the pool to active, the 3 video buffers will be preallocated in the pool.

• videotestsrc acquires a buffer from the configured pool on its srcpad and pushes this into the queue. When videotestsrc has acquired and pushed 3 frames, the next call to gst_buffer_pool_acquire_buffer() will block (assuming the GST_BUFFER_POOL_FLAG_DONTWAIT is not specified).

• When the queue has pushed out a buffer and the sink has rendered it, the refcount of the buffer reaches 0 and the buffer is recycled in the pool. This will wake up the videotestsrc that was blocked, waiting for more buffers and will make it produce the next buffer.

• In this setup, there are at most 3 buffers active in the pipeline and the videotestsrc is rate limited by the rate at which buffers are recycled in the bufferpool.

• When shutting down, videotestsrc will first set the bufferpool on the srcpad to inactive. This causes any pending (blocked) acquire to return with a FLUSHING result and causes the streaming thread to pause.

- .. ! myvideodecoder ! queue ! fakesink

• In this case, the myvideodecoder requires buffers to be aligned to 128 bytes and padded with 4096 bytes. The pipeline starts out with the decoder linked to a fakesink but we will then dynamically change the sink to one that can provide a bufferpool.

• When myvideodecoder negotiates the size with the downstream fakesink element, it will receive a NULL bufferpool because fakesink does not provide a bufferpool. It will then select its own custom bufferpool to start the data transfer.

• At some point we block the queue srcpad, unlink the queue from the fakesink, link a new sink and set the new sink to the PLAYING state. Linking the new sink would automatically send a RECONFIGURE event upstream and, through queue, inform myvideodecoder that it should renegotiate its bufferpool because downstream has been reconfigured.

• Before pushing the next buffer, myvideodecoder has to renegotiate a new bufferpool. To do this, it performs the usual bufferpool negotiation algorithm. If it can obtain and configure a new bufferpool from downstream, it sets its own (old) pool to inactive and unrefs it. This will eventually drain and unref the old bufferpool.

• The new bufferpool is set as the new bufferpool for the srcpad and sinkpad of the queue and set to the active state.

- .. ! myvideodecoder ! queue ! myvideosink

• myvideodecoder has negotiated a bufferpool with the downstream myvideosink to handle buffers of size 320x240. It has now detected a change in the video format and needs to renegotiate to a resolution of 640x480. This requires it to negotiate a new bufferpool with a larger buffer size.

• When myvideodecoder needs to get the bigger buffer, it starts the negotiation of a new bufferpool. It queries a bufferpool from downstream, reconfigures it with the new configuration (which includes the bigger buffer size) and sets the bufferpool to active. The old pool is inactivated and unreffed, which causes the old format to drain.

• It then uses the new bufferpool for allocating new buffers of the new dimension.

• If at some point, the decoder wants to switch to a lower resolution again, it can choose to use the current pool (which has buffers that are larger than the required size) or it can choose to renegotiate a new bufferpool.

- .. ! myvideodecoder ! videoscale ! myvideosink

• myvideosink is providing a bufferpool for upstream elements and wants to change the resolution.

• myvideosink sends a RECONFIGURE event upstream to notify upstream that a new format is desirable. Upstream elements try to negotiate a new format and bufferpool before pushing out a new buffer. The old bufferpools are drained in the regular way.