Memory allocation and management is a very important topic in multimedia. High definition video uses many megabytes to store one single frame of video. It is important to reuse the memory when possible instead of constantly allocating and freeing the memory. Multimedia systems usually use special purpose chips, such as DSPs or GPUs to perform the heavy lifting (especially for video). These special purpose chips have usually strict requirements for the memory that they can operate on and how the memory is accessed. This chapter talks about the memory management features that &GStreamer; plugins can use. We will first talk about the lowlevel GstMemory object that manages access to a piece of memory. We then continue with GstBuffer that is used to exchange data between plugins (and the application) and that uses GstMemory. We talk about GstMeta that can be placed on buffers to give extra info about the buffer and its memory. For efficiently managing buffers of the same size, we take a look at GstBufferPool. To conclude this chapter we take a look at the GST_QUERY_ALLOCATION query that is used to negotiate memory management options between elements. GstMemory is an object that manages a region of memory. The memory object points to a region of memory of maxsize. The area in this memory starting at offset and for size bytes is the accessible region in the memory. the maxsize of the memory can never be changed after the object is created, however, the offset and size can be changed. GstMemory objects are created by a GstAllocator object. Most allocators implement the default gst_allocator_alloc() method but some allocator might implement a different method, for example when additional parameters are needed to allocate the specific memory. Different allocators exist for, for example, system memory, shared memory and memory backed by a DMAbuf file descriptor. To implement support for a new kind of memory type, you must implement a new allocator object as shown below. Data access to the memory wrapped by the GstMemory object is always protected with a gst_memory_map() and gst_memory_unmap() pair. An access mode (read/write) must be given when mapping memory. The map function returns a pointer to the valid memory region that can then be accessed according to the requested access mode. Below is an example of making a GstMemory object and using the gst_memory_map() to access the memory region. WRITEME A GstBuffer is an lightweight object that is passed from an upstream to a downstream element and contains memory and metadata. It represents the multimedia content that is pushed or pull downstream by elements. The buffer contains one or more GstMemory objects that represent the data in the buffer. Metadata in the buffer consists of: DTS and PTS timestamps. These represent the decoding and presentation timestamps of the buffer content and is used by synchronizing elements to schedule buffers. Both these timestamps can be GST_CLOCK_TIME_NONE when unknown/undefined. The duration of the buffer contents. This duration can be GST_CLOCK_TIME_NONE when unknown/undefined. Media specific offsets and offset_end. For video this is the frame number in the stream and for audio the sample number. Other definitions for other media exist. Arbitrary structures via GstMeta, see below. A buffer is writable when the refcount of the object is exactly 1, meaning that only one object is holding a ref to the buffer. You can only modify anything in the buffer when the buffer is writable. This means that you need to call gst_buffer_make_writable() before changing the timestamps, offsets, metadata or adding and removing memory blocks. You can create a buffer with gst_buffer_new () and then add memory objects to it or you can use a convenience function gst_buffer_new_allocate () which combines the two. It's also possible to wrap existing memory with gst_buffer_new_wrapped_full () where you can give the function to call when the memory should be freed. You can access the memory of the buffer by getting and mapping the GstMemory objects individually or by using gst_buffer_map (). The latter merges all the memory into one big block and then gives you a pointer to this block. Below is an example of how to create a buffer and access its memory. With the GstMeta system you can add arbitrary structures on buffers. These structures describe extra properties of the buffer such as cropping, stride, region of interest etc. The metadata system separates API specification (what the metadata and its API look like) and the implementation (how it works). This makes it possible to make different implementations of the same API, for example, depending on the hardware you are running on. After allocating a new buffer, you can add metadata to the buffer with the metadata specific API. This means that you will need to link to the header file where the metadata is defined to use its API. By convention, a metadata API with name FooBar should provide two methods, a gst_buffer_add_foo_bar_meta () and a gst_buffer_get_foo_bar_meta (). Both functions should return a pointer to a FooBarMeta structure that contains the metadata fields. Some of the _add_*_meta () can have extra parameters that will usually be used to configure the metadata structure for you. Let's have a look at the metadata that is used to specify a cropping region for video frames. [...] GstVideoCropMeta *meta; /* buffer points to a video frame, add some cropping metadata */ meta = gst_buffer_add_video_crop_meta (buffer); /* configure the cropping metadata */ meta->x = 8; meta->y = 8; meta->width = 120; meta->height = 80; [...] ]]> An element can then use the metadata on the buffer when rendering the frame like this: [...] GstVideoCropMeta *meta; /* buffer points to a video frame, get the cropping metadata */ meta = gst_buffer_get_video_crop_meta (buffer); if (meta) { /* render frame with cropping */ _render_frame_cropped (buffer, meta->x, meta->y, meta->width, meta->height); } else { /* render frame */ _render_frame (buffer); } [...] ]]> In the next sections we show how you can add new metadata to the system and use it on buffers. First we need to define what our API will look like and we will have to register this API to the system. This is important because this API definition will be used when elements negotiate what kind of metadata they will exchange. The API definition also contains arbitrary tags that give hints about what the metadata contains. This is important when we see how metadata is preserved when buffers pass through the pipeline. If you are making a new implementation of an existing API, you can skip this step and move on to the implementation step. First we start with making the my-example-meta.h header file that will contain the definition of the API and structure for our metadata. typedef struct _MyExampleMeta MyExampleMeta; struct _MyExampleMeta { GstMeta meta; gint age; gchar *name; }; GType my_example_meta_api_get_type (void); #define MY_EXAMPLE_META_API_TYPE (my_example_meta_api_get_type()) #define gst_buffer_get_my_example_meta(b) \ ((MyExampleMeta*)gst_buffer_get_meta((b),MY_EXAMPLE_META_API_TYPE)) ]]> The metadata API definition consists of the definition of the structure that holds a gint and a string. The first field in the structure must be GstMeta. We also define a my_example_meta_api_get_type () function that will register out metadata API definition. We also define a convenience macro gst_buffer_get_my_example_meta () that simply finds and returns the metadata with our new API. Next let's have a look at how the my_example_meta_api_get_type () function is implemented in the my-example-meta.c file. As you can see, it simply uses the gst_meta_api_type_register () function to register a name for the api and some tags. The result is a new pointer GType that defines the newly registered API. Next we can make an implementation for a registered metadata API GType. The implementation detail of a metadata API are kept in a GstMetaInfo structure that you will make available to the users of your metadata API implementation with a my_example_meta_get_info () function and a convenience MY_EXAMPLE_META_INFO macro. You will also make a method to add your metadata implementation to a GstBuffer. Your my-example-meta.h header file will need these additions: Let's have a look at how these functions are implemented in the my-example-meta.c file. age = 0; emeta->name = NULL; return TRUE; } static gboolean my_example_meta_transform (GstBuffer * transbuf, GstMeta * meta, GstBuffer * buffer, GQuark type, gpointer data) { MyExampleMeta *emeta = (MyExampleMeta *) meta; /* we always copy no matter what transform */ gst_buffer_add_my_example_meta (transbuf, emeta->age, emeta->name); return TRUE; } static void my_example_meta_free (GstMeta * meta, GstBuffer * buffer) { MyExampleMeta *emeta = (MyExampleMeta *) meta; g_free (emeta->name) emeta->name = NULL; } const GstMetaInfo * my_example_meta_get_info (void) { static const GstMetaInfo *meta_info = NULL; if (g_once_init_enter (&meta_info)) { const GstMetaInfo *mi = gst_meta_register (MY_EXAMPLE_META_API_TYPE, "MyExampleMeta", sizeof (MyExampleMeta), my_example_meta_init, my_example_meta_free, my_example_meta_transform); g_once_init_leave (&meta_info, mi); } return meta_info; } MyExampleMeta * gst_buffer_add_my_example_meta (GstBuffer *buffer, gint age, const gchar *name) { MyExampleMeta *meta; g_return_val_if_fail (GST_IS_BUFFER (buffer), NULL); meta = (MyExampleMeta *) gst_buffer_add_meta (buffer, MY_EXAMPLE_META_INFO, NULL); meta->age = age; meta->name = g_strdup (name); return meta; } ]]> gst_meta_register () registers the implementation details, like the API that you implement and the size of the metadata structure along with methods to initialize and free the memory area. You can also implement a transform function that will be called when a certain transformation (identified by the quark and quark specific data) is performed on a buffer. Lastly, you implement a gst_buffer_add_*_meta() that adds the metadata implementation to a buffer and sets the values of the metadata. The GstBufferPool object provides a convenient base class for managing lists of reusable buffers. Essential for this object is that all the buffers have the same properties such as size, padding, metadata and alignment. A bufferpool object can be configured to manage a minimum and maximum amount of buffers of a specific size. A bufferpool can also be configured to use a specific GstAllocator for the memory of the buffers. There is support in the bufferpool to enable bufferpool specific options, such as adding GstMeta to the buffers in the pool or such as enabling specific padding on the memory in the buffers. A Bufferpool can be inactivate and active. In the inactive state, you can configure the pool. In the active state, you can't change the configuration anymore but you can acquire and release buffers from/to the pool. In the following sections we take a look at how you can use a bufferpool. Many different bufferpool implementations can exist; they are all subclasses of the base class GstBufferPool. For this example, we will assume we somehow have access to a bufferpool, either because we created it ourselves or because we were given one as a result of the ALLOCATION query as we will see below. The bufferpool is initially in the inactive state so that we can configure it. Trying to configure a bufferpool that is not in the inactive state will fail. Likewise, trying to activate a bufferpool that is not configured will fail. The configuration of the bufferpool is maintained in a generic GstStructure that can be obtained with gst_buffer_pool_get_config(). Convenience methods exist to get and set the configuration options in this structure. After updating the structure, it is set as the current configuration in the bufferpool again with gst_buffer_pool_set_config(). The following options can be configured on a bufferpool: The caps of the buffers to allocate. The size of the buffers. This is the suggested size of the buffers in the pool. The pool might decide to allocate larger buffers to add padding. The minimum and maximum amount of buffers in the pool. When minimum is set to > 0, the bufferpool will pre-allocate this amount of buffers. When maximum is not 0, the bufferpool will allocate up to maximum amount of buffers. The allocator and parameters to use. Some bufferpools might ignore the allocator and use its internal one. Other arbitrary bufferpool options identified with a string. a bufferpool lists the supported options with gst_buffer_pool_get_options() and you can ask if an option is supported with gst_buffer_pool_has_option(). The option can be enabled by adding it to the configuration structure with gst_buffer_pool_config_add_option (). These options are used to enable things like letting the pool set metadata on the buffers or to add extra configuration options for padding, for example. After the configuration is set on the bufferpool, the pool can be activated with gst_buffer_pool_set_active (pool, TRUE). From that point on you can use gst_buffer_pool_acquire_buffer () to retrieve a buffer from the pool, like this: It is important to check the return value of the acquire function because it is possible that it fails: When your element shuts down, it will deactivate the bufferpool and then all calls to acquire will return GST_FLOW_FLUSHNG. All buffers that are acquired from the pool will have their pool member set to the original pool. When the last ref is decremented on the buffer, &GStreamer; will automatically call gst_buffer_pool_release_buffer() to release the buffer back to the pool. You (or any other downstream element) don't need to know if a buffer came from a pool, you can just unref it. WRITEME The ALLOCATION query is used to negotiate GstMeta, GstBufferPool and GstAllocator between elements. Negotiation of the allocation strategy is always initiated and decided by a srcpad after it has negotiated a format and before it decides to push buffers. A sinkpad can suggest an allocation strategy but it is ultimately the source pad that will decide based on the suggestions of the downstream sink pad. The source pad will do a GST_QUERY_ALLOCATION with the negotiated caps as a parameter. This is needed so that the downstream element knows what media type is being handled. A downstream sink pad can answer the allocation query with the following results: An array of possible GstBufferPool suggestions with suggested size, minimum and maximum amount of buffers. An array of GstAllocator objects along with suggested allocation parameters such as flags, prefix, alignment and padding. These allocators can also be configured in a bufferpool when this is supported by the bufferpool. An array of supported GstMeta implementations along with metadata specific parameters. It is important that the upstream element knows what kind of metadata is supported downstream before it places that metadata on buffers. When the GST_QUERY_ALLOCATION returns, the source pad will select from the available bufferpools, allocators and metadata how it will allocate buffers. Below is an example of the ALLOCATION query. #include #include GstCaps *caps; GstQuery *query; GstStructure *structure; GstBufferPool *pool; GstStructure *config; guint size, min, max; [...] /* find a pool for the negotiated caps now */ query = gst_query_new_allocation (caps, TRUE); if (!gst_pad_peer_query (scope->srcpad, query)) { /* query failed, not a problem, we use the query defaults */ } if (gst_query_get_n_allocation_pools (query) > 0) { /* we got configuration from our peer, parse them */ gst_query_parse_nth_allocation_pool (query, 0, &pool, &size, &min, &max); } else { pool = NULL; size = 0; min = max = 0; } if (pool == NULL) { /* we did not get a pool, make one ourselves then */ pool = gst_video_buffer_pool_new (); } config = gst_buffer_pool_get_config (pool); gst_buffer_pool_config_add_option (config, GST_BUFFER_POOL_OPTION_VIDEO_META); gst_buffer_pool_config_set_params (config, caps, size, min, max); gst_buffer_pool_set_config (pool, config); /* and activate */ gst_buffer_pool_set_active (pool, TRUE); [...] ]]> This particular implementation will make a custom GstVideoBufferPool object that is specialized in allocating video buffers. You can also enable the pool to put GstVideoMeta metadata on the buffers from the pool doing gst_buffer_pool_config_add_option (config, GST_BUFFER_POOL_OPTION_VIDEO_META). In many baseclasses you will see the following virtual methods for influencing the allocation strategy: propose_allocation () should suggest allocation parameters for the upstream element. decide_allocation () should decide the allocation parameters from the suggestions received from downstream. Implementors of these methods should modify the given GstQuery object by updating the pool options and allocation options.