GstBuffer^2 ----------- This draft document describes a possible design for arbitrary per-buffer metadata. The proposed changes in this document are not ABI/API compatible with the 0.10 version of GStreamer and should thus only be considered for upcomming unstable versions. Buffer metadata typically includes properties that give more information about the buffer contents. These properties are usually not negotiated and are thus not inside the caps. Some examples of metadata: - timestamp, duration - offset, offset_end - interlacing information - video alignment, cropping, panning information - extra container information such as granulepos, ... - extra global buffer properties Requirements ~~~~~~~~~~~~ - It must be fast * allocation, free, low fragmentation * access to the metadata fields, preferably not much slower than directly accessing a C structure field - It must be extensible. Elements should be able to add new arbitrary metadata without requiring much effort. Also new metadata fields should not break API or ABI. - It plays nice with subbuffers. When a subbuffer is created, the various buffer metadata should be copied/updated correctly. - We should be able to pass metadata in pad_alloc() and get_range() functions to specify extra allocation parameters. - We should be able to attach statically allocated metadata to a buffer. This is for metadata that does not change much. Use cases --------- * DSP vs CPU caches Both DSP and CPU have separate MMUs and memory caches. When we exchange buffers between two subsystems we need to flush caches so that one CPU can see the modifications done by the other CPU. These cashes must only be flushed when one CPU performed a write and the other CPU needs to do a read. In order to implement this we need to be able to mark our read and write operations on the buffer data. GstMiniObject ~~~~~~~~~~~~~ We make GstMiniObject a simple refcounted C structure and also a GLib boxed type. The following fields will be in the structure: struct _GstMiniObject { GType type; /*< public >*/ /* with COW */ /* refcounting */ gint refcount; guint flags; GstMiniObjectCopyFunction copy; GstMiniObjectFreeFunction free; } We will use the regular GSlice allocator or custom object pooling for allocating instances of the mini object. We use the well known refcounting mechanisms to manage the lifetime of the objects. GstEvent, GstCaps, GstQuery, GstMessage ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Have the new GstMiniObject be the first field in these objects. They will probably also replace the copy and free functions with their own implementations. Allocation of the objects will use the regular gst_*_new() functions that will allocate and initialize a parent GstMiniObject of the required size and setting up the custom functions. GstBuffer ~~~~~~~~~ A GstMiniObject will be the parent instance of the GstBuffer object, which is a regular C structure. struct _GstBuffer { GstMiniObject mini_object; gsize free_size; GstCaps *caps; GstBuffer *parent; GstBufferDataMap mmap_func; GstBufferDataUnmap munmap_func; gpointer _gst_padding[10]; }; The Buffer object will contain a pointer to the parent buffer to allow for subbuffers as a first class feature of a GstBuffer. Allocation of the GstBuffer structure will result in the allocation of a memory region of a customizable size (512 bytes). Only the first sizeof (GstBuffer) bytes of this region will initially be used. The remaining bytes will be part of the free metadata region of the buffer. The size of the free region is kept in the free_size field. Buffers point to a GstCaps structure that contains the caps of the buffer data. We add two functions, map and unmap. We introduce explicit read and write transactions on the buffer data in order to be able to implement flushing of the data between different caches and mapping between different MMUs. typedef enum { GST_BUFFER_MAP_NONE, GST_BUFFER_MAP_READ, GST_BUFFER_MAP_WRITE, } GstBufferMapFlags gpointer gst_buffer_map (GstBuffer *, guint offset, guint *size, GstBufferMapFlags); gboolean gst_buffer_unmap (GstBuffer *, gpointer data, guint size); The map functions always create a contiguous memory space for the requested area of the buffer. This might involve memcpy and other expensive operations. It is assumed that more specialized elements can deal with the different memory metadata structures in order to optimize access to the memory. GstBufferMeta ~~~~~~~~~~~~~ A GstBufferMeta is a structure as follows: struct _GstBufferMeta { GstBufferMetaInfo *info; /* tag and info for the meta item */ }; The purpose of the this structure is to serve as a common header for all metadata information that we can attach to a buffer. Specific metadata, such as timing metadata, will have this structure as the first field. For example: struct _GstBufferMetaTiming { GstBufferMeta meta; /* common meta header */ GstClockTime dts; /* decoding timestamp */ GstClockTime pts; /* presentation timestamp */ GstClockTime duration; /* duration of the data */ GstClockTime clock_rate; /* clock rate for the above values */ }; Or another example for the buffer memory region struct _GstBufferMetaMemory { GstBufferMeta meta; /* pointer to data and its size */ gpointer *data; guint size; gpointer *data_orig; GFreeFunc data_free; gpointer data_user; }; GstBufferMetaInfo will point to more information about the metadata and looks like this: struct _GstBufferMetaInfo { GQuark api; /* api name */ GQuark impl; /* implementation name */ gsize size; /* size of the structure */ GstBufferMetaInfo *parent /* parent info */ GstMetaInitFunction init_func; GstMetaFreeFunction free_func; GstMetaCopyFunction copy_func; GstMetaSubFunction sub_func; GstMetaSerializeFunction serialize_func GstMetaDeserializeFunction deserialize_func GstMetaConvFunction conv_func; }; Tag will contain a GQuark of the metadata name. We will be able to refer to specific metadata by name or by its (cached) GQuark. A repository of registered MetaInfo will be maintained by the core. We will register some common metadata structures in core and some media specific info for audio/video/text in -base. Plugins can register additional custom metadata. Along with the metadata description we will have functions to initialize/free (and/or refcount) a specific GstBufferMeta instance. We also have the possibility to add a custom subbuffer function that can be used to modify the metadata when a subbuffer is taken. We also add serialize and deserialize function for the metadata in case we need special logic for reading and writing the metadata. This is needed for GDP payloading of the metadata. We add a conv function to the Info structure that will be called when a buffer should be converted to an old-style buffer for backward compatibility. The purpose of the separate MetaInfo is to not have to carry the free/init functions in each buffer instance but to define them globally. We still want quick access to the info so we need to make the buffer metadata point to the info. Technically we could also specify the field and types in the MetaInfo and provide a generic API to retrieve the metadata fields without the need for a header file. We will not do this yet. The complete buffer with metadata would then look as follows: +-------------------------------------+ GstMiniObject | GType (GstBuffer) | | refcount, flags, copy/free | +-------------------------------------+ GstBuffer | caps, parent, subfunc | +.....................................+ | next ---+ | parent ------> NULL +- | info ------> GstBufferMetaInfo GstBufferMetaTiming | | | | | | dts | | | | pts | | | | duration | | +- | clock_rate | | + . . . . . . . . . . . . . . . . . . + | | next <--+ | parent ------> NULL GstBufferMetaMemory +- | info ------> GstBufferMetaInfo | | | | | | data | | | | size | | | | mallocdata | | | | data_free | | +- | data_user | | + . . . . . . . . . . . . . . . . . . + . . . API examples ~~~~~~~~~~~~ Buffers are created using the normal gst_buffer_new functions. The standard fields are initialized as usual. A memory area that is bigger than the structure size is allocated for the buffer metadata. The remaining free area is stored in the free_size field. gst_buffer_new (); After creating a buffer, the application can set caps. and add other metadata information. In order to modify metadata, a reference to the MetaInfo should be obtained. This can be done like this: GstBufferMetaInfo *info; info = gst_buffer_meta_get_info (GQuark tag); Usually the info will be obtained only once in order to avoid lock contention on the global pool of meta info. The core will also provide convenience functions for the core metainfo. Once a reference to the info has been obtained, the associated metadata can be added or modified on a buffer. For example, to modify the timing info on a buffer, one could use the following sequence: GstBufferMetaInfo *info; GstBufferMetaTiming *timing; info = gst_buffer_meta_get_info (GST_META_TIMING_QUARK); timing = gst_buffer_get_meta (buffer, info, TRUE); /* TRUE = create if absent */ timing->timestamp = 0; timing->duration = 20 * GST_MSECOND; The _get_meta() function returns a pointer to the metadata structure associated with the GST_META_TIMING_QUARK info. For the core meta info, we will provide convenience code that uses the cached GstBufferMetaInfo, making the above code a little more simple. GstBufferMetaTiming *timing; timing = gst_buffer_get_meta_timing (buffer, TRUE); /* TRUE = create if absent */ timing->timestamp = 0; timing->duration = 20 * GST_MSECOND; Note that for each of the metadata that we will add to buffers, we need a struct definition and a registered MetaInfo. We will also provide an API to iterate the different metainfo structures. A possible simple API would look like this: GstBufferMeta *current = NULL; /* passing NULL gives the first entry */ current = gst_buffer_meta_get_next (buffer, current); /* passing a GstBufferMeta returns the next */ current = gst_buffer_meta_get_next (buffer, current); Memory management ~~~~~~~~~~~~~~~~~ * allocation We will initially allocate a reasonable sized GstBuffer structure (say 512 bytes) and we will set the free_size to the maximum amount of metadata we can store. Since the complete buffer structure, including a large area for metadata, is allocated in one go, we can reduce the number of memory allocations while still providing dynamic metadata. When adding metadata, we need to call the init function of the associated metadata info structure. Since adding the metadata requires the caller to pass a handle to the info, this operation does not require table lookups. Per-metadata memory initialisation is needed because not all metadata is initialized in the same way. We need to, for example, set the timestamps to NONE in the MetaTiming structures. The init/free functions can also be used to implement refcounting for a metadata structure. This can be useful when a structure is shared between buffers. When the free_size of the GstBuffer is exhausted, we will allocate new memory for each newly added BufferMeta and use the next pointers to point to this. It is expected that this does not occur often and we might be able to optimize this transparently in the future. * free When a GstBuffer is freed, we potentially might have to call a custom free function on the metadata info. In the case of the Memory metadata, we need to call the associated free function to free the memory. When freeing a GstBuffer, the custom buffer free function will iterate all of the metadata in the buffer and call the associated free functions in the MetaInfo associated with the entries. Usually, this function will be NULL. Subbuffers ~~~~~~~~~~ Subbuffers are a first class feature of the GstBuffer. Creating a subbuffer from a GstBuffer will allocate a new GstBuffer and ref the parent buffer. It will then iterate all of the metadata entries for the parent buffer and call the associated sub_func in the MetaInfo. This allows each metadata structure to implement the actions needed to update the metadata of the subbuffer. A pointer to the old and new memory location of the metadata is passed to the sub_func. The default implementation will simply copy the metadata. Custom implementations can adjust the values. For example, when making a subbuffer, the timing metadata needs to be reset to NONE when the start offset is different. Serialization ~~~~~~~~~~~~~ When buffer should be sent over the wire or be serialized in GDP, we need a way to perform custom serialization and deserialization on the metadata. For this we add the serialize and deserialize functions to the metadata info. Possible use cases are to make sure we write out the fields with a specific size and endianness. Transformations ~~~~~~~~~~~~~~~ After certain transformations, the metadata on a buffer might not be relevant anymore. Consider, for example, metadata that lists certain regions of interest on the video data. If the video is scaled or rotated, the coordinates might not make sense anymore. A transform element should be able to adjust or remove the associated metadata when it becomes invalid. We can make the transform element aware of the metadata so that it can adjust or remove in an intelligent way. Since we allow arbitrary metadata, we can't do this for all metadata and thus we need some other way. One proposition is to tag the metadata type with keywords that specify what it functionally refers too. We could, for example, tag the metadata for the regions of interest with a tag that notes that the metadata refers to absolute pixel positions. A transform could then know that the metadata is not valid anymore when the position of the pixels changed (due to rotation, flipping, scaling and so on). Other use cases ~~~~~~~~~~~~~~~ Making the GstBufferMetaMemory (for making the buffer point to the associated memory region) as metadata on a GstBuffer, as opposed to making it an integral part of GstBuffer, allows for some more interesting ways to transfer data. We could for example make a new GstBufferMetaIOVec metadata structure like this: struct _GstBufferMetaIOVec { GstBufferMeta meta; /* pointer to data and its size */ GFreeFunc data_free; gpointer data_user; guint len; struct iovec *iov; }; This would allow us to transfer data in a scatter/gather array. Since the fields in the buffer metadata are now explicit, elements that don't support this kind of metadata can gracefully degrade. Another use case for not having the Memory metadata in the buffers would be for _pad_alloc() and get_range(). We can pass a GstBuffer with the requested metadata fields to those functions and have the _get_range() or pad_alloc() implementations add (or use, in the case of a file reader) the memory metadata. Relationship with GstCaps ~~~~~~~~~~~~~~~~~~~~~~~~~ The difference between GstCaps, used in negotiation, and the metadata is not clearly defined. We would like to think of the GstCaps containing the information needed to functionally negotiate the format between two elements. The Metadata should then only contain variables that can change between each buffer. For example, for video we would have width/height/framerate in the caps but then have the more technical details, such as stride, data pointers, pan/crop/zoom etc in the metadata. A scheme like this would still allow us to functionally specify the desired video resolution while the implementation details would be inside the metadata. Compatibility ~~~~~~~~~~~~~ We need to make sure that elements exchange metadata that they both understand, This is particulary important when the metadata describes the data layout in memory (such as strides). Currently the only way to communicate buffer formats between elements is by using caps. We would like to use the caps system to negotiate the metadata that will be put on buffers. We would like to add to the caps on the buffer (and pad) an array of metadata structures (as strings) that is on the buffer. This way, an element can quickly know what metadata to look for. In order to remain compatibility with older plugins we need to convert buffers that use metadata to specify a non-standard data layout to the old format. We need to do this before handing buffers to old elements. We will require elements that are metadata aware to set a flag on their pads; any buffer passed on that pad will be converted to the old layout when the flag is not set. Notes ~~~~~ Some structures that we need to be able to add to buffers. * Clean Aperture * Arbitrary Matrix Transform * Aspect ratio * Pan/crop/zoom * Video strides Some of these overlap, we need to find a minimal set of metadata structures that allows us to define all use cases. Video Buffers ------------- #define GST_VIDEO_MAX_PLANES 4 struct GstVideoPlane { guint8 *data; guint size; guint stride; guint8 *data_orig; guint size_orig; GFreeFunc data_free; gpointer data_user; }; struct GstBufferVideoMeta { GstBufferMeta meta GstBufferVideoFlags flags guint n_planes; GstVideoPlane plane[GST_VIDEO_MAX_PLANES]; };