mirror of
https://gitlab.freedesktop.org/gstreamer/gstreamer.git
synced 2024-09-08 13:18:52 +00:00
450 lines
16 KiB
Text
450 lines
16 KiB
Text
GstBuffer^2
|
|
-----------
|
|
|
|
This draft document describes a possible design for arbitrary per-buffer
|
|
metadata.
|
|
|
|
The proposed changes in this document are not ABI/API compatible with the 0.10
|
|
version of GStreamer and should thus only be considered for upcomming unstable
|
|
versions.
|
|
|
|
Buffer metadata typically includes properties that give more information about
|
|
the buffer contents. These properties are usually not negotiated and are thus
|
|
not inside the caps.
|
|
|
|
Some examples of metadata:
|
|
|
|
- timestamp, duration
|
|
- offset, offset_end
|
|
- interlacing information
|
|
- video alignment, cropping, panning information
|
|
- extra container information such as granulepos, ...
|
|
- extra global buffer properties
|
|
|
|
|
|
Requirements
|
|
------------
|
|
|
|
- It must be fast
|
|
* allocation, free, low fragmentation
|
|
* access to the metadata fields, preferably not much slower than directly
|
|
accessing a C structure field
|
|
- It must be extensible. Elements should be able to add new arbitrary metadata
|
|
without requiring much effort. Also new metadata fields should not break API
|
|
or ABI.
|
|
- It plays nice with subbuffers. When a subbuffer is created, the various
|
|
buffer metadata should be copied/updated correctly.
|
|
- We should be able to pass metadata in pad_alloc() and get_range() functions
|
|
to specify extra allocation parameters.
|
|
- We should be able to attach statically allocated metadata to a buffer. This
|
|
is for metadata that does not change much.
|
|
|
|
|
|
GstMiniObject
|
|
-------------
|
|
|
|
We make GstMiniObject a simple refcounted C structure and also a GLib boxed
|
|
type. The following fields will be in the structure:
|
|
|
|
struct _GstMiniObject {
|
|
GType type;
|
|
|
|
/*< public >*/ /* with COW */
|
|
/* refcounting */
|
|
gint refcount;
|
|
guint flags;
|
|
|
|
GstMiniObjectCopyFunction copy;
|
|
GstMiniObjectFreeFunction free;
|
|
}
|
|
|
|
We will use the regular GSlice allocator or custom object pooling for allocating
|
|
instances of the mini object.
|
|
|
|
We use the well known refcounting mechanisms to manage the lifetime of the
|
|
objects.
|
|
|
|
|
|
GstEvent, GstCaps, GstQuery, GstMessage
|
|
---------------------------------------
|
|
|
|
Have the new GstMiniObject be the first field in these objects. They will probably
|
|
also replace the copy and free functions with their own implementations.
|
|
|
|
Allocation of the objects will use the regular gst_*_new() functions that will
|
|
allocate and initialize a parent GstMiniObject of the required size and setting up
|
|
the custom functions.
|
|
|
|
|
|
GstBuffer
|
|
---------
|
|
|
|
A GstMiniObject will be the parent instance of the GstBuffer object, which is a
|
|
regular C structure.
|
|
|
|
struct _GstBuffer {
|
|
GstMiniObject mini_object;
|
|
|
|
gsize free_size;
|
|
|
|
GstCaps *caps;
|
|
GstBuffer *parent;
|
|
|
|
gpointer _gst_padding[10];
|
|
};
|
|
|
|
The Buffer object will contain a pointer to the parent buffer to allow for subbuffers
|
|
as a first class feature of a GstBuffer.
|
|
|
|
Allocation of the GstBuffer structure will result in the allocation of a memory region
|
|
of a customizable size (512 bytes). Only the first sizeof (GstBuffer) bytes of this
|
|
region will initially be used. The remaining bytes will be part of the free metadata
|
|
region of the buffer. The size of the free region is kept in the free_size field.
|
|
|
|
Buffers point to a GstCaps structure that contains the caps of the buffer data.
|
|
|
|
|
|
GstBufferMeta
|
|
-------------
|
|
|
|
A GstBufferMeta is a structure as follows:
|
|
|
|
struct _GstBufferMeta {
|
|
GstBufferMetaInfo *info; /* tag and info for the meta item */
|
|
GstBufferMeta *next; /* pointer to the next item */
|
|
}
|
|
|
|
The purpose of the this structure is to serve as a common header for all metadata
|
|
information that we can attach to a buffer. Specific metadata, such as timing metadata,
|
|
will have this structure as the first field. For example:
|
|
|
|
struct _GstBufferMetaTiming {
|
|
GstBufferMeta meta; /* common meta header */
|
|
|
|
GstClockTime dts; /* decoding timestamp */
|
|
GstClockTime pts; /* presentation timestamp */
|
|
GstClockTime duration; /* duration of the data */
|
|
GstClockTime clock_rate; /* clock rate for the above values */
|
|
};
|
|
|
|
Or another example for the buffer memory region
|
|
|
|
struct _GstBufferMetaMemory {
|
|
GstBufferMeta meta;
|
|
|
|
/* pointer to data and its size */
|
|
guint8 *data;
|
|
guint size;
|
|
guint8 *malloc_data;
|
|
GFreeFunc data_free;
|
|
gpointer data_user;
|
|
};
|
|
|
|
GstBufferMetaInfo will point to more information about the metadata and looks like this:
|
|
|
|
struct _GstBufferMetaInfo {
|
|
GQuark tag; /* tag name */
|
|
gsize size; /* size of the structure */
|
|
|
|
GstMetaInitFunction init_func;
|
|
GstMetaFreeFunction free_func;
|
|
GstMetaSubFunction sub_func;
|
|
GstMetaSerializeFunction serialize_func
|
|
GstMetaDeserializeFunction deserialize_func
|
|
}
|
|
|
|
Tag will contain a GQuark of the metadata name. We will be able to refer to specific
|
|
metadata by name or by its (cached) GQuark. A repository of registered MetaInfo
|
|
will be maintained by the core. We will register some common metadata structures
|
|
in core and some media specific info for audio/video/text in -base. Plugins can
|
|
register additional custom metadata.
|
|
|
|
Along with the metadata description we will have functions to initialize/free (and/or refcount)
|
|
a specific GstBufferMeta instance. We also have the possibility to add a custom subbuffer
|
|
function that can be used to modify the metadata when a subbuffer is taken.
|
|
|
|
We also add serialize and deserialize function for the metadata in case we need special
|
|
logic for reading and writing the metadata. This is needed for GDP payloading of the
|
|
metadata.
|
|
|
|
The purpose of the separate MetaInfo is to not have to carry the free/init functions in
|
|
each buffer instance but to define them globally. We still want quick access to the info
|
|
so we need to make the buffer metadata point to the info.
|
|
|
|
Technically we could also specify the field and types in the MetaInfo and
|
|
provide a generic API to retrieve the metadata fields without the need for a
|
|
header file. We will not do this yet.
|
|
|
|
The complete buffer with metadata would then look as follows:
|
|
|
|
+-------------------------------------+
|
|
GstMiniObject | GType (GstBuffer) |
|
|
| refcount, flags, copy/free |
|
|
+-------------------------------------+
|
|
GstBuffer | caps, parent, subfunc |
|
|
+.....................................+
|
|
+- | info ------> GstBufferMetaInfo
|
|
GstBufferMetaTiming | | next ---+
|
|
| | | |
|
|
| | dts | |
|
|
| | pts | |
|
|
| | duration | |
|
|
+- | clock_rate | |
|
|
+ . . . . . . . . . . . . . . . . . . + |
|
|
+- | info <--+ -> GstBufferMetaInfo
|
|
GstBufferMetaMemory | | next ---+
|
|
| | | |
|
|
| | data | |
|
|
| | size | |
|
|
| | mallocdata | |
|
|
| | data_free | |
|
|
+- | data_user | |
|
|
+ . . . . . . . . . . . . . . . . . . + .
|
|
. .
|
|
|
|
API examples
|
|
------------
|
|
|
|
Buffers are created using the normal gst_buffer_new functions. The standard fields
|
|
are initialized as usual. A memory area that is bigger than the structure size
|
|
is allocated for the buffer metadata. The remaining free area is stored in the
|
|
free_size field.
|
|
|
|
gst_buffer_new ();
|
|
|
|
After creating a buffer, the application can set caps. and add other metadata
|
|
information.
|
|
|
|
In order to modify metadata, a reference to the MetaInfo should be obtained.
|
|
This can be done like this:
|
|
|
|
GstBufferMetaInfo *info;
|
|
|
|
info = gst_buffer_meta_get_info (GQuark tag);
|
|
|
|
Usually the info will be obtained only once in order to avoid lock contention on
|
|
the global pool of meta info. The core will also provide convenience functions
|
|
for the core metainfo.
|
|
|
|
Once a reference to the info has been obtained, the associated metadata can be
|
|
added or modified on a buffer.
|
|
|
|
For example, to modify the timing info on a buffer, one could use the following
|
|
sequence:
|
|
|
|
GstBufferMetaInfo *info;
|
|
GstBufferMetaTiming *timing;
|
|
|
|
info = gst_buffer_meta_get_info (GST_META_TIMING_QUARK);
|
|
|
|
timing = gst_buffer_get_meta (buffer, info, TRUE); /* TRUE = create if absent */
|
|
timing->timestamp = 0;
|
|
timing->duration = 20 * GST_MSECOND;
|
|
|
|
The _get_meta() function returns a pointer to the metadata structure associated
|
|
with the GST_META_TIMING_QUARK info.
|
|
|
|
For the core meta info, we will provide convenience code that uses the cached
|
|
GstBufferMetaInfo, making the above code a little more simple.
|
|
|
|
GstBufferMetaTiming *timing;
|
|
|
|
timing = gst_buffer_get_meta_timing (buffer, TRUE); /* TRUE = create if absent */
|
|
timing->timestamp = 0;
|
|
timing->duration = 20 * GST_MSECOND;
|
|
|
|
Note that for each of the metadata that we will add to buffers, we need a struct
|
|
definition and a registered MetaInfo.
|
|
|
|
|
|
We will also provide an API to iterate the different metainfo structures. A
|
|
possible simple API would look like this:
|
|
|
|
GstBufferMeta *current = NULL;
|
|
|
|
/* passing NULL gives the first entry */
|
|
current = gst_buffer_meta_get_next (buffer, current);
|
|
|
|
/* passing a GstBufferMeta returns the next */
|
|
current = gst_buffer_meta_get_next (buffer, current);
|
|
|
|
|
|
|
|
Memory management
|
|
-----------------
|
|
|
|
* allocation
|
|
|
|
We will initially allocate a reasonable sized GstBuffer structure (say 512
|
|
bytes) and we will set the free_size to the maximum amount of metadata we can
|
|
store.
|
|
|
|
Since the complete buffer structure, including a large area for metadata, is
|
|
allocated in one go, we can reduce the number of memory allocations while still
|
|
providing dynamic metadata.
|
|
|
|
When adding metadata, we need to call the init function of the associated
|
|
metadata info structure. Since adding the metadata requires the caller to pass
|
|
a handle to the info, this operation does not require table lookups.
|
|
|
|
Per-metadata memory initialisation is needed because not all metadata is
|
|
initialized in the same way. We need to, for example, set the timestamps to
|
|
NONE in the MetaTiming structures.
|
|
|
|
The init/free functions can also be used to implement refcounting for a metadata
|
|
structure. This can be useful when a structure is shared between buffers.
|
|
|
|
When the free_size of the GstBuffer is exhausted, we will allocate new memory
|
|
for each newly added BufferMeta and use the next pointers to point to this. It
|
|
is expected that this does not occur often and we might be able to optimize
|
|
this transparently in the future.
|
|
|
|
* free
|
|
|
|
When a GstBuffer is freed, we potentially might have to call a custom free
|
|
function on the metadata info. In the case of the Memory metadata, we need to
|
|
call the associated free function to free the memory.
|
|
|
|
When freeing a GstBuffer, the custom buffer free function will iterate all of
|
|
the metadata in the buffer and call the associated free functions in the
|
|
MetaInfo associated with the entries. Usually, this function will be NULL.
|
|
|
|
|
|
Subbufers
|
|
---------
|
|
|
|
Subbuffers are a first class feature of the GstBuffer.
|
|
|
|
Creating a subbuffer from a GstBuffer will allocate a new GstBuffer and ref the
|
|
parent buffer. It will then iterate all of the metadata entries for the parent
|
|
buffer and call the associated sub_func in the MetaInfo.
|
|
|
|
This allows each metadata structure to implement the actions needed to update
|
|
the metadata of the subbuffer.
|
|
|
|
A pointer to the old and new memory location of the metadata is passed to the
|
|
sub_func. The default implementation will simply copy the metadata. Custom
|
|
implementations can adjust the values. For example, when making a subbuffer, the
|
|
timing metadata needs to be reset to NONE when the start offset is different.
|
|
|
|
|
|
Serialization
|
|
-------------
|
|
|
|
When buffer should be sent over the wire or be serialized in GDP, we need a way
|
|
to perform custom serialization and deserialization on the metadata.
|
|
|
|
For this we add the serialize and deserialize functions to the metadata info.
|
|
Possible use cases are to make sure we write out the fields with a specific size
|
|
and endianness.
|
|
|
|
|
|
Transformations
|
|
---------------
|
|
|
|
After certain transformations, the metadata on a buffer might not be relevant
|
|
anymore.
|
|
|
|
Consider, for example, metadata that lists certain regions of interest
|
|
on the video data. If the video is scaled or rotated, the coordinates might not
|
|
make sense anymore. A transform element should be able to adjust or remove the
|
|
associated metadata when it becomes invalid.
|
|
|
|
We can make the transform element aware of the metadata so that it can adjust or
|
|
remove in an intelligent way. Since we allow arbitrary metadata, we can't do
|
|
this for all metadata and thus we need some other way.
|
|
|
|
One proposition is to tag the metadata type with keywords that specify what it
|
|
functionally refers too. We could, for example, tag the metadata for the regions
|
|
of interest with a tag that notes that the metadata refers to absolute pixel
|
|
positions. A transform could then know that the metadata is not valid anymore
|
|
when the position of the pixels changed (due to rotation, flipping, scaling and
|
|
so on).
|
|
|
|
|
|
Other use cases
|
|
---------------
|
|
|
|
Making the GstBufferMetaMemory (for making the buffer point to the associated
|
|
memory region) as metadata on a GstBuffer, as opposed to making it an integral
|
|
part of GstBuffer, allows for some more interesting ways to transfer data.
|
|
|
|
We could for example make a new GstBufferMetaIOVec metadata structure like this:
|
|
|
|
struct _GstBufferMetaIOVec {
|
|
GstBufferMeta meta;
|
|
|
|
/* pointer to data and its size */
|
|
GFreeFunc data_free;
|
|
gpointer data_user;
|
|
guint len;
|
|
struct iovec *iov;
|
|
};
|
|
|
|
This would allow us to transfer data in a scatter/gather array. Since the fields
|
|
in the buffer metadata are now explicit, elements that don't support this kind
|
|
of metadata can gracefully degrade.
|
|
|
|
Another use case for not having the Memory metadata in the buffers would be for
|
|
_pad_alloc() and get_range(). We can pass a GstBuffer with the requested
|
|
metadata fields to those functions and have the _get_range() or pad_alloc()
|
|
implementations add (or use, in the case of a file reader) the memory metadata.
|
|
|
|
|
|
Relationship with GstCaps
|
|
-------------------------
|
|
|
|
The difference between GstCaps, used in negotiation, and the metadata is not
|
|
clearly defined.
|
|
|
|
We would like to think of the GstCaps containing the information needed to
|
|
functionally negotiate the format between two elements. The Metadata should then
|
|
only contain variables that can change between each buffer.
|
|
|
|
For example, for video we would have width/height/framerate in the caps but then
|
|
have the more technical details, such as stride, data pointers, pan/crop/zoom
|
|
etc in the metadata.
|
|
|
|
A scheme like this would still allow us to functionally specify the desired
|
|
video resolution while the implementation details would be inside the metadata.
|
|
|
|
|
|
Compatibility
|
|
-------------
|
|
|
|
We need to make sure that elements exchange metadata that they both understand,
|
|
This is particulary important when the metadata describes the data layout in
|
|
memory (such as strides).
|
|
|
|
Currently the only way to communicate buffer formats between elements is by
|
|
using caps. We would like to use the caps system to negotiate the metadata that
|
|
will be put on buffers.
|
|
|
|
We would like to add to the caps on the buffer (and pad) an array of metadata
|
|
structures (as strings) that is on the buffer. This way, an element can
|
|
quickly know what metadata to look for.
|
|
|
|
In order to remain compatibility with older plugins we need to convert buffers
|
|
that use metadata to specify a non-standard data layout to the old format. We
|
|
need to do this before handing buffers to old elements. We will require elements
|
|
that are metadata aware to set a flag on their pads; any buffer passed on that
|
|
pad will be converted to the old layout when the flag is not set.
|
|
|
|
|
|
Notes:
|
|
------
|
|
|
|
Some structures that we need to be able to add to buffers.
|
|
|
|
Clean Aperture
|
|
Arbitrary Matrix Transform
|
|
Aspect ratio
|
|
Pan/crop/zoom
|
|
Video strides
|
|
|
|
Some of these overlap, we need to find a minimal set of metadata structures that
|
|
allows us to define all use cases.
|
|
|
|
|
|
|
|
|