gstreamer/docs/design/draft-buffer2.txt

GstBuffer^2
-----------

This draft document describes a possible design for arbitrary per-buffer
metadata.

The proposed changes in this document are not ABI/API compatible with the 0.10
version of GStreamer and should thus only be considered for upcomming unstable
versions.

Buffer metadata typically includes properties that give more information about
the buffer contents. These properties are usually not negotiated and are thus
not inside the caps.

Some examples of metadata:

 - timestamp, duration
 - offset, offset_end
 - interlacing information
 - video alignment, cropping, panning information
 - extra container information such as granulepos, ...
 - extra global buffer properties


Requirements
~~~~~~~~~~~~

 - It must be fast
    * allocation, free, low fragmentation
    * access to the metadata fields, preferably not much slower than directly
      accessing a C structure field
 - It must be extensible. Elements should be able to add new arbitrary metadata
   without requiring much effort. Also new metadata fields should not break API
   or ABI.
 - It plays nice with subbuffers. When a subbuffer is created, the various
   buffer metadata should be copied/updated correctly.
 - We should be able to pass metadata in pad_alloc() and get_range() functions
   to specify extra allocation parameters.
 - We should be able to attach statically allocated metadata to a buffer. This
   is for metadata that does not change much.

Use cases
---------

 * DSP vs CPU caches

 Both DSP and CPU have separate MMUs and memory caches. When we exchange buffers
 between two subsystems we need to flush caches so that one CPU can see the
 modifications done by the other CPU. These cashes must only be flushed when one
 CPU performed a write and the other CPU needs to do a read.

 In order to implement this we need to be able to mark our read and write
 operations on the buffer data.



GstMiniObject
~~~~~~~~~~~~~

We make GstMiniObject a simple refcounted C structure and also a GLib boxed
type. The following fields will be in the structure:

struct _GstMiniObject {
  GType type;

  /*< public >*/ /* with COW */
  /* refcounting */
  gint       refcount;
  guint      flags;

  GstMiniObjectCopyFunction copy;
  GstMiniObjectFreeFunction free;
}

We will use the regular GSlice allocator or custom object pooling for allocating
instances of the mini object.

We use the well known refcounting mechanisms to manage the lifetime of the
objects.


GstEvent, GstCaps, GstQuery, GstMessage
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Have the new GstMiniObject be the first field in these objects. They will probably
also replace the copy and free functions with their own implementations.

Allocation of the objects will use the regular gst_*_new() functions that will
allocate and initialize a parent GstMiniObject of the required size and setting up
the custom functions.


GstBuffer
~~~~~~~~~

A GstMiniObject will be the parent instance of the GstBuffer object, which is a
regular C structure.

struct _GstBuffer {
  GstMiniObject          mini_object;

  gsize                  free_size;

  GstCaps               *caps;
  GstBuffer             *parent;

  GstBufferDataMap       mmap_func;
  GstBufferDataUnmap     munmap_func;

  gpointer _gst_padding[10];
};

The Buffer object will contain a pointer to the parent buffer to allow for subbuffers
as a first class feature of a GstBuffer.

Allocation of the GstBuffer structure will result in the allocation of a memory region
of a customizable size (512 bytes). Only the first sizeof (GstBuffer) bytes of this
region will initially be used. The remaining bytes will be part of the free metadata
region of the buffer. The size of the free region is kept in the free_size field.

Buffers point to a GstCaps structure that contains the caps of the buffer data.

We add two functions, map and unmap. We introduce explicit read and write transactions
on the buffer data in order to be able to implement flushing of the data between
different caches and mapping between different MMUs.

  typedef enum {
    GST_BUFFER_MAP_NONE,
    GST_BUFFER_MAP_READ,
    GST_BUFFER_MAP_WRITE,
  } GstBufferMapFlags

  gpointer gst_buffer_map   (GstBuffer *, guint offset, guint *size, GstBufferMapFlags);
  gboolean gst_buffer_unmap (GstBuffer *, gpointer data,  guint size);

The map functions always create a contiguous memory space for the requested
area of the buffer. This might involve memcpy and other expensive operations. It
is assumed that more specialized elements can deal with the different memory
metadata structures in order to optimize access to the memory.


GstBufferMeta
~~~~~~~~~~~~~

A GstBufferMeta is a structure as follows:

  struct _GstBufferMeta {
    GstBufferMetaInfo *info;    /* tag and info for the meta item */
  };

The purpose of the this structure is to serve as a common header for all metadata
information that we can attach to a buffer. Specific metadata, such as timing metadata,
will have this structure as the first field. For example:

  struct _GstBufferMetaTiming {
    GstBufferMeta  meta;        /* common meta header */
 
    GstClockTime   dts;         /* decoding timestamp */
    GstClockTime   pts;         /* presentation timestamp */
    GstClockTime   duration;    /* duration of the data */
    GstClockTime   clock_rate;  /* clock rate for the above values */
  };

Or another example for the buffer memory region

  struct _GstBufferMetaMemory {
    GstBufferMeta      meta;
   
    /* pointer to data and its size */
    gpointer          *data;
    guint              size;
    gpointer          *data_orig;
    GFreeFunc          data_free;
    gpointer           data_user;
  };

GstBufferMetaInfo will point to more information about the metadata and looks like this:

  struct _GstBufferMetaInfo {
    GQuark                     api;       /* api name */
    GQuark                     impl;      /* implementation name */
    gsize                      size;      /* size of the structure */
    GstBufferMetaInfo         *parent     /* parent info */

    GstMetaInitFunction        init_func;
    GstMetaFreeFunction        free_func;
    GstMetaCopyFunction        copy_func;
    GstMetaSubFunction         sub_func;
    GstMetaSerializeFunction   serialize_func
    GstMetaDeserializeFunction deserialize_func
    GstMetaConvFunction        conv_func;
  };

Tag will contain a GQuark of the metadata name. We will be able to refer to specific
metadata by name or by its (cached) GQuark.  A repository of registered MetaInfo
will be maintained by the core. We will register some common metadata structures
in core and some media specific info for audio/video/text in -base. Plugins can
register additional custom metadata.

Along with the metadata description we will have functions to initialize/free (and/or refcount)
a specific GstBufferMeta instance. We also have the possibility to add a custom subbuffer
function that can be used to modify the metadata when a subbuffer is taken.

We also add serialize and deserialize function for the metadata in case we need special
logic for reading and writing the metadata. This is needed for GDP payloading of the
metadata.

We add a conv function to the Info structure that will be called when a buffer
should be converted to an old-style buffer for backward compatibility.

The purpose of the separate MetaInfo is to not have to carry the free/init functions in
each buffer instance but to define them globally. We still want quick access to the info
so we need to make the buffer metadata point to the info.

Technically we could also specify the field and types in the MetaInfo and
provide a generic API to retrieve the metadata fields without the need for a
header file. We will not do this yet.
 
The complete buffer with metadata would then look as follows:

                         +-------------------------------------+
GstMiniObject            |     GType (GstBuffer)               |
                         |     refcount, flags, copy/free      |
                         +-------------------------------------+
GstBuffer                |     caps, parent, subfunc           |
                         +.....................................+
                         |     next                           ---+
                         |     parent                         ------> NULL
                      +- |     info                           ------> GstBufferMetaInfo
GstBufferMetaTiming   |  |                                     | |
                      |  |     dts                             | |
                      |  |     pts                             | |
                      |  |     duration                        | |
                      +- |     clock_rate                      | |
                         + . . . . . . . . . . . . . . . . . . + |
                         |     next                           <--+
                         |     parent                         ------> NULL
GstBufferMetaMemory   +- |     info                           ------> GstBufferMetaInfo
                      |  |                                     | |
                      |  |     data                            | |
                      |  |     size                            | |
                      |  |     mallocdata                      | |
                      |  |     data_free                       | |
                      +- |     data_user                       | |
                         + . . . . . . . . . . . . . . . . . . + .
                         .                                       .


API examples
~~~~~~~~~~~~

Buffers are created using the normal gst_buffer_new functions. The standard fields
are initialized as usual. A memory area that is bigger than the structure size
is allocated for the buffer metadata. The remaining free area is stored in the
free_size field.

  gst_buffer_new ();

After creating a buffer, the application can set caps. and add other metadata
information. 

In order to modify metadata, a reference to the MetaInfo should be obtained.
This can be done like this:

  GstBufferMetaInfo *info;

  info = gst_buffer_meta_get_info (GQuark tag);

Usually the info will be obtained only once in order to avoid lock contention on
the global pool of meta info. The core will also provide convenience functions
for the core metainfo.

Once a reference to the info has been obtained, the associated metadata can be
added or modified on a buffer.

For example, to modify the timing info on a buffer, one could use the following
sequence:

  GstBufferMetaInfo *info;
  GstBufferMetaTiming *timing;

  info = gst_buffer_meta_get_info (GST_META_TIMING_QUARK);
  
  timing = gst_buffer_get_meta (buffer, info, TRUE); /* TRUE = create if absent */
  timing->timestamp = 0;
  timing->duration = 20 * GST_MSECOND;

The _get_meta() function returns a pointer to the metadata structure associated
with the GST_META_TIMING_QUARK info.

For the core meta info, we will provide convenience code that uses the cached
GstBufferMetaInfo, making the above code a little more simple.

  GstBufferMetaTiming *timing;

  timing = gst_buffer_get_meta_timing (buffer, TRUE); /* TRUE = create if absent */
  timing->timestamp = 0;
  timing->duration = 20 * GST_MSECOND;
 
Note that for each of the metadata that we will add to buffers, we need a struct
definition and a registered MetaInfo. 


We will also provide an API to iterate the different metainfo structures. A
possible simple API would look like this:

 GstBufferMeta *current = NULL;

 /* passing NULL gives the first entry */ 
 current = gst_buffer_meta_get_next (buffer, current);

 /* passing a GstBufferMeta returns the next */
 current = gst_buffer_meta_get_next (buffer, current);



Memory management
~~~~~~~~~~~~~~~~~

* allocation

  We will initially allocate a reasonable sized GstBuffer structure (say 512
  bytes) and we will set the free_size to the maximum amount of metadata we can
  store.

  Since the complete buffer structure, including a large area for metadata, is
  allocated in one go, we can reduce the number of memory allocations while still
  providing dynamic metadata.

  When adding metadata, we need to call the init function of the associated
  metadata info structure. Since adding the metadata requires the caller to pass
  a handle to the info, this operation does not require table lookups.

  Per-metadata memory initialisation is needed because not all metadata is
  initialized in the same way. We need to, for example, set the timestamps to
  NONE in the MetaTiming structures.

  The init/free functions can also be used to implement refcounting for a metadata
  structure. This can be useful when a structure is shared between buffers.

  When the free_size of the GstBuffer is exhausted, we will allocate new memory
  for each newly added BufferMeta and use the next pointers to point to this. It
  is expected that this does not occur often and we might be able to optimize
  this transparently in the future.

* free

  When a GstBuffer is freed, we potentially might have to call a custom free
  function on the metadata info. In the case of the Memory metadata, we need to
  call the associated free function to free the memory.
  
  When freeing a GstBuffer, the custom buffer free function will iterate all of
  the metadata in the buffer and call the associated free functions in the
  MetaInfo associated with the entries. Usually, this function will be NULL.


Subbuffers
~~~~~~~~~~

Subbuffers are a first class feature of the GstBuffer. 

Creating a subbuffer from a GstBuffer will allocate a new GstBuffer and ref the
parent buffer. It will then iterate all of the metadata entries for the parent
buffer and call the associated sub_func in the MetaInfo.

This allows each metadata structure to implement the actions needed to update
the metadata of the subbuffer. 

A pointer to the old and new memory location of the metadata is passed to the
sub_func. The default implementation will simply copy the metadata. Custom
implementations can adjust the values. For example, when making a subbuffer, the
timing metadata needs to be reset to NONE when the start offset is different.


Serialization
~~~~~~~~~~~~~

When buffer should be sent over the wire or be serialized in GDP, we need a way
to perform custom serialization and deserialization on the metadata.

For this we add the serialize and deserialize functions to the metadata info.
Possible use cases are to make sure we write out the fields with a specific size
and endianness.


Transformations
~~~~~~~~~~~~~~~

After certain transformations, the metadata on a buffer might not be relevant
anymore.

Consider, for example, metadata that lists certain regions of interest
on the video data. If the video is scaled or rotated, the coordinates might not
make sense anymore. A transform element should be able to adjust or remove the
associated metadata when it becomes invalid. 

We can make the transform element aware of the metadata so that it can adjust or
remove in an intelligent way. Since we allow arbitrary metadata, we can't do
this for all metadata and thus we need some other way.

One proposition is to tag the metadata type with keywords that specify what it
functionally refers too. We could, for example, tag the metadata for the regions
of interest with a tag that notes that the metadata refers to absolute pixel
positions. A transform could then know that the metadata is not valid anymore
when the position of the pixels changed (due to rotation, flipping, scaling and
so on).


Other use cases
~~~~~~~~~~~~~~~

Making the GstBufferMetaMemory (for making the buffer point to the associated
memory region) as metadata on a GstBuffer, as opposed to making it an integral
part of GstBuffer, allows for some more interesting ways to transfer data.

We could for example make a new GstBufferMetaIOVec metadata structure like this:

  struct _GstBufferMetaIOVec {
    GstBufferMeta  meta;
   
    /* pointer to data and its size */
    GFreeFunc       data_free;
    gpointer        data_user;
    guint           len;
    struct iovec   *iov;
  };

This would allow us to transfer data in a scatter/gather array. Since the fields
in the buffer metadata are now explicit, elements that don't support this kind
of metadata can gracefully degrade.

Another use case for not having the Memory metadata in the buffers would be for
_pad_alloc() and get_range(). We can pass a GstBuffer with the requested
metadata fields to those functions and have the _get_range() or pad_alloc()
implementations add (or use, in the case of a file reader) the memory metadata. 


Relationship with GstCaps
~~~~~~~~~~~~~~~~~~~~~~~~~

The difference between GstCaps, used in negotiation, and the metadata is not
clearly defined. 

We would like to think of the GstCaps containing the information needed to
functionally negotiate the format between two elements. The Metadata should then
only contain variables that can change between each buffer.

For example, for video we would have width/height/framerate in the caps but then
have the more technical details, such as stride, data pointers, pan/crop/zoom
etc in the metadata.

A scheme like this would still allow us to functionally specify the desired
video resolution while the implementation details would be inside the metadata.


Compatibility
~~~~~~~~~~~~~

We need to make sure that elements exchange metadata that they both understand,
This is particulary important when the metadata describes the data layout in
memory (such as strides).

Currently the only way to communicate buffer formats between elements is by
using caps. We would like to use the caps system to negotiate the metadata that
will be put on buffers.

We would like to add to the caps on the buffer (and pad) an array of metadata
structures (as strings) that is on the buffer. This way, an element can
quickly know what metadata to look for.

In order to remain compatibility with older plugins we need to convert buffers
that use metadata to specify a non-standard data layout to the old format. We
need to do this before handing buffers to old elements. We will require elements
that are metadata aware to set a flag on their pads; any buffer passed on that
pad will be converted to the old layout when the flag is not set.


Notes
~~~~~

Some structures that we need to be able to add to buffers.

* Clean Aperture
* Arbitrary Matrix Transform
* Aspect ratio
* Pan/crop/zoom
* Video strides

Some of these overlap, we need to find a minimal set of metadata structures that
allows us to define all use cases.



Video Buffers
-------------

 #define GST_VIDEO_MAX_PLANES 4

 struct GstVideoPlane {
   guint8             *data;
   guint               size;
   guint               stride;

   guint8             *data_orig;
   guint               size_orig;
   GFreeFunc           data_free;
   gpointer            data_user;
 };

 struct GstBufferVideoMeta {
   GstBufferMeta       meta

   GstBufferVideoFlags flags

   guint               n_planes;
   GstVideoPlane       plane[GST_VIDEO_MAX_PLANES];
 };