design: update design docs

docs/design/part-buffer.txt

@@ -0,0 +1,150 @@
+GstBuffer
+---------
+
+This document describes the design for buffers.
+
+A GstBuffer is the object that is passed from an upstream element to a
+downstream element and contains memory and metadata information.
+
+Requirements
+~~~~~~~~~~~~
+
+ - It must be fast
+    * allocation, free, low fragmentation
+ - Must be able to attach multiple memory blocks to the buffer
+ - Must be able to attach artibtrary metadata to buffers
+ - efficient handling of subbuffer, copy, span, trim
+
+Writability
+-----------
+
+The Buffers is writable when the refcount is 1. This means that:
+
+ - metadata can be added/removed and the metadata can be changed
+ - GstMemory blocks can be added/removed
+
+The individual memory blocks have their own refcounting and READONLY flags
+that might influence their writability.
+
+Buffers can be made writable with gst_buffer_make_writable(). This will copy the
+buffer with the metadata and will ref the memory in the buffer. This means that
+the memory is not automatically copied when copying buffers.
+
+
+Managing GstMemory
+------------------
+
+A GstBuffer contains an array of pointers to GstMemory objects. 
+
+When the buffer is writable, gst_buffer_take_memory() can be used to add a
+new GstMemory object to the buffer. When the array of memory is full, memory
+will be merged to make room for the new memory object.
+
+gst_buffer_n_memory() is used to get the amount of memory blocks on the
+GstBuffer.
+
+With gst_buffer_peek_memory(), memory can be retrieved from the memory array.
+The desired access pattern for the memory block should be specified so that
+appropriate checks can be made and, in case of GST_MAP_WRITE, a writable copy
+can be constructed when needed.
+
+gst_buffer_remove_memory_range() and gst_buffer_remove_memory() can be used to
+remove memory from the GstBuffer.
+
+
+Subbuffers
+----------
+
+Subbuffers are made by copying only a region of the memory blocks and copying
+all of the metadata.
+
+
+Span
+----
+
+Spanning will merge together the data of 2 buffers into a new buffer
+
+
+Data access
+-----------
+
+ Accessing the data of the buffer can happen by retrieving the individual
+ GstMemory objects in the GstBuffer or my using the gst_buffer_map() and
+ gst_buffer_unmap() function.
+
+ The _map and _unmap function will always return the memory of all blocks as one
+ large contiguous region of memory. Using the _map and _unmap function might be
+ more convenient that accessing the individual memory blocks at the expense of
+ being more expensive because it might perform memcpy operations.
+
+ For buffers with only one GstMemory object (the most common case), _map and
+ _unmap have no performance penalty at all.
+
+
+* Read access with 1 memory block
+
+  The memory block is accessed and mapped for read access.
+  The memory block is unmapped after usage
+
+* write access with 1 memory block
+
+  The buffer should be writable or this operation will fail..
+  The memory block is accessed. If the memory block is readonly, a copy is made
+  and the original memory block is replaced with this copy. then the memory
+  block is mapped in write mode.
+  The memory block is unmapped after usage.
+
+* Read access with multiple memory blocks
+
+  The memory blocks are combined into one large memory block. If the buffer is
+  writable, The memory blocks are replace with this new memory block. If the
+  buffer is not writable, the memory is returned as is.
+  The memory block is then mapped in read mode.
+
+  When the memory is unmapped after usage and the buffer has multiple memory
+  blocks, this means that the map operation was not able to store the combined
+  buffer and it thus returned memory that should be freed. Otherwise, the memory
+  is unmapped.
+
+* Write access with multiple memory blocks
+
+  The buffer should be writable or the operation fails. The memory blocks are
+  combined into one large memory block and the existing blocks are replaced with
+  this new block. The memory is then mapped in write mode.
+  The memory is unmapped after usage.
+
+
+Use cases
+---------
+
+Generating RTP packets from h264 video
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+We receive as input a GstBuffer with an encoded h264 image and we need to
+create RTP packets containing this h264 data as the payload. We typically need
+to fragment the h264 data into multiple packets, each with their own RTP and
+payload specific header.
+
+                       +-------+-------+---------------------------+--------+
+  input H264 buffer:   | NALU1 | NALU2 |  .....                    | NALUx  |
+                       +-------+-------+---------------------------+--------+
+                             |
+                             V
+  array of             +-+ +-------+  +-+ +-------+            +-+ +-------+
+  output buffers:      | | | NALU1 |  | | | NALU2 |   ....     | | | NALUx |
+                       +-+ +-------+  +-+ +-------+            +-+ +-------+
+                       :           :  :           :
+                       \-----------/  \-----------/
+                         buffer 1        buffer 2
+
+The output buffer array consists of x buffers consisting of an RTP payload header
+and a subbuffer of the original input H264 buffer. Since the rtp headers and
+the h264 data don't need to be contiguous in memory, they are added to the buffer
+as separate GstMemory blocks and we can avoid to memcpy the h264 data into
+contiguous memory.
+
+A typical udpsink will then use something like sendmsg to send the memory regions
+on the network inside one UDP packet. This will further avoid having to memcpy
+data into contiguous memory.
+
+

docs/design/part-bufferlist.txt

@@ -87,7 +87,7 @@ The metadata of the bufferlist is always the metadata of the first buffer of the
 first group in the bufferlist. This means that:
 
   - Before pushing the list to a pad, negotiation happens with (only) the caps of
-    the first buffer in the list. Caps of other buffers is ignore.
+    the first buffer in the list. Caps of other buffers is ignored.
 
   - synchronisation happens on the timestamp of the first buffer in the list.