gstreamer/docs/design/part-element-transform.txt

Transform elements
------------------

Transform elements transform input buffers to output buffers based
on the sink and source caps. 

An important requirement for a transform is that the ouput caps are completely
defined by the input caps and vice versa. This means that a typical decoder
element can NOT be implemented with a transform element, this is because the
output caps like width and height of the decompessed video frame, for example,
are endcoded in the stream and thus not defined by the input caps.

Typical transform elements include:

 - audio convertors (audioconvert, audioresample,...)
 - video convertors (colorspace, videoscale, ...)
 - filters (capfilter, volume, colorbalance, ...)

The implementation of the transform element has to take care of
the following things:

 - efficient negotiation both up and downstream
 - efficient buffer alloc and other buffer management

Some transform elements can operate in different modes:

 - passthrough (no changes are done on the input buffers)
 - in-place (changes made directly to the incomming buffers without requiring a
   copy or new buffer allocation)
 - metadata changes only 
 
Depending on the mode of operation the buffer allocation strategy might change.

The transform element should at any point be able to renegotiate sink and src
caps as well as change the operation mode.

In addition, the transform element will typically take care of the following
things as well:

 - flushing, seeking
 - state changes
 - timestamping, this is typically done by copying the input timestamps to the
   output buffers but subclasses should be able to override this.
 - QoS, avoiding calls to the subclass transform function
 - handle scheduling issues such as push and pull based operation.

In the next sections, we will describe the behaviour of the transform element in
each of the above use cases. We focus mostly on the buffer allocation strategies
and caps negotiation.

Processing
~~~~~~~~~~

A transform has 2 main processing functions:

 - transform():

   Transform the input buffer to the output buffer. The output buffer is
   guaranteed to be writable and different from the input buffer.

 - transform_ip():

   Transform the input buffer in-place. The input buffer is writable and of
   bigger or equal size than the output buffer. 

A transform can operate in the following modes:

 - passthrough:

   The element will not make changes to the buffers, buffers are pushed straight
   through, caps on both sides need to be the same. The element can optionally
   implement a transform_ip() function to take a look at the data, the buffer
   does not have to be writable.

 - in-place:

   Changes can be made to the input buffer directly to obtain the output buffer.
   The transform must implement a transform_ip() function.

 - copy-transform

   The transform is performed by copying and transforming the input buffer to a
   new output buffer. The transform must implement a transform() function.

When no transform() function is provided, only in-place and passthrough
operation is allowed, this means that source and destination caps must be equal
or that the source buffer size is bigger or equal than the destination buffer.

When no transform_ip() function is provided, only passthrough and
copy-transforms are supported. Providing this function is an optimisation that
can avoid a buffer copy.

When no functions are provided, we can only process in passthrough mode.


Negotiation
~~~~~~~~~~~

Typical (re)negotiation of the transform element in push mode always goes from
sink to src, this means triggers the following sequence:
 
  - the sinkpad receives a buffer with new caps, this triggers the setcaps
    function on the sinkpad before handing the buffer to transform.
  - the transform function figures out what it can convert these caps to.
  - try to see if we can configure the caps unmodified on the peer. We need to
    do this because we prefer to not do anything.
  - the transform configures itself to transform from the new sink caps to the
    target src caps
  - the transform processes and sets the output caps on the src pad

We call this downstream negotiation (DN) and it goes roughly like this:

               sinkpad              transform               srcpad
     setcaps()    |                    |                      |
     ------------>|  find_transform()  |                      |
                  |------------------->|                      |
                  |                    |       setcaps()      |
                  |                    |--------------------->|
                  | <configure caps> <-|                      |


These steps configure the element for a transformation from the input caps to
the output caps.

The transform has 3 function to perform the negotiation:

   - transform_caps():
     
     Transform the caps on a certain pad to all the possible supported caps on
     the other pad. The input caps are guaranteed to be a simple caps with just
     one structure. The caps do not have to be fixed.
      
   - fixate_caps():

     Given a caps on one pad, fixate the caps on the other pad. The target caps
     are writable.

   - set_caps():

     Configure the transform for a transformation between src caps and dest
     caps. Both caps are guaranteed to be fixed caps.

If no transform_caps() is defined, we can only perform the identity transform,
by default. 

If no set_caps() is defined, we don't care about caps. In that case we also
assume nothing is going to write to the buffer and we don't enforce a writable
buffer for the transform_ip function, when present.

One common function that we need for the transform element is to find the best
transform from one format (src) to another (dest). Since the function is
bidirectional, we will use the src->dest negotiation. Some requirements of this
function are:

   - has a fixed src caps
   - finds a fixed dest caps that the transform element can transform to
   - the dest caps are compatible and can be accepted by peer elements
   - the transform function prefers to make src caps == dest caps
   - the transform function can optionally fixate dest caps.

The find_transform() function goes like this:

   - start from src aps, these caps are fixed.
   - check if the caps are acceptable for us as src caps. This is usually
     enforced by the padtemplate of the element.
   - calculate all caps we can transform too with transform_caps()
   - if the original caps are a subset of the transforms, try to see if the
     the caps are acceptable for the peer. If this is possible, we can
     perform passthrough and make src == dest.  This is performed by simply
     calling gst_pad_peer_accept_caps().
   - if the caps are not fixed, we need to fixate it, start by taking the peer
     caps and intersect with them.
   - for each of the transformed caps retrieved with transform_caps():
      - try to fixate the caps with fixate_caps()
      - if the caps are fixated, check if the peer accepts them with
	_peer_accept_caps(), if the peer accepts, we have found a dest caps.
   - if we run out of caps, we fail to find a transform.
   - if we found a destination caps, configure the transform with set_caps().

After this negotiation process, the transform element is usually in a steady
state. We can identify these steady states:

  - src and sink pads both have the same caps. Note that when the caps are equal
    on both pads, the input and output buffers automatically have the same size.
    The element can operate on the buffers in the following ways: (Same caps, SC)

      - passthrough: buffers are inspected but no metadata or buffer data
	is changed. The input buffers don't need to be writable. The input
	buffer is simply pushed out again without modifications. (SCP)

               sinkpad              transform               srcpad
       chain()    |                    |                      |
     ------------>|   handle_buffer()  |                      |
                  |------------------->|      pad_push()      |
                  |                    |--------------------->|
                  |                    |                      |

      - in-place: buffers are modified in-place, this means that the input
	buffer is modified to produce a new output buffer. This requires the
	input buffer to be writable. If the input buffer is not writable, a new
	buffer has to be allocated with pad-alloc. (SCI)

               sinkpad              transform               srcpad
       chain()    |                    |                      |
     ------------>|   handle_buffer()  |                      |
                  |------------------->|                      |
                  |                    |   [!writable]        |
                  |                    |     pad-alloc()      |
                  |                    |--------------------->|
                  |  [caps-changed]  .-|  [caps-changed]      |
                  |   <reconfigure>  | |     setcaps()        |
                  |                  '>|--------------------->|
                  |                  .-|                      |
                  |  <transform_ip>  | |                      |
                  |                  '>|                      |
                  |                    |      pad_push()      |
                  |                    |--------------------->|
                  |                    |                      |

      - copy transform: a new output buffer is allocated with pad-alloc and data
        from the input buffer is transformed into the output buffer. (SCC)

               sinkpad              transform               srcpad
       chain()    |                    |                      |
     ------------>|   handle_buffer()  |                      |
                  |------------------->|                      |
                  |                    |     pad_alloc()      |
                  |                    |--------------------->|
                  |  [caps-changed]  .-|   [caps-changed]     |
                  |   <reconfigure>  | |      setcaps()       |
                  |                  '>|--------------------->|
                  |                  .-|                      |
                  |     <transform>  | |                      |
                  |                  '>|                      |
                  |                    |      pad_push()      |
                  |                    |--------------------->|
                  |                    |                      |

  - src and sink pads have different caps. The element can operate on the
    buffers in the following way: (Different Caps, DC)

      - in-place: input buffers are modified in-place. This means that the input
	buffer has a size that is larger or equal to the output size. The input
	buffer will be resized to the size of the output buffer. If the input
	buffer is not writable or the output size is bigger than the input size,
	we need to pad-alloc a new buffer. (DCI)

               sinkpad              transform               srcpad
       chain()    |                    |                      |
     ------------>|   handle_buffer()  |                      |
                  |------------------->|                      |
                  |                    | [!writable || !size] |
                  |                    |     pad-alloc        |
                  |                    |--------------------->|
                  |  [caps-changed]  .-|  [caps-changed]      |
                  |   <reconfigure>  | |     setcaps()        |
                  |                  '>|--------------------->|
                  |                  .-|                      |
                  |  <transform_ip>  | |                      |
                  |                  '>|                      |
                  |                    |      pad_push()      |
                  |                    |--------------------->|
                  |                    |                      |

      - copy transform: a new output buffer is allocated and the data from the
	input buffer is transformed into the output buffer. The flow is exactly
	the same as the case with the same-caps negotiation. (DCC)
 
We can immeditatly observe that the copy transform states will need to
allocate a buffer from a downstream element using pad-alloc. When the transform
element is receiving a non-writable buffer in the in-place state, it will also
need to perform a pad-alloc. There is no reason why the passthrough state would
perform a pad-alloc. This is important because upstream re-negotiation can only
happen when the transform uses pad-alloc for all outgoing buffers.

This steady state changes when one of the following actions occur:

  - the sink pad receives new caps, this triggers the above downstream
    renegotation process, see above for the flow.
  - the src pad is instructed to produce new caps because of new caps from
    pad-alloc, this only happens when the transform calls pad-alloc on the
    srcpad in order to produce a new output buffer.
  - the transform element wants to renegotiate (because of changed properties,
    for example). This essentially clears the current steady state and
    triggers the downstream and upstream renegotiation process.

Parallel to the downstream negotiation process there is an upstream negotiation
process. The handling and proxy of buffer-alloc is the most comple part of the
transform element. This upstream negotiation process has 3 cases: (UN)

  - upstream calls the buffer-alloc function of the transform sinkpad and this
    call is proxied downstream (UNP)
  - upstream calls the buffer-alloc function of the transform sinkpad, the
    transform does not proxy the call but returns a buffer itself (UNU)
  - the transform calls the pad-alloc function downstream to allocate a new
    output buffer (but not because of a proxied buffer-alloc) (UNA)

The case where the pad-alloc is called because an output buffer must be
generated in the chain function is handled above in the copy-transform and the
in-place transform when the input buffer is not writable or the input buffer
size is smaller than the output size.

We are left with the last case (proxy an incomming pad-alloc or not). We have 2
possibilities here:

   - pad-alloc is called with the same caps as are currently being handled by
     the transform on the sinkcaps. Note that this will only be true when the
     transform element is completely negotiated because of data processing, see
     above. Then the element is not yet negotiated, we proceed with the case
     where sinkcaps are different from thos in the buffer-alloc.
     
     * If the transform is using copy-transform, we don't need to proxy because
       we will call pad-alloc when generating an output buffer. 

               sinkpad              transform               srcpad
   buffer_alloc() |                    |                      |
  --------------->|                    |                      |
                  |                    |                      |
                  |-. [same caps &&    |                      |
   return default | |  copy-trans]     |                      |
     <------------|<'                  |                      |
                  |                    |                      |

     * If the transform is using in-place and insize < outsize, we proxy
       the pad-alloc with the srccaps. If the caps are unmodified, we proxy
       the buffer after changing the caps and size. 

               sinkpad              transform               srcpad
   buffer_alloc() |                    |                      |
  --------------->|                    |                      |
                  |   [same caps &&    |                      |
                  |    in-place]       |                      |
                  |------------------->|    pad_alloc()       |
                  |                    |--------------------->|
                  | [caps unchanged]   |                      |
      return      |  adjust_buffer     |                      |
    <----------------------------------|                      |
                  |                    |                      |
                  |                    |                      |

     * If the transform is using in-place and insize < outsize, we proxy
       the pad-alloc with the srccaps. If the caps are modified find the best
       transform from these new caps and return a buffer of this size/caps
       instead. 

               sinkpad              transform               srcpad
   buffer_alloc() |                    |                      |
  --------------->|                    |                      |
                  |   [same caps &&    |                      |
                  |    in-place]       |    pad-alloc()       |
                  |------------------------------------------>|
                  | [caps changed]   .-|                      |
                  | find_transform() | |                      |
      return      |                  '>|                      |
    <----------------------------------|                      |
                  |                    |                      |

     * If the transform is using in-place and insize >= outsize, we cannot proxy
       the pad-alloc because the resulting buffer would be too small to return
       anyway. 

     * If the transform is using passthrough, we can proxy the pad-alloc to the
       source pad. If the caps change, find the best transform and return a
       buffer of those caps and size instead.

               sinkpad              transform               srcpad
   buffer_alloc() |                    |                      |
  --------------->| [same caps &&      |                      |
                  |  passtrough]       |     pad-alloc()      |
                  |------------------------------------------>|
                  | [caps changed]   .-|                      |
                  | find_transform() | |                      |
      return      |                  '>|                      |
    <----------------------------------|                      |
                  |                    |                      |

   - pad-alloc is called with different caps than are currently being handled by
     the transform on the sinkcaps we have to try to negotiate a new
     configuration for the transform element.

      * we perform the standard way to finding a best transform using
        find_transform() and we call the pad-alloc function with these caps.
	If we get different caps from pad-alloc, we find the best format to
	transform these to and return those caps instead.


               sinkpad              transform               srcpad
   buffer_alloc() |                    |                      |
  --------------->|                    |                      |
                  |  find_transform()  |                      |
                  |------------------->|                      |
                  |                    |  pad-alloc()         |
                  |                    |--------------------->|
      return      | [caps unchanged]   |                      |
    <----------------------------------|                      |
                  |                    |                      |
                  | [caps changed]   .-|                      |
                  | find_transform() | |                      |
      return      |                  '>|                      |
    <----------------------------------|                      |
                  |                    |                      |

In order to perform passthrough buffer-alloc or pad-alloc, we need to be able
to get the size of the output buffer after the transform.

For passthrough buffer-alloc, this is trivial: the input size equals the output
size.

For the copy transform or the in-place transform we need additional function to
retrieve the size. There are two functions:

  - transform_size()
    
    Given a caps and a size on one pad, and a caps on the other pad, calculate
    the size of the other buffer. This function is able to perform all size
    transforms and is the prefered method of transforming a size.

  - get_unit_size()

    When the input size and output size are always a multiple of eachother
    (audio conversion, ..) we can define a more simple get_unit_size() function.
    The transform will use this function to get the same amount of units in the
    source and destination buffers.

 For performance reasons, the mapping between caps and size is kept in a cache.


Issues
~~~~~~

passthrough and in-place transforms (with writable buffers) never need to
perform a pad-alloc on the srcpad. This means that if upstream negotiation
happens, the transform element will never know about it.

The transform element will keep therefore track of the allocation pattern of
the peer elements. We can see the following cases:

- upstream peer calls buffer-alloc on the sinkpad of the transform. In some
  cases (see above) this call gets proxied or not.

- upstream peer does never call buffer-alloc.

We will keeps state about this allocation pattern and perform the following in
each case respectively:

- Upstream calls buffer-alloc: In passthrough and (some) in-place we proxy
  this call onto the downstream element. If the caps are changed, we mark
  a flag that we will require a new pad-alloc for the output of the next
  output buffer.

- upstream peer does not call buffer-alloc: We always perform a pad-alloc
  when processing buffers. We can further optimize by only looking at the
  returned caps instead of doing a full, needless buffer copy.


Use cases
~~~~~~~~~

 videotestsrc ! ximagesink
  
   - resizing happens because videotestsrc performs pad-alloc.

 videotestsrc peer-alloc=0 ! ximagesink

   - resizing cannot happen because videotestsrc never performs pad-alloc.

 videotestsrc ! videoscale ! ximagesink
  
   - videoscale is initially configured in passthrough mode, pad-alloc from
     videotestsrc is proxied through videoscale.
   - pad-alloc will renegotiate a new size in videotestsrc.

 videotestsrc peer-alloc=0 ! videoscale ! ximagesink
  
   - videoscale is initially configured in passthrough mode.
   - videoscale performs pad-alloc because no buffer-alloc is called on the
     sinkpad
   - resizing the videosink makes videoscale perform the scaling.
   
Problematic
~~~~~~~~~~~

    filesrc location=~/media/moveyourfeet.mov ! decodebin !
    ffmpegcolorspace ! videoscale ! ffmpegcolorspace ! ximagesink -v