gstreamer/docs/random/wtay/negotiation3

Caps Negotiation
================


Definitions
-----------

- GstCaps is a structure that holds a mimetype and a set of properties. The properties are
  either fixed or non-fixed, they contain ranges or lists.

- filter caps: caps set by the application to constrain a link to a specific GstCaps.

- allowed caps: caps calculated when connecting elements, these are the types that are
  possible on this connection. If no types are possible, the link is refused.

- pad template caps: caps put on padtemplates to describe the possible media types of
  the pad.

- pad caps: the caps of the pad as it is currently negotiated.


General overview
----------------

Caps negotiation is the process of two pads agreeing on a common fixed GstCaps structure
that describes the media type that will be exchanged between those pads.

We define that caps negotiation only happens in the PLAYING state, when there is actual
dataflow and elements thus know what kind of data they are dealing with, or when 
linking elements.

Caps negotiation is first performed when linking elements. If the elements can agree
on a media type at link time then negotiation will not have to be performed at runtime.
Usually this is not possible, so the pads remain unnegotiated until at runtime.

Caps negotiation is initiated by an element that wants to send data on a non-negotiated
pad, thus the source pad. The core only provides a policy and convenience methods to aid 
the element in performing the negotiation. The core does not keep track of what pads
are negotiated or not, this is a task of the pads.

Caps negotiation is normally started by the source element right before it sends out 
data (case 1).
Caps negotiation can also be redone by a sink element that wants to receive another 
format from a downstream element (case 2).

There are two functions handling the negotiation, the link function and the do_link
function. The link function is always called first and must eventually call do_link on
the peer pad.  The link function must figure out a compatible format for the connection
and then call the do_link function on the peer pad. The do_link function can only 
accept or refuse the provided caps.

For autopluggers it is important to know when the pad is ready to start the negotiation.
It is also important to know when the negotiation failed and it must be possible to 
restart the negotiation with another element. This functionality will be provided
with signals.

Pad functions
-------------

!
! const GstCaps* gst_pad_iterate_caps (GstPad *pad, gint position);
!
Returns the nth possible caps that describes the media type that can flow over
this pad. This function should not return static caps but caps that are 
dependent of the media type and the peer connections of the element.

This function can be called at any time so that an autoplugger can know the
exact types of the pads at any time.

!
! gboolean gst_pad_accept_caps (GstPad *pad, GstCaps *caps);
!
Checks that a given caps is acceptable for this pad. Returns FALSE if the
pad cannot handle the caps.

!
! gboolean gst_pad_link (GstPad *pad, GstPad *peer, GstCaps *caps);
!
Tells the pad to start negotiation with the given filtercaps. The
caps need not be fixed and serves as a filter for performing the negotiation
with the peerpad. 

!
! gboolean gst_pad_do_link (GstPad *pad, GstPad *peer, GstCaps *caps);
!
Configures the pad to accept data with the given caps.  The caps must be fixed. 

!
! const GstCaps* gst_pad_get_negotiated_caps (GstPad *pad);
!
Get the negotiated caps of a pad or NULL if the pad is not negotiated.


Linking Elements
----------------

When linking elements with the gst_pad_link function, the core will call
the link function on the srcpad. If that pad returns GST_PAD_LINK_OK,
the link is established, else the link fails.

Since the link function of the pad will call the do_link function of
the peerpad, the caps will be set on both pads.

It is not required to decide on a caps at link time, a plugin can choose to
dynamically renegotiate at runtime.

When a link fails, the core emits a signal link_failed, which the application
can catch to try to establish a new link with another element, for example.

!
! def gst_pad_link_filtered (pad1, pad2, filtercaps):
!
!   ... get srcpad and sinkpad ...
!   srcpad  = (pad1 == GST_PAD_SRC ? pad1 : pad2);
!   sinkpad = (pad1 == GST_PAD_SINK ? pad1 : pad2);
!
!   ... more checks to see if the pads are of different types and
!   ... that they live in the same scheduler etc...
!
!   res = gst_pad_link (srcpad, sinkpad, filtercaps)
!   if (res == GST_PAD_LINK_OK)
!     ... perform other setup ...
!   else
!     res = signal_emit (srcpad, "link_failed")
!
!   return res
!

Pad link is just a convenience function that passes a NULL filtercaps:
! 
! def gst_pad_link (pad1, pad2):
!   gst_pad_link_filtered (pad1, pad2, NULL)
!


Dynamic renegotiation
---------------------

Dynamic renegotiation happens at runtime when the element knows the exact media
type it is handling.

The element that wants to redecide on the data type of a connection just calls
gst_pad_relink on one of it's pads. The core will call unlink and link on the
pads again as if this were a new connection. Since the iterate_caps function
returns other values while linking, the new link will renegotiate to a new
format or the link will fail.

Prototype of the relink function:
!
! def gst_pad_relink_filtered (pad, filtercaps):
!   gst_pad_unlink (pad, peerpad)
!   gst_pad_link_filtered (pad, peerpad, filtercaps)
!

Again the relink function is a convenience function for not having to pass 
filtercaps.
!
! def gst_pad_relink (pad):
!   gst_pad_relink_filtered (pad, NULL)
!

The core, however, should optimize the relink function so that it does a minimal
amount of work, like not informing the scheduler about the unlink/link call in
case of success.

Error recovery
--------------

When a pad is ready to negotiate and it has no peer pad, it fires the "need-link"
signal. Autopluggers can use this signal to start plugging elements to the pad.

When a link fails because of a renegotiation, the "link-failed" signal is fired
so that autopluggers can try other elements.


Default implementations
-----------------------

!
! gst_pad_src_link_func (pad, peerpad, filtercaps) 
! {
!   srcpad->negotiated_caps = NULL;
!   gboolean possible = FALSE;
!
!   for (i=0; peercaps = gst_pad_iterate_caps (peerpad, i); i++) 
!   {
!     for (j=0; srccaps = gst_pad_iterate_caps (srcpad, j); j++) 
!     {
!       test = gst_caps_intersect (srccaps, peercaps, filtercaps);
!       if (test == GST_CAPS_EMPTY)
!         continue;
!
!       /* non empty caps, something is possible */
!       possible = TRUE;
!
!       if (!gst_caps_is_fixed (caps))
!         continue;
!
!       if (gst_pad_accepts_caps (peerpad, test))
!       {
!         if (gst_pad_do_link (peerpad, srcpad, test))
!         {
!           srcpad->negotiated_caps = test;
!           goto done;
!         }
!         else
!         {
!           /* weird, it accepted but did not want to link */
!         }
!       }
!       else
!       {
!         /* caps are not accepted by peer, try next */
!       }
!     }
!   }
! done:
!   if (!possible)
!     return GST_PAD_LINK_FAILED;
!   else
!     return GST_PAD_LINK_OK;
! }
!

gst_pad_iterate_caps returns caps in the following order:

 1) preferred caps (if any)
 2) negotiated caps (if any)
 3) profile caps (if any, filtered against caps of current media type)
 4) padtemplate caps (filtered against caps of current media type)

1 = a caps describing the media type that would result in optimal
    processing of the current media type
2 = a caps that it is currently handling
3 = a caps describing the user configured default values
4 = a caps describing all other possibilities that this pad can
    produce or consume given the current media type if any.


generic flow 1:

 Linkage

         src                         sink
          |                           |
          | <--- link(sink, A)        |
check if  |                           |
A is ok   |                           |
or filter |                           |
against   |                           |
allowed   |                           |
src caps  |                           |
          |  iterate_caps             |
if A not  +-------------------------> |
fixed     |                           |
          | <-------------------------+
filter    |                           |
against A |                           |
to get A' |                           |
          |  can_accept(A')           |
if A' fix +-------------------------> |
          |                           | check if A' is ok
          |        yes                |
          | <-------------------------+
          |                           |
if A' not |                           |
fixed,    |                           |
A'=null   |                           |
          |  do_link(src, A')         |
          +-------------------------> |
          |        ok                 | store src, A'
          | <-------------------------+
store     |                           |
sink, A'  |                           |



Unlink

         src                         sink
          |                           |
          | <--- unlink()             |
          |                           |
          |  unlink()                 |
          +-------------------------> |
unref     |                           | unref src, format
sink, fmt |                           |


  Dynamic negotiation of B

         src                         sink
          |  can_accept(B)            |
          +-------------------------> |
          |                           | check if B is ok
          |        yes                |
          | <-------------------------+
          |                           |
          |       do_link(B)          |
          +-------------------------> |
          |        ok                 | store B
          | <-------------------------+
store     |                           |
B         |                           |



TODO
-----

Write test objects to simulate different behaviour.


tests (in python):

   vts -> colorspace -> ximagsink

  relink with different colorspaces while running

  
   vts -> videoscale -> ximagesink

  relink to different sizes while running, scaling vts or ximagesink.

   sinesrc -> audiorate -> audiosink

  where audiosink only has one possible fixed caps.