gstreamer/docs/random/wtay/capsnego2-docs

purpose
-------

GStreamer has a very flexible mechanism to describe media types using
mime types and key/value pairs (GstCaps). The definition of media types 
is entirely done by the plugins which can set the media type to one or 
more of the plugins GstPads. 

Pads with the same mime type and 'compatible' properties are allowed to
connect. It is possible that a pad can accept or produce many different 
media types.

The purpose of caps negotiation is to provide a framework for plugins so
that they can agree on a common media type for their pads.


Capabilities (GstCaps)
----------------------

The core component in the caps negotiation system are GstCaps. They consist
of:

 - a name (ex. "my_audio_capabilities") 
 - a mime-type (ex. audio/raw, audio/mp3, ...)
 - a list of key/value pairs (ex. channels=2, rate=44100, ...)

The list of key/value pairs is maintained by the GstProps object.

The GstProps object maintains a GList of GstPropsEntries. An entry has
a key, which is always a string constant (internally converted to a GQuark)
and a value, which can be one of the following:

 - an integer constant (ex. 5)
 - a float constant (ex. 1.0)
 - a string constant (ex. "int")
 - a boolean constant (ex. FALSE)
 - a fourcc constant (ex. I420) 
     * fourcc codes are usually used to describe a video format

In addition to these constant values, the following variable values are
supported too:

 - an integer range (ex. 0-200)
 - a float range (ex. 1.0-3.0)
 - a list of values (ex. 1, 2, 5). 
     * A List cannot contain another list and the
       entries in the list should all have the
       same type (int, float, string, fourcc). It is
       allowed to mix integers/floats and  
       integer/float ranges in a list.

A capability is usually described as follows:

                                GST_CAPS_NEW (
      capability name --->        "my_audio_capabilities",
      mime-type --------->        "audio/raw",
                          (        "format",	GST_PROPS_STRING ("int"),
      GstProps ---------> (        "channels",	GST_PROPS_INT_RANGE (1, 2),
       (list of entries)  (        "rate",	GST_PROPS_INT (44100)
                                )

				(-----------)   (--------------------------)
                                  entry key         entry value

Two capabilities can be chained together to form a larger capability:

                              (  GST_CAPS_NEW (
                              (    "my_mp3_capability",
	                      (    "audio/mp3",
	one capability ---->  (    NULL    
        created by chaining   (  ),
        two capabilities.     (  GST_CAPS_NEW (
                              (    "my_vorbis_capability",
	                      (    "audio/vorbis",
	                      (    NULL
                              (  )

Capabilities always work as constraints, this means that they constrain the
media type to the given mime-type and properties. By this definition a NULL 
GstCaps or a NULL GstProps means: no constraints.


Variable capabilities vs. fixed capabilities
--------------------------------------------

Definition:

  A GstProps structure is said to be fixed if it doesn't contain lists or
  ranges, otherwise it is a variable GstProps. A variable GstProps, by definitin
  does not constrain the media type to a set of fixed properties.

  A GstCaps is said to be fixed if it is not chained and it doesn't contain
  a variable GstProps component.


GstCaps compatibility
---------------------

<write me>

GstCaps intersection
--------------------

<write me>


GstCaps usage
-------------

GstCaps are used in the following data structures:

 - padtemplates. 
    * padtemplates are added to elementfactory to describe the possible
      pads that an element created from said factory can have.
    * padtemplates contain the name, direction and presence of the pads.
    * padtemplates also describe the media types that this element can
      accept/produce using GstCaps.
    * padtemplates can provide fixed or variable GstCaps for the pads.
    * padtemplates can be used by the element to create its pads and is
      highly recommended.
    * the padtemplate GstCaps are saved into the registry so that the
      media types an element can operate on, are known without having to
      bring the element into memory.
      
 - pad caps
    * pad caps are _fixed_ caps attached to a pad to describe the exact media
      type this pad is handling. A pad with caps is said to be a "tainted" pad.
      
 - connection filters
    * a connection filter is created when two pads are connected. It describes 
      the media type(s) that _can_ flow through this connection. 

 - application connection filters
    * When the application connects two pads, it can specify an application
      connection filter that will act as additional constraints on the media types
      that can flow through the connection.

Connection filters and application filters are cleared when two connected pads 
are disconnected. Pad caps are not cleared.  Padtemplates are immutable and 
never cleared.


The GstPad get_caps function
----------------------------

the gst_pad_get_caps function returns the caps of a given pad. The pad caps are
calculated as:

 - if the pad has pad caps, return those
 - else if the pad has a getcaps function, call the function and return the result
 - else if the pad has a padtemplate, return the padtemplate caps
 - else return NULL


Purpose of caps negotiation
---------------------------

The purpose of the caps negotiation procedure is to set "pad caps" on a pad
so that it is compatible with the "connection filter". This has to be done 
_before_ any data passing happens through the connection.  Data passing between two 
pads is _not_ allowed when the pad caps are not compatible with the connection 
filter or when the pad caps of two pads participating in the connection are not
equal.

Real caps negotiation starts as soon as an element is in the READY state or higher.
This means that the connect functions of the pads are only called when the element
is at least READY. The intersection between two pads is made at connect time, 
regardless of element state.


GstPad connection
-----------------

When two pads are connected get_caps is called on both pads to get their caps.
Then the intersection between those caps is made, this will give us all the
possible media types that can flow through this pad connection. Optionally the
application can provide additional caps, the pad intersection is also made with
the application caps. 

The intersection and the optional application caps are stored in the two pads.

If the intersection is NULL, the two pads have no common types and the connection
is refused.

If the intersection is a fixed caps, this means there is only one possible media type
that can be used for this connection.

For all not NULL intersections the pad connect functions are called with the 
intersection. Depending on the result of the connect function the connection is
allowed or refused.

If the intersection is fixed and the pad connect functions agreed to the caps, 
the caps are set on the pads.

Note that pad caps are never set for non fixed caps.

!Example 1:
! 
! 1. before connecting the pads, they both have a set of possible media types,
!    on the pad, through the getcaps function or on the padtemplate (here
!    represented with capital letters)
!
!    srcpad                       sinkpad
!       A                           B
!       B                           F
!       C                           A
!	                            G
!
! 2. when performing the connection, the intersection between the two sets of caps
!    is made.
!    
!    srcpad                       sinkpad
!       A )                      (  B
!       B )-------->  A   <------(  F
!       C )           B          (  A
! 	                         (  G
! 
! 3. In this case the intersection is not a fixed caps (it's a chained caps).
!    the connect function of the two pads are called (if any), the connect
!    function can accept of refuse the caps.
! 
! 4. if the caps are accepted, the intersection caps are set on both pads.
! 
! 5. plugins will typically not configure themselves if they get variable caps.
!    It is possible though for a plugin to select one of the caps, fixate
!    some properties and refine the filter to fixed caps (see later)
!


!Example 2:
! 
! 1. we take two pads that intersect to fixed caps (B):
! 
!    srcpad                       sinkpad
!       A  )                     (  B
!       B  )------->  B  <-------(  F
! 
! 2. both pad connect functions are called. 
! 
! 3. assuming the connect functions did not refuse the connection, the caps
!    are set on both pads (because they are fixed).
! 
!    srcpad  (B)--------------(B) sinkpad
!       A                           B
!       B                           F
! 
   
!Example 3:
! 
! 1. we take two pads, one with pad caps, another one with a padtemplate. 
! 
!    srcpad (B)                   sinkpad
!            !                   (  B
!            ------>  B  <-------(  F
! 
! 2. the pad get_caps function will return the pad caps of the srcpad and
!    the padtemplate caps of the sinkpad. The intersection is made, yielding
!    a fixed caps (B) which is sent to both connect functions (if any)
!
! 3. If the connect function(s) didn't refuse the connection the fixed caps
!    are set on both pads. The intersection is also kept in the pad object.
!
!    srcpad  (B)--------------(B) sinkpad
!                                   B
!                                   F
!
! 4. when fixed caps are received on the sinkpad the plugin will typically 
!    configure itself to deal with the format.
!


Pad connect functions
---------------------

A Pad can be notified when another pad is connected or reconnected to it with
fixed or variable caps. This notification will be done with the optional 
GstPadConnectFunction callback that an element can provide for a pad.
Remember that this connection is _always_ called, not only with fixed caps
but also with variable caps (if any).

We will explain some of the common situations with some examples.

!Example 4:
!
! 1. We have a simple mpeg audio decoder connected to a simple audio sink.
!    The mpeg decoder doesn't have a connect function on 
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!