gstreamer/docs/random/wtay/caps-negociation

caps negotiation
================

1) purpose
----------

The pads expose the media types they can handle using a mime
type and a set of properties. Before the pad is created or
used to pass buffers, we only know the global 'range' of media
data this pad can accept. When the element has had a chance to
look at the media data, only then it knows the exact values of the 
properties.

example1:
!
! The mp3 decoder exposes the capabilities of its src pad
! with the following caps:
!
!     'mpg123_src':
!       MIME type: 'audio/raw':
!       format: Integer: 16
!       depth: Integer: 16
!       rate: Integer range: 11025 - 48000
!       channels: Integer range: 1 - 2

as you can see in example1, the padtemplate has both a range
(for the audio rate) and a list (for the number of channels)
for its properties.

only when the mpg123 element has decoded the first mpeg audio
header, it knows the exact values of the rate and channels
properties.

suppose that we want to connect this src pad to the sink pad
of an audiosink with the properties given in example2:

example2:
!
!     'audiosink_sink':
!       MIME type: 'audio/raw':
!       format: Integer: 16
!       depth: List:
!         Integer: 8
!         Integer: 16
!       rate: Integer range: 8000 - 44000
!       channels: Integer range: 1 - 2

we can see that connecting the mpg123 src pad with the
audiosinks sink pad can cause a potential problem with the
rate property.

When the mpg123 decoder decides to output raw audio with a
48000Hz samplerate, the audiosink will not be able to handle
it. The conservative approach would be to disallow the connection
between the two incompatible pads. This rules out any potential
problems but severely limits the amount of possible connections
between the elements. 

Another approach would be to allow the connection (and mark it
as dangerous) and let the two elements figure out a suitable
media type at runtime. This procedure is called caps negotiation.


2) a bit of history
-------------------

The typing of the data that was attached to a buffer used to be
done using GstMeta* (and it still is as of 11 feb 2001). With
the new GstCaps and GstProps system this typing is gradually moved
to the pads and to the padtemplates. This has several advantages:

  - the typing of the data tends to be static. The type of media
    doesn't change for every buffer.

  - Moving the typing up to the pad(templates) allows us to save
    them into the registry and allows us to figure out what pads
    are compatible.

  - the current metadata implementation needs header files. this may
    change when we also use properties for metadata.

example3:
!
! This is the current GstMeta structure that travels with audio buffers
!
!  struct _MetaAudioRaw {
!    GstMeta meta;
!
!    /* formatting information */
!    gint format;
!    gint channels;
!    gint frequency;
!    gint bps;
!  };
	    

The question still remains what purpose the metadata will serve
now that we expose the media type in the pads. Some possibilities:

  - interesting information, not describing the data itself but the
    context in which the data was generated (suggested buffer size,
    timestamps, etc...)

  - user app metadata. 

In this proposal we also assume that the current use of metadata using
GstMeta is deprecated and that we move this information to the properties
of the pads.


3) the pad/padtemplates caps
----------------------------

All elements have to provide a padtemplate for their pads.

The padtemplates provide a range of possible media types this pad can
src/sink. the main purpose for the padtemplates is to allow a
rough guess at which pads are compatible before even a single buffer
has been processed by the element.

pads are usually created from the templates. When the pad is created
it has no GstCaps* attached to it yet. The possible caps this pad
can have is exposed in the padtemplate.


4) the connect function
-----------------------

when two pads are connected the following steps will take
place: 

 - if both pads have caps, the caps are checked. If the caps
   are incompatible, the padtemplates are checked, if they
   are compatible, caps negotiation is performed.

 - if one of the pads has caps, the caps is checked against
   the padtemplate of the peer pad. If they are incompatible,
   the padtemplates are compared, if they are incompatible,
   caps negotiation is performed.
   
 - if none of the pads have caps, the padtemplates are checked, 
   if they are incompatible, a warning is issued.
   

5) when the element knows the media type it is handling
-------------------------------------------------------

When the element has received its first buffer it will know
the media type it is handling by inspecting the buffer.

before pushing the data out to its peer element(s), the element
will set its src pad with the appropriate caps and properties.
These caps must follow the following rules:

  - the caps must be compatible with the padtemplates of this 
    pad.

  - the caps cannot contain ranges or lists.

by setting the caps of the src pad, the following procedure
happens:

  - if the peer pad has a negotiate function, it is called and
    negotiation is performed (see later)

  - if the peer pad has no negotiate function, the padtemplate
    is checked:

    - if the caps are compatible with the template, the peer pad
      receives the same caps as the src pad.

    - if the caps are not compatible, the negotiation fails and
      the elements are disconnected. A signal is emitted to inform
      the user app or the manager of the element to find a solution.
 
the caps can be set with the gst_pad_set_caps function, which accepts
the following parameters:

   - the pad to set the caps to
   - a GstCaps* structure

example4:
!
!  an audio element setting its src pad caps (need something easier):
!
!     gst_pad_set_caps (
!       pad,
!       gst_caps_new_with_props (
!         "src_caps",                       /* name */
!         "audio/raw",                      /* mime */
!         gst_props_new (
!           "format",   GST_PROPS_INT (AFMT_S16_LE),
!           "depth",    GST_PROPS_INT (16),
!           "rate",     GST_PROPS_INT (44100),
!           "channels", GST_PROPS_INT (2),
!           NULL
!         )
!       )
!     );

The _set_caps method will trigger the caps negotiation with the
peer pad (if connected). 


6) caps negotiation function
----------------------------

the negotiate function of a pad is called whenever the peer pad 
modifies the caps using the gst_pad_set_caps function.

The negotiate function has to return a gboolean indicating the 
new caps are acceptable. When it accepts the caps, both pads will
be set to the negotiated caps.

example5:
!
! this is the caps negotiation function implemented by an element on
! one of its sink pads.
!
!  static gboolean
!  gst_pad_caps_negotiate (GstPad *pad, GstCaps *caps)
!  {
!    /* we don't accept anything else than audio/raw */
!    if (strcmp (gst_caps_get_mime (caps), "audio/raw"))
!      return FALSE;
!
!    if (gst_caps_get_int_prop (caps, "format") != AFMT_S16_LE)
!      return FALSE;
!      
!    /* we accept everything else */
!    return TRUE;
!  }

When the negotiate function returns FALSE (it does not accept the
specified caps of the peer pad), 


figure1
!
!
!  element             pad                pad->peer
!   !
!   ! _negotiate(pad)   
!   !------------------>!
!                       ! gst_pad_negotiate()
!                       !------. 
!                       !<-----'               
!                       ! _caps_negotiate()
!                       !--------------------->!
!
!


the element has some of its internal properties changed. It wants
to renegotiate the caps with its peer element. The element does:

  gst_pad_renegotiate (element->srcpad);

this will trigger the class method of the pad and 




7) use cases
------------