gstreamer/docs/random/omega/caps2

The elementfactory for a given element will contain some information about the capabilities of element's
pads or potential pads.  An indication will be provided as to whether the pad always exists, always
exists once data is present, or *might* exist once data is present (the latter case is for things like
the MPEG system parsers, where an audio stream might or might not exist).


First, an entirely normal example:
----------------------------------

(-----------)        (----------)        (-------------)
! disksrc   !        ! mpg123   !        ! audiosink   !
!         src        sink     src        sink          !
!           !        !          !        !             !
(-----------)        (----------)        (-------------)

We start with only the disksrc.  The typefind filter is attached to the disksrc, and via typefind magic
the properties of the disksrc are found to be:

  disksrc->src->caps = {
    "audio/mp3",
    "layer",   GST_CAPS_INT (3),
    "bitrate", GST_CAPS_INT (128),
    NULL
  };

A look through the plugin registry shows that we have an element called mpg123 that has the following
caps:

  static GstCapsFactory mpg123_sink_caps = {
    "audio/mp3",
    "layer",   GST_CAPS_INT_RANGE (1, 3),
    "bitrate", GST_CAPS_INT_RANGE (8, 320),
    NULL
  };

The caps of the disksrc fit within those parameters, so we instantiate an mpg123 and attach it to the
disksrc.  The connection succeeds negotiation and as a result the mpg123 specifies its output caps as:

  mpg123->src->caps = {
    "audio/raw",
    "format",   GST_CAPS_BITFIELD (S16),
    "depth",    GST_CAPS_INT (16),
    "rate",     GST_CAPS_INT (44100),
    "channels", GST_CAPS_INT (2),
    NULL
  };

Again from the plugin registry we find an element audiosink that has appropriate caps:

  static GstCapsFactory audiosink_src_caps = {
    "audio/raw",
    "format",   GST_CAPS_BITFIELD (S16,....),
    "depth",    GST_CAPS_INT (16),
    "rate",     GST_CAPS_INT_RANGE (4000, 96000),
    "channels", GST_CAPS_INT_RANGE (1, 2),
    NULL
  };

A copy of the audiosink is instantiated and attached, negotiation goes smoothly, and we're done.  No
dataflow has occurred, no failure found, etc.  An ideal autoplug.


Now, a slightly more convoluted example:
----------------------------------------

Start with the same graph:

(-----------)        (----------)        (-------------)
! disksrc   !        ! mpg123   !        ! audiosink   !
!         src        sink     src        sink          !
!           !        !          !        !             !
(-----------)        (----------)        (-------------)

Run typefind on the disksrc's output, get the same output caps:

  disksrc->src->caps = {
    "audio/mp3",
    "layer",   GST_CAPS_INT (3),
    "bitrate", GST_CAPS_INT (128),
    NULL
  };

Find and attach mpg123, get the following output caps this time:

  mpg123->src->caps = {
    "audio/raw",
    "format",   GST_CAPS_BITFIELD (S16),
    "depth",    GST_CAPS_INT (16),
    "rate",     GST_CAPS_INT (44100),
    "channels", GST_CAPS_INT (1),
    NULL
  };

Note that this time we have a mono output.  A look into the audiosink caps shows that we have a match.
So we instantiate a copy.  Oops.  We now find that the caps for the input pad on our audiosink have
changed:

  mpg123->src->caps = {
    "audio/raw",
    "format",   GST_CAPS_BITFIELD (S16,...),
    "depth",    GST_CAPS_INT (16),
    "rate",     GST_CAPS_INT (11025, 48000),
    "channels", GST_CAPS_INT (2),
    NULL
  };

Whoops.  It seems that the sound card we've got in this machine (FIXME how on earth to deal with
multiple sound cards???) doesn't support mono output *at all*.  This is a problem.  We now find that we
have no options as far as directly matching the mpg123 to the audiosink.

A look through our (fictitious) plugin registry shows at least one element that at least has audio/raw
on both input and output (since both mpg123 and audiosink have open pads with this mime type).  A closerlook shows that its caps are:

  static GstCapsFactory mono2stereo_sink_caps = {
    "audio/raw",
    "channels", GST_CAPS_INT (1),
    NULL
  };
  static GstCapsFactory mono2stereo_src_caps = {
    "audio/raw",
    "channels", GST_CAPS_INT (2),
    NULL
  };

Wow, that's a perfect match.  Instantiate, attach to mpg123, no problems.  Attach to audiosink, no
problems.  Done.  When we start up the pipeline, we should get absolutely no callbacks from pads saying
"help me, I've fallen and..., er, I don't like this buffer!".


A really messy case:
--------------------

Start with a disksrc, typefind it, get the following:

  disksrc->src->caps = {
    "audio/mp3",
    "layer",   GST_CAPS_INT (3),
    "bitrate", GST_CAPS_INT (128),
    NULL
  };

Look through the plugin registry, find mpg123.  Instantiate it, attach it.  It spits out audio
parameters as usual:

  mpg123->src->caps = {
    "audio/raw",
    "format",   GST_CAPS_BITFIELD (S16),
    "depth",    GST_CAPS_INT (16),
    "rate",     GST_CAPS_INT (44100),
    "channels", GST_CAPS_INT (2),
    NULL
  };

Now we instantiate an audiosink plugin.  This time, we're sunk:

  mpg123->src->caps = {
    "audio/raw",
    "format",   GST_CAPS_BITFIELD (S8,U8),
    "depth",    GST_CAPS_INT (8),
    "rate",     GST_CAPS_INT_RANGE (11025, 22050),
    "channels", GST_CAPS_INT (1),
    NULL
  };

ACK!  It's one of those Disney Sound Source things.  We've got a problem here that isn't obviously
solvable.  However, there happens to be another mp3 decoder sitting around.  It's got the same
properties as mpg123, but a lower merit value.  Let's instantiate one and attach it.  We get the
following output pad caps:

  mp3decoder->src->caps = {
    "audio/raw",
    "format",   GST_CAPS_BITFIELD (S8,S16),
    "depth",    GST_CAPS_INT_RANGE (8,16),
    "rate",     GST_CAPS_INT_RANGE (8000, 44100),
    "channels", GST_CAPS_INT (1,2),
    NULL
  };

Well, that matches the audiosink.  We try attaching it, and during negotiation the mp3decoder finds
sufficient common ground with the castrated audiosink and sets its output pad to match the best of the
options: S8 at 22050 KHz.



Next to impossible scenario: DVD
--------------------------------

Start with a dvdsrc.  It's output pad caps are:

  static GstCapsFactory dvdsrc_src_caps = {
    "video/mpeg",
    "mpegversion",  GST_CAPS_INT (2),
    "systemstream", GST_CAPS_BOOLEAN (TRUE),
    NULL
  };

The type would be classified as incomplete via some mechanism.  This might cause the autoplug code to go
and run the typefind function.  It would flesh the type out to the following:

  dvdsrc->src->caps = {
    "video/mpeg",
    "mpegversion",  GST_CAPS_INT (2),
    "systemstream", GST_CAPS_BOOLEAN (TRUE),
    "videostreams", GST_CAPS_INT (1),
    "audiostreams", GST_CAPS_INT (3),
    "bitrate",      GST_CAPS_INT (40960),
    NULL,
  };

Wow, that helped a lot.  A check through the plugin registry shows that the mpeg2parse will match those
properties:

  static GstCapsFactory mpeg2parse_sink_caps = {
    "video/mpeg",
    "mpegversion",  GST_CAPS_INT (2),
    "systemstream", GST_CAPS_BOOLEAN (TRUE),
    NULL
  };

(In retrospect, it may not be necessary to run typefind if there's match this good right away.  Only run
typefind when there's no exact match.)
Since there are no output pads yet, we have to actually push data through the pipeline.  The moment a
buffer or two get to the mpeg2parse element, it promptly goes and creates an output pad, probably of the
following caps:

  mpeg2parse_video_src_caps = {
    "video/mpeg",
    "mpegversion",  GST_CAPS_RANGE (1,2),
    "systemstream", GST_CAPS_BOOLEAN (FALSE),
    NULL
  };

This seems to be a task for typefind again.  But since data is flowing, we have to be careful with the
buffers.  (This is the case in any typefind maneuver, but more so when one really can't rewind the
source without consequences)  The autoplug system attaches a special pseudo-element to mpeg2parse's new
output pad, and attaches the typefind element to the end of that.  The pseudo-element takes the buffer,
stores it, and passes a copy off to the attached element, in this case typefind.  This repeats until
typefind has determined the type, at which point the typefind is removed, and the newly found element is
attached instead.

The pseudo-element is 'rewound' and the stored buffers flow out and into the newly attached element.
When the cache of buffers is gone, a signal fires and the autoplug system removes the pseudo-element and
reconnects the pipeline.

In this case, the typefind function will find the following:

  mpeg2parse_video_src_caps = {
    "video/mpeg",
    "mpegversion",  GST_CAPS_INT (2),
    "systemstream", GST_CAPS_BOOLEAN (FALSE),
    "bitrate",      GST_CAPS_INT (36864),
    "width",        GST_CAPS_INT (720),
    "height",       GST_CAPS_INT (480),
    "framerate",    GST_CAPS_FLOAT (29.97002997),
    "chromaformat", GST_CAPS_INT (1),		[GST_CAPS_STRING ("4:2:0") ?]
    NULL
  };

Back to the plugin registry, we find our only choice is mpeg2dec, which has input caps of:

  static GstCapsFactory mpeg2dec_sink_caps = {
    "video/mpeg",
    "mpegversion",  GST_CAPS_RANGE (1,2),
    "systemstream", GST_CAPS_BOOLEAN (FALSE),
    NULL
  };

Once again it just so happens that we really didn't need to do the typefind at all.  But it can't hurt
unless the typefind is slow and painful, which we can guess won't be the case since the choices are
rather limited by the fact that there's already a MIME type attached, meaning we can drastically reduce
the number of typefind functions we try (down to one, actually).

However, since we *have* run the typefind, upon attachment of the input pad of mpeg2dec, the output pad
looks like the following:

  mpeg2dec_src_caps = {
    "video/raw",
    "fourcc",    GST_CAPS_LIST (
			GST_CAPS_FOURCC ("YV12"),	[identical...]
			GST_CAPS_FOURCC ("IYUV"),
			GST_CAPS_FOURCC ("I420"),
                 ),
    "width",     GST_CAPS_INT (720),
    "height",    GST_CAPS_INT (480),
    "framerate", GST_CAPS_FLOAT (29.97002997),
    NULL
  };

Currently only videosink supports the output of video/raw.  It claims a list of FOURCCs but nothing
more:

  static GstCapsFactory videosink_sink_caps = {
    "video/raw",
    "fourcc", GST_CAP_LIST ( GST_CAPS_FOURCC ("YV12"),
		GST_CAPS_FOURCC ("IYUV"), GST_CAPS_FOURCC ("I420"),
		GST_CAPS_FOURCC ("YUY2"), GST_CAPS_FOURCC ("UYVY"),
		[ etc... ],
              ),
    NULL
  };

When instantiated, we potentially have the same problem as with the audiosink: we don't necessarily know
which hardware output to use.  Somehow we have to solve the problem of setting some element arguments
before we can get useful information out of them as to the properties.  In this case anyway, if the
videosink were to find only one output possibility, it would trim the list of FOURCCs it can deal with
to what the hardware can handle, as well as add further properties:

  videosink_sink_caps = {
    "video/raw",
    "fourcc", GST_CAPS_LIST (GST_CAPS_FOURCC ("YV12"),
			GST_CAPS_FOURCC ("YUY2"),
              ),
    "width",  GST_CAPS_INT_RANGE (4,1020),
    "height", GST_CAPS_INT_RANGE (4,1020),
    NULL
  };

We can now connect the mpeg2dec output to the videosink, and we now have displaying video.

. . . .