gstreamer/docs/manual/basics-pads.xml

<chapter id="chapter-pads" xreflabel="Pads and capabilities">
  <title>Pads and capabilities</title>
  <para>
    As we have seen in <xref linkend="chapter-elements"/>, the pads are
    the element's interface to the outside world. Data streams from one
    element's source pad to another element's sink pad. The specific
    type of media that the element can handle will be exposed by the
    pad's capabilities. We will talk more on capabilities later in this
    chapter (see <xref linkend="section-caps"/>).
  </para>

  <sect1 id="section-pads">
    <title>Pads</title>
    <para>
      A pad type is defined by two properties: its direction and its
      availability. As we've mentioned before, &GStreamer; defines two
      pad directions: source pads and sink pads. This terminology is
      defined from the view of within the element: elements receive data
      on their sink pads and generate data on their source pads.
      Schematically, sink pads are drawn on the left side of an element,
      whereas source pads are drawn on the right side of an element. In
      such graphs, data flows from left to right.
      <footnote>
        <para>
          In reality, there is no objection to data flowing from a
          source pad to the sink pad of an element upstream (to the
          left of this element in drawings). Data will, however, always
          flow from a source pad of one element to the sink pad of
          another.
        </para>
      </footnote>
    </para>

    <para>
      Pad directions are very simple compared to pad availability. A pad
      can have any of three availabilities: always, sometimes and on
      request. The meaning of those three types is exactly as it says:
      always pads always exist, sometimes pad exist only in certain
      cases (and can disappear randomly), and on-request pads appear
      only if explicitly requested by applications.
    </para>

    <sect2 id="section-pads-dynamic">
      <title>Dynamic (or sometimes) pads</title>
      <para>
        Some elements might not have all of their pads when the element is
        created. This can happen, for example, with an Ogg demuxer element.
        The element will read the Ogg stream and create dynamic pads for
        each contained elementary stream (vorbis, theora) when it detects
        such a stream in the Ogg stream. Likewise, it will delete the pad
        when the stream ends. This principle is very useful for demuxer
        elements, for example.
      </para> 
      <para> 
        Running <application>gst-inspect oggdemux</application> will show
        that the element has only one pad: a sink pad called 'sink'. The
        other pads are <quote>dormant</quote>. You can see this in the pad
        template because there is an <quote>Exists: Sometimes</quote>
	property. Depending on the type of Ogg file you play, the pads will
        be created. We will see that this is very important when you are
        going to create dynamic pipelines. You can attach a signal handler
        to an element to inform you when the element has created a new pad
        from one of its <quote>sometimes</quote> pad templates. The
        following piece of code is an example of how to do this:
      </para> 
      <programlisting><!-- example-begin pad.c a -->
#include &lt;gst/gst.h&gt;

static void
cb_new_pad (GstElement *element,
	    GstPad     *pad,
	    gpointer    data)
{
  gchar *name;

  name = gst_pad_get_name (pad);
  g_print ("A new pad %s was created\n", name);
  g_free (name);

  /* here, you would setup a new pad link for the newly created pad */
<!-- example-end pad.c a -->[..]
<!-- example-begin pad.c b -->
}

int 
main (int   argc,
      char *argv[]) 
{
  GstElement *pipeline, *source, *demux;
  GMainLoop *loop;

  /* init */
  gst_init (&amp;argc, &amp;argv);

  /* create elements */
  pipeline = gst_pipeline_new ("my_pipeline");
  source = gst_element_factory_make ("filesrc", "source");
  g_object_set (source, "location", argv[1], NULL);
  demux = gst_element_factory_make ("oggdemux", "demuxer");

  /* you would normally check that the elements were created properly */

  /* put together a pipeline */
  gst_bin_add_many (GST_BIN (pipeline), source, demux, NULL);
  gst_element_link_pads (source, "src", demux, "sink");

  /* listen for newly created pads */
  g_signal_connect (demux, "pad-added", G_CALLBACK (cb_new_pad), NULL);

  /* start the pipeline */
  gst_element_set_state (GST_ELEMENT (pipeline), GST_STATE_PLAYING);
  loop = g_main_loop_new (NULL, FALSE);
  g_main_loop_run (loop);
<!--example-end pad.c b -->
[..]<!-- example-begin pad.c c --><!--
  return 0;
--><!-- example-end pad.c c -->
<!-- example-begin pad.c d -->
}
      <!-- example-end pad.c d --></programlisting>
      <para>
        It is not uncommon to add elements to the pipeline only from within
        the "pad-added" or "new-decoded-pad" callback. If you do this, don't
        forget to set the state of the newly-added elements to the target
        state of the pipeline using
        <function>gst_element_set_state ()</function> or
        <function>gst_element_sync_state_with_parent ()</function>.
      </para>
    </sect2>

    <sect2 id="section-pads-request">
      <title>Request pads</title>
      <para> 
        An element can also have request pads. These pads are not created
        automatically but are only created on demand. This is very useful
        for multiplexers, aggregators and tee elements. Aggregators are
        elements that merge the content of several input streams together
        into one output stream. Tee elements are the reverse: they are
        elements that have one input stream and copy this stream to each
        of their output pads, which are created on request. Whenever an
        application needs another copy of the stream, it can simply request
        a new output pad from the tee element.
      </para> 
      <para> 
        The following piece of code shows how you can request a new output
        pad from a <quote>tee</quote> element:
      </para>
      <programlisting>
static void
some_function (GstElement *tee)
{
  GstPad * pad;
  gchar *name;

  pad = gst_element_get_request_pad (tee, "src%d");
  name = gst_pad_get_name (pad);
  g_print ("A new pad %s was created\n", name);
  g_free (name);

  /* here, you would link the pad */
[..]

  /* and, after doing that, free our reference */
  gst_object_unref (GST_OBJECT (pad));
}
      </programlisting>
      <para> 
        The <function>gst_element_get_request_pad ()</function> method
        can be used to get a pad from the element based on the name of
        the pad template. It is also possible to request a pad that is
        compatible with another pad template. This is very useful if
        you want to link an element to a multiplexer element and you
        need to request a pad that is compatible. The method
        <function>gst_element_get_compatible_pad ()</function> can be
        used to request a compatible pad, as shown in the next example.
        It will request a compatible pad from an Ogg multiplexer from
        any input.
      </para> 
      <programlisting>
static void
link_to_multiplexer (GstPad     *tolink_pad,
		     GstElement *mux)
{
  GstPad *pad;
  gchar *srcname, *sinkname;

  srcname = gst_pad_get_name (tolink_pad);
  pad = gst_element_get_compatible_pad (mux, tolink_pad);
  gst_pad_link (tolinkpad, pad);
  sinkname = gst_pad_get_name (pad);
  gst_object_unref (GST_OBJECT (pad));

  g_print ("A new pad %s was created and linked to %s\n", srcname, sinkname);
  g_free (sinkname);
  g_free (srcname);
}
      </programlisting>
    </sect2>
  </sect1>

  <sect1 id="section-caps">
    <title>Capabilities of a pad</title>
    <para> 
      Since the pads play a very important role in how the element is
      viewed by the outside world, a mechanism is implemented to describe
      the data that can flow or currently flows through the pad by using
      capabilities. Here, we will briefly describe what capabilities are
      and how to use them, enough to get an understanding of the concept.
      For an in-depth look into capabilities and a list of all capabilities
      defined in &GStreamer;, see the <ulink type="http"
      url="http://gstreamer.freedesktop.org/data/doc/gstreamer/head/pwg/html/index.html">Plugin
      Writers Guide</ulink>.
    </para>
    <para>
      Capabilities are attached to pad templates and to pads. For pad
      templates, it will describe the types of media that may stream
      over a pad created from this template. For pads, it can either
      be a list of possible caps (usually a copy of the pad template's
      capabilities), in which case the pad is not yet negotiated, or it
      is the type of media that currently streams over this pad, in
      which case the pad has been negotiated already.
    </para>

    <sect2 id="section-caps-structure">
      <title>Dissecting capabilities</title>
      <para>
        A pad's capabilities are described in a <classname>GstCaps</classname>
        object. Internally, a <ulink type="http"
        url="../../gstreamer/html/gstreamer-GstCaps.html"><classname>GstCaps</classname></ulink>
        will contain one or more <ulink type="http"
        url="../../gstreamer/html/gstreamer-GstStructure.html"><classname>GstStructure</classname></ulink>
        that will describe one media type. A negotiated pad will have
        capabilities set that contain exactly <emphasis>one</emphasis>
        structure. Also, this structure will contain only
        <emphasis>fixed</emphasis> values. These constraints are not
        true for unnegotiated pads or pad templates.
      </para>
      <para>
        As an example, below is a dump of the capabilities of the
        <quote>vorbisdec</quote> element, which you will get by running
        <command>gst-inspect vorbisdec</command>. You will see two pads:
        a source and a sink pad. Both of these pads are always available,
        and both have capabilities attached to them. The sink pad will
        accept vorbis-encoded audio data, with the media type
        <quote>audio/x-vorbis</quote>. The source pad will be used
        to send raw (decoded) audio samples to the next element, with
        a raw audio media type (in this case,
	<quote>audio/x-raw</quote>).  The source pad will also
        contain properties for the audio samplerate and the amount of
        channels, plus some more that you don't need to worry about
        for now.
      </para>
      <programlisting>

Pad Templates:
  SRC template: 'src'
    Availability: Always
    Capabilities:
      audio/x-raw
                 format: F32LE
                   rate: [ 1, 2147483647 ]
               channels: [ 1, 256 ]

  SINK template: 'sink'
    Availability: Always
    Capabilities:
      audio/x-vorbis
      </programlisting>
    </sect2>

    <sect2 id="section-caps-props">
      <title>Properties and values</title>
      <para> 
        Properties are used to describe extra information for 
        capabilities. A property consists of a key (a string) and
        a value. There are different possible value types that can be used:
      </para> 
      <itemizedlist>
        <listitem>
          <para>
            Basic types, this can be pretty much any
            <classname>GType</classname> registered with Glib. Those
            properties indicate a specific, non-dynamic value for this
            property. Examples include:
          </para>
          <itemizedlist>
            <listitem>
              <para>
                An integer value (<classname>G_TYPE_INT</classname>):
                the property has this exact value. 
              </para>
            </listitem>
            <listitem>
              <para>
    	        A boolean value (<classname>G_TYPE_BOOLEAN</classname>):
                the property is either TRUE or FALSE.
              </para>
            </listitem>
            <listitem>
              <para>
                A float value (<classname>G_TYPE_FLOAT</classname>):
                the property has this exact floating point value.
              </para>
            </listitem>
            <listitem>
              <para>
                A string value (<classname>G_TYPE_STRING</classname>):
                the property contains a UTF-8 string.
              </para>
            </listitem>
            <listitem>
              <para>
                A fraction value (<classname>GST_TYPE_FRACTION</classname>):
                contains a fraction expressed by an integer numerator and
                denominator.
              </para>
            </listitem>
          </itemizedlist>
        </listitem>
        <listitem>
          <para>
            Range types are <classname>GType</classname>s registered by
            &GStreamer; to indicate a range of possible values. They are
            used for indicating allowed audio samplerate values or
            supported video sizes. The two types defined in &GStreamer;
            are:
          </para>
          <itemizedlist>
           <listitem>
              <para>
                An integer range value
                (<classname>GST_TYPE_INT_RANGE</classname>): the property
                denotes a range of possible integers, with a lower and an
                upper boundary. The <quote>vorbisdec</quote> element, for
                example, has a rate property that can be between 8000 and
                50000.
              </para>
            </listitem>
            <listitem>
              <para>
    	        A float range value
                (<classname>GST_TYPE_FLOAT_RANGE</classname>): the property
                denotes a range of possible floating point values, with a
                lower and an upper boundary.
              </para>
            </listitem>
            <listitem>
              <para>
    	        A fraction range value
                (<classname>GST_TYPE_FRACTION_RANGE</classname>): the property
                denotes a range of possible fraction values, with a
                lower and an upper boundary.
              </para>
            </listitem>
          </itemizedlist>
        </listitem>
        <listitem>
          <para>
    	    A list value (<classname>GST_TYPE_LIST</classname>): the
            property can take any value from a list of basic values
            given in this list.
          </para>
          <para>
            Example: caps that express that either
            a sample rate of 44100 Hz and a sample rate of 48000 Hz
            is supported would use a list of integer values, with
            one value being 44100 and one value being 48000.
          </para>
        </listitem>
        <listitem>
          <para>
            An array value (<classname>GST_TYPE_ARRAY</classname>): the
            property is an array of values. Each value in the array is a
            full value on its own, too. All values in the array should be
            of the same elementary type. This means that an array can
            contain any combination of integers, lists of integers, integer
            ranges together, and the same for floats or strings, but it can
            not contain both floats and ints at the same time.
          </para>
          <para>
            Example: for audio where there are more than two channels involved
            the channel layout needs to be specified (for one and two channel
            audio the channel layout is implicit unless stated otherwise in the
            caps). So the channel layout would be an array of integer enum
            values where each enum value represents a loudspeaker position.
            Unlike a <classname>GST_TYPE_LIST</classname>, the values in an
            array will be interpreted as a whole.
          </para>
        </listitem>
      </itemizedlist>
    </sect2>
  </sect1>

  <sect1 id="section-caps-api">
    <title>What capabilities are used for</title>
    <para> 
      Capabilities (short: caps) describe the type of data that is streamed
      between two pads, or that one pad (template) supports. This makes them
      very useful for various purposes:
    </para> 
    <itemizedlist>
      <listitem>
        <para>
          Autoplugging: automatically finding elements to link to a
          pad based on its capabilities. All autopluggers use this
          method.
        </para>
      </listitem>
      <listitem>
        <para>
          Compatibility detection: when two pads are linked, &GStreamer;
          can verify if the two pads are talking about the same media
          type. The process of linking two pads and checking if they
          are compatible is called <quote>caps negotiation</quote>.
        </para>
      </listitem>
      <listitem>
        <para>
          Metadata: by reading the capabilities from a pad, applications
          can provide information about the type of media that is being
          streamed over the pad, which is information about the stream
          that is currently being played back.
        </para>
      </listitem>
      <listitem>
        <para>
          Filtering: an application can use capabilities to limit the
          possible media types that can stream between two pads to a
          specific subset of their supported stream types. An application
          can, for example, use <quote>filtered caps</quote> to set a
          specific (fixed or non-fixed) video size that should stream
          between two pads. You will see an example of filtered caps
          later in this manual, in <xref linkend="section-data-spoof"/>.
          You can do caps filtering by inserting a capsfilter element into
          your pipeline and setting its <quote>caps</quote> property. Caps
          filters are often placed after converter elements like audioconvert,
          audioresample, videoconvert or videoscale to force those
          converters to convert data to a specific output format at a
          certain point in a stream.
        </para>
      </listitem>
    </itemizedlist>

    <sect2 id="section-caps-metadata">
      <title>Using capabilities for metadata</title>
      <para> 
        A pad can have a set (i.e. one or more) of capabilities attached
        to it. Capabilities (<classname>GstCaps</classname>) are represented
        as an array of one or more <classname>GstStructure</classname>s, and
        each <classname>GstStructure</classname> is an array of fields where
        each field consists of a field name string (e.g. "width") and a
        typed value (e.g. <classname>G_TYPE_INT</classname> or
        <classname>GST_TYPE_INT_RANGE</classname>).
      </para>
      <para>
        Note that there is a distinct difference between the
        <emphasis>possible</emphasis> capabilities of a pad (ie. usually what
        you find as caps of pad templates as they are shown in gst-inspect),
        the <emphasis>allowed</emphasis> caps of a pad (can be the same as
        the pad's template caps or a subset of them, depending on the possible
        caps of the peer pad) and lastly <emphasis>negotiated</emphasis> caps
        (these describe the exact format of a stream or buffer and contain
        exactly one structure and have no variable bits like ranges or lists,
        ie. they are fixed caps).
      </para>
      <para>
        You can get values of properties in a set of capabilities
        by querying individual properties of one structure. You can get
        a structure from a caps using
        <function>gst_caps_get_structure ()</function> and the number of
        structures in a <classname>GstCaps</classname> using
        <function>gst_caps_get_size ()</function>.
      </para>
      <para>
        Caps are called <emphasis>simple caps</emphasis> when they contain
        only one structure, and <emphasis>fixed caps</emphasis> when they
        contain only one structure and have no variable field types (like
        ranges or lists of possible values). Two other special types of caps
        are <emphasis>ANY caps</emphasis> and <emphasis>empty caps</emphasis>.
      </para>
      <para>
        Here is an example of how to extract the width and height from
        a set of fixed video caps:
      <programlisting>
static void
read_video_props (GstCaps *caps)
{
  gint width, height;
  const GstStructure *str;

  g_return_if_fail (gst_caps_is_fixed (caps));

  str = gst_caps_get_structure (caps, 0);
  if (!gst_structure_get_int (str, "width", &amp;width) ||
      !gst_structure_get_int (str, "height", &amp;height)) {
    g_print ("No width/height available\n");
    return;
  }

  g_print ("The video size of this set of capabilities is %dx%d\n",
	   width, height);
}
      </programlisting>
      </para>
    </sect2>

    <sect2 id="section-caps-filter">
      <title>Creating capabilities for filtering</title>
      <para> 
        While capabilities are mainly used inside a plugin to describe the
        media type of the pads, the application programmer often also has
        to have basic understanding of capabilities in order to interface
        with the plugins, especially when using filtered caps. When you're
        using filtered caps or fixation, you're limiting the allowed types of
        media that can stream between two pads to a subset of their supported
        media types. You do this using a <classname>capsfilter</classname>
        element in your pipeline. In order to do this, you also need to
        create your own <classname>GstCaps</classname>. The easiest way to
        do this is by using the convenience function
        <function>gst_caps_new_simple ()</function>:
      </para>
      <para>
      <programlisting>
static gboolean
link_elements_with_filter (GstElement *element1, GstElement *element2)
{
  gboolean link_ok;
  GstCaps *caps;

  caps = gst_caps_new_simple ("video/x-raw",
  	      "format", G_TYPE_STRING, "I420",
	      "width", G_TYPE_INT, 384,
	      "height", G_TYPE_INT, 288,
	      "framerate", GST_TYPE_FRACTION, 25, 1,
	      NULL);

  link_ok = gst_element_link_filtered (element1, element2, caps);
  gst_caps_unref (caps);

  if (!link_ok) {
    g_warning ("Failed to link element1 and element2!");
  }

  return link_ok;
}
      </programlisting>
      This will force the data flow between those two elements to
      a certain video format, width, height and framerate (or the linking
      will fail if that cannot be achieved in the context of the elements
      involved). Keep in mind that when you use <function>
      gst_element_link_filtered ()</function> it will automatically create
      a <classname>capsfilter</classname> element for you and insert it into
      your bin or pipeline between the two elements you want to connect (this
      is important if you ever want to disconnect those elements because then
      you will have to disconnect both elements from the capsfilter instead).
      </para>
      <para>
        In some cases, you will want to create a more elaborate set of
        capabilities to filter a link between two pads. Then, this function
        is too simplistic and you'll want to use the method
        <function>gst_caps_new_full ()</function>:
      </para>
      <programlisting>
static gboolean
link_elements_with_filter (GstElement *element1, GstElement *element2)
{
  gboolean link_ok;
  GstCaps *caps;
                                                                                
  caps = gst_caps_new_full (
      gst_structure_new ("video/x-raw",
			 "width", G_TYPE_INT, 384,
			 "height", G_TYPE_INT, 288,
			 "framerate", GST_TYPE_FRACTION, 25, 1,
			 NULL),
      gst_structure_new ("video/x-bayer",
			 "width", G_TYPE_INT, 384,
			 "height", G_TYPE_INT, 288,
			 "framerate", GST_TYPE_FRACTION, 25, 1,
			 NULL),
      NULL);

  link_ok = gst_element_link_filtered (element1, element2, caps);
  gst_caps_unref (caps);

  if (!link_ok) {
    g_warning ("Failed to link element1 and element2!");
  }

  return link_ok;
}
      </programlisting>
      <para>
        See the API references for the full API of
	<ulink type="http"
	  url="&URLAPI;GstStructure.html"><classname>GstStructure</classname></ulink>
	and <ulink type="http"
	  url="&URLAPI;GstCaps.html"><classname>GstCaps</classname></ulink>.
      </para>
    </sect2>
  </sect1>

  <sect1 id="section-pads-ghost">
    <title>Ghost pads</title>
    <para>
      You can see from <xref linkend="section-bin-noghost-img"/> how a bin
      has no pads of its own. This is where "ghost pads" come into play.
    </para>
    <figure float="1" id="section-bin-noghost-img">
      <title>Visualisation of a <ulink type="http"
      url="../../gstreamer/html/GstBin.html"><classname>GstBin</classname></ulink>
      element without ghost pads</title>
      <mediaobject>
        <imageobject>
          <imagedata scale="75" fileref="images/bin-element-noghost.&image;"
          format="&IMAGE;"/>
        </imageobject>
      </mediaobject>  
    </figure>
    <para>
      A ghost pad is a pad from some element in the bin that can be
      accessed directly from the bin as well. Compare it to a symbolic
      link in UNIX filesystems. Using ghost pads on bins, the bin also
      has a pad and can transparently be used as an element in other
      parts of your code.
    </para>
    
    <figure float="1" id="section-bin-ghost-img">
      <title>Visualisation of a <ulink type="http"
      url="../../gstreamer/html/GstBin.html"><classname>GstBin</classname></ulink>
      element with a ghost pad</title>
      <mediaobject>
        <imageobject>
          <imagedata scale="75" fileref="images/bin-element-ghost.&image;"
          format="&IMAGE;"/>
        </imageobject>
      </mediaobject>  
    </figure>
    <para>
      <xref linkend="section-bin-ghost-img"/> is a representation of a
      ghost pad. The sink pad of element one is now also a pad of the bin.
      Because ghost pads look and work like any other pads, they can be added 
      to any type of elements, not just to a <classname>GstBin</classname>,
      just like ordinary pads.
    </para>
    <para>
      A ghostpad is created using the function
      <function>gst_ghost_pad_new ()</function>:
    </para>
    <programlisting><!-- example-begin ghostpad.c a -->
#include &lt;gst/gst.h&gt;

int
main (int   argc,
      char *argv[])
{
  GstElement *bin, *sink;
  GstPad *pad;

  /* init */
  gst_init (&amp;argc, &amp;argv);

  /* create element, add to bin */
  sink = gst_element_factory_make ("fakesink", "sink");
  bin = gst_bin_new ("mybin");
  gst_bin_add (GST_BIN (bin), sink);

  /* add ghostpad */
  pad = gst_element_get_static_pad (sink, "sink");
  gst_element_add_pad (bin, gst_ghost_pad_new ("sink", pad));
  gst_object_unref (GST_OBJECT (pad));
<!-- example-end ghostpad.c a -->
[..]<!-- example-begin ghostpad.c b --><!--
  return 0;
--><!-- example-end ghostpad.c b -->
<!-- example-begin ghostpad.c c -->
}
    <!-- example-end ghostpad.c c --></programlisting>
    <para>
      In the above example, the bin now also has a pad: the pad called
      <quote>sink</quote> of the given element. The bin can, from here
      on, be used as a substitute for the sink element. You could, for
      example, link another element to the bin.
    </para>
  </sect1>
</chapter>