gstreamer/docs/pwg/advanced-negotiation.xml

<chapter id="chapter-negotiation" xreflabel="Caps negotiation">
  <title>Caps negotiation</title>
  <para>
    Caps negotiation is the act of finding a media format (GstCaps) between
    elements that they can handle. This process in &GStreamer; can in most
    cases find an optimal solution for the complete pipeline. In this section
    we explain how this works.
  </para>

  <sect1 id="section-nego-basics">
    <title>Caps negotiation basics</title>
    <para>
      In &GStreamer;, negotiation of the media format always follows the
      following simple rules:
    </para>
    <itemizedlist>
      <listitem>
        <para>
          A downstream element suggest a format on its sinkpad and places the
          suggestion in the result of the CAPS query performed on the sinkpad.
          See also <xref linkend="section-nego-getcaps"/>.
        </para>
      </listitem>
      <listitem>
        <para>
          An upstream element decides on a format. It sends the selected media
          format downstream on its source pad with a CAPS event. Downstream
          elements reconfigure themselves to handle the media type in the CAPS
          event on the sinkpad.
        </para>
      </listitem>
      <listitem>
        <para>
          An upstream element can inform downstream that it would like to
          suggest a new format by sending a RECONFIGURE event upstream. The
          RECONFIGURE event simply instructs an upstream element to restart
          the negotiation phase. Because the element that sent out the
          RECONFIGURE event is now suggesting another format, the format
          in the pipeline might change.
        </para>
      </listitem>
    </itemizedlist>
    <para>
      In addition to the CAPS and RECONFIGURE event and the CAPS query, there
      is an ACCEPT_CAPS query to quickly check if a certain caps can
      be accepted by an element.
    </para>
    <para>
      All negotiation follows these simple rules. Let's take a look at some
      typical uses cases and how negotiation happens.
    </para>
  </sect1>

  <sect1 id="section-nego-usecases">
    <title>Caps negotiation use cases</title>
    <para>
      In what follows we will look at some use cases for push-mode scheduling.
      The pull-mode scheduling negotiation phase is discussed in
      <xref linkend="section-nego-pullmode"/> and is actually similar as we
      will see. 
    </para>
    <para>
      Since the sink pads only suggest formats and the source pads need to
      decide, the most complicated work is done in the source pads.
      We can identify 3 caps negotiation use cases for the source pads:
    </para>
    <itemizedlist>
      <listitem>
        <para>
          Fixed negotiation. An element can output one format only.
          See <xref linkend="section-nego-fixed"/>.
        </para>
      </listitem>
      <listitem>
        <para>
          Transform negotiation. There is a (fixed) transform between the
          input and output format of the element, usually based on some
          element property. The caps that the element will produce depend
          on the upstream caps and the caps that the element can accept
          depend on the downstream caps.
          See <xref linkend="section-nego-transform"/>.
        </para>
      </listitem>
      <listitem>
        <para>
          Dynamic negotiation. An element can output many formats.
          See <xref linkend="section-nego-dynamic"/>.
        </para>
      </listitem>
    </itemizedlist>

    <sect2 id="section-nego-fixed">
      <title>Fixed negotiation</title>
      <para>
        In this case, the source pad can only produce a fixed format. Usually
        this format is encoded inside the media. No downstream element can
        ask for a different format, the only way that the source pad will
        renegotiate is when the element decides to change the caps itself.
      </para>
      <para>
        Elements that could implement fixed caps (on their source pads) are,
        in general, all elements that are not renegotiable. Examples include:
      </para>
      <itemizedlist>
        <listitem>
          <para>
            A typefinder, since the type found is part of the actual data stream
            and can thus not be re-negotiated. The typefinder will look at the
            stream of bytes, figure out the type, send a CAPS event with the
            caps and then push buffers of the type.
          </para>
        </listitem>
        <listitem>
          <para>
            Pretty much all demuxers, since the contained elementary data
            streams are defined in the file headers, and thus not
            renegotiable.
          </para>
        </listitem>
        <listitem>
          <para>
            Some decoders, where the format is embedded in the data stream
            and not part of the peercaps <emphasis>and</emphasis> where the
            decoder itself is not reconfigurable, too.
          </para>
        </listitem>
      </itemizedlist>
      <para>
        <function>gst_pad_use_fixed_caps()</function> is used on the source
        pad with fixed caps. As long as the pad is not negotiated, the default
        CAPS query will return the caps presented in the padtemplate. As soon
        as the pad is negotiated, the CAPS query will return the negotiated
        caps (and nothing else). These are the relevant code snippets for fixed
        caps source pads.
      </para>
      <programlisting>
<![CDATA[
[..]
  pad = gst_pad_new_from_static_template (..);
  gst_pad_use_fixed_caps (pad);
[..]
]]>
      </programlisting>
      <para>
        The fixed caps can then be set on the pad by calling
        <function>gst_pad_set_caps ()</function>.
      </para>
      <programlisting>
<![CDATA[
[..]
    caps = gst_caps_new_simple ("audio/x-raw",
        "format", G_TYPE_STRING, GST_AUDIO_NE(F32),
        "rate", G_TYPE_INT, <samplerate>,
        "channels", G_TYPE_INT, <num-channels>, NULL);
    if (!gst_pad_set_caps (pad, caps)) {
      GST_ELEMENT_ERROR (element, CORE, NEGOTIATION, (NULL),
          ("Some debug information here"));
      return GST_FLOW_ERROR;
    }
[..]
]]>
      </programlisting>
      <para>
        All other elements that need to be configured for the format should
        implement full caps negotiation, which will be explained in the next
        few sections.
      </para>
    </sect2>

    <sect2 id="section-nego-transform">
      <title>Transform negotiation</title>
      <para>
      </para>
    </sect2>

    <sect2 id="section-nego-dynamic">
      <title>Dynamic negotiation</title>
      <para>
      </para>
    </sect2>
  </sect1>

  <sect1 id="section-nego-pullmode">
    <title>Pull-mode Caps negotiation</title>
    <para>
    </para>
  </sect1>

<!--
  <sect1 id="section-nego-old">
    <title>Old stuff</title>
    <para>
      Let's take the case of a file source, linked to a demuxer, linked to a
      decoder, linked to a converter with a caps filter and finally an audio
      output. When data flow originally starts, the demuxer will parse the
      file header (e.g. the Ogg headers), and notice that there is, for
      example, a Vorbis stream in this Ogg file. Noticing that, it will
      create an output pad for the Vorbis elementary stream and set a
      Vorbis-caps on it. Lastly, it adds the pad. As of this point, the pad
      is ready to be used to stream data, and so the Ogg demuxer is now done.
      This pad is <emphasis>not</emphasis> re-negotiable, since the type of
      the data stream is embedded within the data.
    </para>
    <para>
      The Vorbis decoder will decode the Vorbis headers and the Vorbis data
      coming in on its sinkpad. Now, some decoders may be able to output in
      multiple output formats, for example both 16-bit integer output and
      floating-point output, whereas other decoders may be able to only decode
      into one specific format, e.g. only floating-point (32-bit) audio. Those
      two cases have consequences for how caps negotiation should be
      implemented in this decoder element. In the one case, it is possible to
      use fixed caps, and you're done. In the other case, however, you should
      implement the possibility for <emphasis>renegotiation</emphasis> in this
      element, which is the possibility for the data format to be changed to
      another format at some point in the future. We will discuss how to do
      this in one of the sections further on in this chapter.
    </para>
    <para>
      The filter can be used by applications to force, for example, a specific
      channel configuration (5.1/surround or 2.0/stereo), on the pipeline, so
      that the user can enjoy sound coming from all its speakers. The audio
      sink, in this example, is a standard ALSA output element (alsasink).
      The converter element supports any-to-any, and the filter will make sure
      that only a specifically wanted channel configuration streams through
      this link (as provided by the user's channel configuration preference).
      By changing this preference while the pipeline is running, some elements
      will have to renegotiate <emphasis>while the pipeline is
      running</emphasis>. This is done through upstream caps renegotiation.
      That, too, will be discussed in detail in a section further below.
    </para>
    <para>
      In order for caps negotiation on non-fixed links to work correctly,
      pads can optionally implement a query function that tells peer elements
      what formats it supports and/or prefers. When upstream renegotiation is
      triggered, this becomes important.
    </para>
    <para>
      Downstream elements are notified of a newly set caps with a
      GST_EVENT_CAPS on the sinkpad. So when the vorbis decoder sets a caps on
      its source pad (to configure the output format), the converter will
      receive a caps event. 
      When an element receives a buffer, it should check if it has received
      all needed format information in a CAPS event previously. If it hasn't,
      it should return an error from the chain function.
    </para>
  </sect1>
-->

  <sect1 id="section-nego-downstream" xreflabel="Downstream caps negotiation">
    <title>Downstream caps negotiation</title>
    <para>
      Downstream negotiation takes place when a format needs to be set on a
      source pad to configure the output format, but this element allows
      renegotiation because its format is configured on the sinkpad caps,
      or because it supports multiple formats. The requirements for doing
      the actual negotiation differ slightly.
    </para>

    <sect2 id="section-nego-downstream-embed"
        xreflabel="Negotiating caps embedded in input caps">
      <title>Negotiating caps embedded in input caps</title>
      <para>
        Many elements, particularly effects and converters, will be able
        to parse the format of the stream from their input caps, and decide
        the output format right at that time already. For those elements, all
        (downstream) caps negotiation can be done from the 
        <function>_event ()</function> function when a GST_EVENT_CAPS is
        received on the sinkpad. This CAPS event is received whenever the
        format changes or when no format was negotiated yet. It will always
        be called before you receive the buffer in the format specified in
        the CAPS event.
      </para>
      <para>
        In the <function>_event ()</function>-function, the element can
        forward the CAPS event to the next element and, if that pad accepts the
        format too, the element can parse the relevant parameters from the
        caps and configure itself internally. The caps passed to this function
        is <emphasis>always</emphasis> a subset of the template caps, so
        there's no need for extensive safety checking. The following example
        should give a clear indication of how such a function can be
        implemented:
      </para>
      <programlisting><!-- example-begin forwardcaps.c a --><!--
#include "init.func"
static GstCaps *
gst_my_filter_getcaps (GstPad * pad)
{
  return  NULL;
}
--><!-- example-end forwardcaps.c a -->
<!-- example-begin forwardcaps.c b -->
static gboolean
gst_my_filter_sink_event (GstPad    *pad,
		          GstObject *parent,
		          GstEvent  *event)
{
  gboolean ret;
  GstMyFilter *filter = GST_MY_FILTER (parent);

  switch (GST_EVENT_TYPE (event)) {
    case GST_EVENT_CAPS:
    {
      GstCaps *caps;
      GstStructure *s;

      gst_event_parse_caps (event, &amp;caps);

      /* forward-negotiate */
      ret = gst_pad_set_caps (filter-&gt;srcpad, caps);
      if (!ret)
        return FALSE;

      /* negotiation succeeded, so now configure ourselves */
      s = gst_caps_get_structure (caps, 0);
      gst_structure_get_int (s, "rate", &amp;filter-&gt;samplerate);
      gst_structure_get_int (s, "channels", &amp;filter-&gt;channels);
      break;
    }
    default:
      ret = gst_pad_event_default (pad, parent, event);
      break;
  }
  return ret;
}
<!-- example-end forwardcaps.c b -->
<!-- example-begin forwardcaps.c c --><!--
#include "chain.func"
#include "state.func"
#include "register.func"
      --><!-- example-end forwardcaps.c c --></programlisting>
      <para>
        There may also be cases where the filter actually is able to
        <emphasis>change</emphasis> the format of the stream. In those cases,
        it will negotiate a new format. Obviously, the element should first
        attempt to configure <quote>pass-through</quote>, which means that
        it does not change the stream's format. However, if that fails,
        then it should call <function>gst_pad_get_allowed_caps ()</function>
        on its sourcepad to get a list of supported formats on the outputs,
        and pick the first. The return value of that function is guaranteed
        to be a subset of the template caps or NULL when there is no peer.
      </para>
      <para>
        Let's look at the example of an element that can convert between
        samplerates, so where input and output samplerate don't have to be
        the same:
      </para>
      <programlisting><!-- example-begin convertcaps.c a --><!--
#include "init.func"
static GstCaps *
gst_my_filter_getcaps (GstPad * pad)
{
  return NULL;
}
static GstBuffer *
gst_my_filter_convert (GstMyFilter *filter, GstBuffer *in)
{
  return NULL;
}
static gboolean
gst_my_filter_event (GstPad * pad, GstEvent * event)
{
  return gst_pad_event_default (pad, event);
}
--><!-- example-end convertcaps.c a -->
<!-- example-begin convertcaps.c b -->
static gboolean
gst_my_filter_setcaps (GstMyFilter *filter,
		       GstCaps *caps)
{
  if (gst_pad_set_caps (filter-&gt;sinkpad, caps)) {
    filter-&gt;passthrough = TRUE;
  } else {
    GstCaps *othercaps, *newcaps;
    GstStructure *s = gst_caps_get_structure (caps, 0), *others;

    /* no passthrough, setup internal conversion */
    gst_structure_get_int (s, "channels", &amp;filter-&gt;channels);
    othercaps = gst_pad_get_allowed_caps (filter-&gt;srcpad);
    others = gst_caps_get_structure (othercaps, 0);
    gst_structure_set (others,
        "channels", G_TYPE_INT, filter-&gt;channels, NULL);

    /* now, the samplerate value can optionally have multiple values, so
     * we "fixate" it, which means that one fixed value is chosen */
    newcaps = gst_caps_copy_nth (othercaps, 0);
    gst_caps_unref (othercaps);
    gst_pad_fixate_caps (filter-&gt;srcpad, newcaps);
    if (!gst_pad_set_caps (filter-&gt;srcpad, newcaps))
      return FALSE;

    /* we are now set up, configure internally */
    filter-&gt;passthrough = FALSE;
    gst_structure_get_int (s, "rate", &amp;filter-&gt;from_samplerate);
    others = gst_caps_get_structure (newcaps, 0);
    gst_structure_get_int (others, "rate", &amp;filter-&gt;to_samplerate);
  }

  return TRUE;
}

static gboolean
gst_my_filter_sink_event (GstPad    *pad,
		          GstObject *parent,
		          GstEvent  *event)
{
  gboolean ret;
  GstMyFilter *filter = GST_MY_FILTER (parent);

  switch (GST_EVENT_TYPE (event)) {
    case GST_EVENT_CAPS:
    {
      GstCaps *caps;

      gst_event_parse_caps (event, &amp;caps);
      ret = gst_my_filter_setcaps (filter, caps);
      break;
    }
    default:
      ret = gst_pad_event_default (pad, parent, event);
      break;
  }
  return ret;
}

static GstFlowReturn
gst_my_filter_chain (GstPad    *pad,
		     GstObject *parent,
		     GstBuffer *buf)
{
  GstMyFilter *filter = GST_MY_FILTER (parent);
  GstBuffer *out;

  /* push on if in passthrough mode */
  if (filter-&gt;passthrough)
    return gst_pad_push (filter-&gt;srcpad, buf);

  /* convert, push */
  out = gst_my_filter_convert (filter, buf);
  gst_buffer_unref (buf);

  return gst_pad_push (filter-&gt;srcpad, out);
}
<!-- example-end convertcaps.c b -->
<!-- example-begin convertcaps.c c --><!--
#include "state.func"
#include "register.func"
      --><!-- example-end convertcaps.c c --></programlisting>
    </sect2>

    <sect2 id="section-nego-downstream-parse"
        xreflabel="Parsing and setting caps">
      <title>Parsing and setting caps</title>
      <para>
        Other elements, such as certain types of decoders, will not be able
        to parse the caps from their input, simply because the input format
        does not contain the information required to know the output format
        yet; rather, the data headers need to be parsed, too. In many cases,
        fixed-caps will be enough, but in some cases, particularly in cases
        where such decoders are renegotiable, it is also possible to use
        full caps negotiation.
      </para>
      <para>
        Fortunately, the code required to do so is very similar to the last
        code example in <xref linkend="section-nego-downstream-embed"/>, with
        the difference being that the caps is selected in the <function>_chain
        ()</function>-function rather than in the <function>_event
        ()</function>-function. The rest, as for getting all allowed caps from
        the source pad, fixating and such, is all the same. Re-negotiation,
        which will be handled in the next section, is very different for such
        elements, though.
      </para>
    </sect2>
  </sect1>

  <sect1 id="section-nego-upstream" xreflabel="Upstream caps (re)negotiation">
    <title>Upstream caps (re)negotiation</title>
    <para>
      Upstream negotiation's primary use is to renegotiate (part of) an
      already-negotiated pipeline to a new format. Some practical examples
      include to select a different video size because the size of the video
      window changed, and the video output itself is not capable of rescaling,
      or because the audio channel configuration changed.
    </para>
    <para>
      Upstream caps renegotiation is requested by sending a GST_EVENT_RECONFIGURE
      event upstream. The idea is that it will instruct the upstream element
      to reconfigure its caps by doing a new query for the allowed caps and then
      choosing a new caps. The element that sends out the RECONFIGURE event
      would influence the selection of the new caps by returning the new
      prefered caps from its GST_QUERY_CAPS query function. The RECONFIGURE
      event will set the GST_PAD_FLAG_NEED_RECONFIGURE on all pads that it
      travels over.
    </para>
    <para>
      It is important to note here that different elements actually have
      different responsibilities here:
    </para>
    <itemizedlist>
      <listitem>
        <para>
          Elements that can be reconfigured on the srcpad should check its
          NEED_RECONFIGURE flag with
          <function>gst_pad_check_reconfigure ()</function> and it should
          start renegotiation when the function returns TRUE.
        </para>
      </listitem>
      <listitem>
        <para>
          Elements that want to propose a new format upstream need to send
          a RECONFIGURE event and be prepared to answer the CAPS query with
          the new prefered format. It should be noted that when there is no
          upstream element that can (or wants) to renegotiate, the element
          needs to deal with the currently configured format.
        </para>
      </listitem>
    </itemizedlist>
  </sect1>

  <sect1 id="section-nego-getcaps" xreflabel="Implementing a CAPS query function">
    <title>Implementing a CAPS query function</title>
    <para>
      A <function>_query ()</function>-function with the GST_QUERY_CAPS query
      type is called when a peer element would like to know which formats
      this pad supports, and in what order of preference. The return value
      should be all formats that this elements supports, taking into account
      limitations of peer elements further downstream or upstream, sorted by
      order of preference, highest preference first.
    </para>
    <para>
    </para>
    <programlisting><!-- example-begin getcaps.c a --><!--
#include "init.func"
--><!-- example-end getcaps.c a -->
<!-- example-begin getcaps.c b -->
static gboolean
gst_my_filter_query (GstPad *pad, GstObject * parent, GstQuery * query)
{
  gboolean ret;
  GstMyFilter *filter = GST_MY_FILTER (parent);

  switch (GST_QUERY_TYPE (query)) {
    case GST_QUERY_CAPS
    {
      GstPad *otherpad;
      GstCaps *temp, *caps, *filter, *tcaps;
      gint i;

      otherpad = (pad == filter-&gt;srcpad) ? filter-&gt;sinkpad :
                                              filter-&gt;srcpad;
      caps = gst_pad_get_allowed_caps (otherpad);

      gst_query_parse_caps (query, &amp;filter);

      /* We support *any* samplerate, indifferent from the samplerate
       * supported by the linked elements on both sides. */
      for (i = 0; i &lt; gst_caps_get_size (caps); i++) {
        GstStructure *structure = gst_caps_get_structure (caps, i);

        gst_structure_remove_field (structure, "rate");
      }

      /* make sure we only return results that intersect our
       * padtemplate */
      tcaps = gst_pad_get_pad_template_caps (pad);
      if (tcaps) {
        temp = gst_caps_intersect (caps, tcaps);
        gst_caps_unref (caps);
        gst_caps_unref (tcaps);
        caps = temp;
      }
      /* filter against the query filter when needed */
      if (filter) {
        temp = gst_caps_intersect (caps, filter);
        gst_caps_unref (caps);
        caps = temp;
      }
      gst_query_set_caps_result (query, caps);
      gst_caps_unref (caps);
      ret = TRUE;
      break;
    }
    default:
      ret = gst_pad_query_default (pad, parent, query);
      break;
  }
  return ret;
}
<!-- example-end getcaps.c b -->
<!-- example-begin getcaps.c c --><!--
static gboolean
gst_my_filter_setcaps (GstPad * pad, GstCaps * caps)
{
  return FALSE;
}
#include "chain.func"
#include "state.func"
#include "register.func"
    --><!-- example-end getcaps.c c --></programlisting>
    <para>
      Using all the knowledge you've acquired by reading this chapter, you
      should be able to write an element that does correct caps negotiation.
      If in doubt, look at other elements of the same type in our git
      repository to get an idea of how they do what you want to do.
    </para>
  </sect1>
</chapter>