mirror of
https://gitlab.freedesktop.org/gstreamer/gstreamer.git
synced 2024-09-08 13:18:52 +00:00
439 lines
18 KiB
XML
439 lines
18 KiB
XML
<chapter id="chapter-negotiation" xreflabel="Caps negotiation">
|
|
<title>Caps negotiation</title>
|
|
<para>
|
|
Caps negotiation is the process where elements configure themselves
|
|
and each other for streaming a particular media format over their pads.
|
|
Since different types of elements have different requirements for the
|
|
media formats they can negotiate to, it is important that this process
|
|
is generic and implements all those use cases correctly.
|
|
</para>
|
|
<para>
|
|
In this chapter, we will discuss downstream negotiation and upstream
|
|
negotiation from a pipeline perspective, implicating the responsibilities
|
|
of different types of elements in a pipeline, and we will introduce the
|
|
concept of <emphasis>fixed caps</emphasis>.
|
|
</para>
|
|
|
|
<sect1 id="section-nego-requirements" xreflabel="Caps negotiation use cases">
|
|
<title>Caps negotiation use cases</title>
|
|
<para>
|
|
Let's take the case of a file source, linked to a demuxer, linked to a
|
|
decoder, linked to a converter with a caps filter and finally an audio
|
|
output. When data flow originally starts, the demuxer will parse the
|
|
file header (e.g. the Ogg headers), and notice that there is, for
|
|
example, a Vorbis stream in this Ogg file. Noticing that, it will
|
|
create an output pad for the Vorbis elementary stream and set a
|
|
Vorbis-caps on it. Lastly, it adds the pad. As of this point, the pad
|
|
is ready to be used to stream data, and so the Ogg demuxer is now done.
|
|
This pad is <emphasis>not</emphasis> re-negotiable, since the type of
|
|
the data stream is embedded within the data.
|
|
</para>
|
|
<para>
|
|
The Vorbis decoder will decode the Vorbis headers and the Vorbis data
|
|
coming in on its sinkpad. Now, some decoders may be able to output in
|
|
multiple output formats, for example both 16-bit integer output and
|
|
floating-point output, whereas other decoders may be able to only decode
|
|
into one specific format, e.g. only floating-point (32-bit) audio. Those
|
|
two cases have consequences for how caps negotiation should be
|
|
implemented in this decoder element. In the one case, it is possible to
|
|
use fixed caps, and you're done. In the other case, however, you should
|
|
implement the possibility for <emphasis>renegotiation</emphasis> in this
|
|
element, which is the possibility for the data format to be changed to
|
|
another format at some point in the future. We will discuss how to do
|
|
this in one of the sections further on in this chapter.
|
|
</para>
|
|
<para>
|
|
The filter can be used by applications to force, for example, a specific
|
|
channel configuration (5.1/surround or 2.0/stereo), on the pipeline, so
|
|
that the user can enjoy sound coming from all its speakers. The audio
|
|
sink, in this example, is a standard ALSA output element (alsasink).
|
|
The converter element supports any-to-any, and the filter will make sure
|
|
that only a specifically wanted channel configuration streams through
|
|
this link (as provided by the user's channel configuration preference).
|
|
By changing this preference while the pipeline is running, some elements
|
|
will have to renegotiate <emphasis>while the pipeline is
|
|
running</emphasis>. This is done through upstream caps renegotiation.
|
|
That, too, will be discussed in detail in a section further below.
|
|
</para>
|
|
<para>
|
|
In order for caps negotiation on non-fixed links to work correctly,
|
|
pads can optionally implement a function that tells peer elements what
|
|
formats it supports and/or prefers. When upstream renegotiation is
|
|
triggered, this becomes important.
|
|
</para>
|
|
<para>
|
|
Downstream elements are notified of a newly set caps only when data
|
|
is actually passing their pad. This is because caps is attached to
|
|
buffers during data flow. So when the vorbis decoder sets a caps on
|
|
its source pad (to configure the output format), the converter will
|
|
not yet be notified. Instead, the converter will only be notified
|
|
when the decoder pushes a buffer over its source pad to the converter.
|
|
Right before calling the chain-function in the converter, &GStreamer;
|
|
will check whether the format that was previously negotiated still
|
|
applies to this buffer. If not, it first calls the setcaps-function
|
|
of the converter to configure it for the new format. Only after that
|
|
will it call the chain function of the converter.
|
|
</para>
|
|
</sect1>
|
|
|
|
<sect1 id="section-nego-fixedcaps" xreflabel="Fixed caps">
|
|
<title>Fixed caps</title>
|
|
<para>
|
|
The simplest way in which to do caps negotiation is setting a fixed
|
|
caps on a pad. After a fixed caps has been set, the pad can not be
|
|
renegotiated from the outside. The only way to reconfigure the pad
|
|
is for the element owning the pad to set a new fixed caps on the pad.
|
|
Fixed caps is a setup property for pads, called when creating the pad:
|
|
</para>
|
|
<programlisting>
|
|
[..]
|
|
pad = gst_pad_new_from_template (..);
|
|
gst_pad_use_fixed_caps (pad);
|
|
[..]
|
|
</programlisting>
|
|
<para>
|
|
The fixed caps can then be set on the pad by calling
|
|
<function>gst_pad_set_caps ()</function>.
|
|
</para>
|
|
<programlisting>
|
|
[..]
|
|
caps = gst_caps_new_simple ("audio/x-raw-float",
|
|
"width", G_TYPE_INT, 32,
|
|
"endianness", G_TYPE_INT, G_BYTE_ORDER,
|
|
"buffer-frames", G_TYPE_INT, <bytes-per-frame>,
|
|
"rate", G_TYPE_INT, <samplerate>,
|
|
"channels", G_TYPE_INT, <num-channels>, NULL);
|
|
if (!gst_pad_set_caps (pad, caps)) {
|
|
GST_ELEMENT_ERROR (element, CORE, NEGOTIATION, (NULL),
|
|
("Some debug information here"));
|
|
return GST_FLOW_ERROR;
|
|
}
|
|
[..]
|
|
</programlisting>
|
|
<para>
|
|
Elements that could implement fixed caps (on their source pads) are,
|
|
in general, all elements that are not renegotiable. Examples include:
|
|
</para>
|
|
<itemizedlist>
|
|
<listitem>
|
|
<para>
|
|
A typefinder, since the type found is part of the actual data stream
|
|
and can thus not be re-negotiated.
|
|
</para>
|
|
</listitem>
|
|
<listitem>
|
|
<para>
|
|
Pretty much all demuxers, since the contained elementary data
|
|
streams are defined in the file headers, and thus not
|
|
renegotiable.
|
|
</para>
|
|
</listitem>
|
|
<listitem>
|
|
<para>
|
|
Some decoders, where the format is embedded in the data stream
|
|
and not part of the peercaps <emphasis>and</emphasis> where the
|
|
decoder itself is not reconfigurable, too.
|
|
</para>
|
|
</listitem>
|
|
</itemizedlist>
|
|
<para>
|
|
All other elements that need to be configured for the format should
|
|
implement full caps negotiation, which will be explained in the next
|
|
few sections.
|
|
</para>
|
|
</sect1>
|
|
|
|
<sect1 id="section-nego-downstream" xreflabel="Downstream caps negotiation">
|
|
<title>Downstream caps negotiation</title>
|
|
<para>
|
|
Downstream negotiation takes place when a format needs to be set on a
|
|
source pad to configure the output format, but this element allows
|
|
renegotiation because its format is configured on the sinkpad caps,
|
|
or because it supports multiple formats. The requirements for doing
|
|
the actual negotiation differ slightly.
|
|
</para>
|
|
|
|
<sect2 id="section-nego-downstream-embed"
|
|
xreflabel="Negotiating caps embedded in input caps">
|
|
<title>Negotiating caps embedded in input caps</title>
|
|
<para>
|
|
Many elements, particularly effects and converters, will be able
|
|
to parse the format of the stream from their input caps, and decide
|
|
the output format right at that time already. When renegotiation
|
|
takes place, some may merely need to "forward" the renegotiation
|
|
backwards upstream (more on that later). For those elements, all
|
|
(downstream) caps negotiation can be done in something that we
|
|
call the <function>_setcaps ()</function> function. This function is
|
|
called when a buffer is pushed over a pad, but the format on this
|
|
buffer is not the same as the format that was previously negotiated
|
|
(or, similarly, no format was negotiated yet so far).
|
|
</para>
|
|
<para>
|
|
In the <function>_setcaps ()</function>-function, the element can
|
|
forward the caps to the next element and, if that pad accepts the
|
|
format too, the element can parse the relevant parameters from the
|
|
caps and configure itself internally. The caps passed to this function
|
|
is <emphasis>always</emphasis> a subset of the template caps, so
|
|
there's no need for extensive safety checking. The following example
|
|
should give a clear indication of how such a function can be
|
|
implemented:
|
|
</para>
|
|
<programlisting><!-- example-begin forwardcaps.c a --><!--
|
|
#include "init.func"
|
|
static GstCaps *
|
|
gst_my_filter_getcaps (GstPad * pad)
|
|
{
|
|
return NULL;
|
|
}
|
|
--><!-- example-end forwardcaps.c a -->
|
|
<!-- example-begin forwardcaps.c b -->
|
|
static gboolean
|
|
gst_my_filter_setcaps (GstPad *pad,
|
|
GstCaps *caps)
|
|
{
|
|
GstMyFilter *filter = GST_MY_FILTER (GST_OBJECT_PARENT (pad));
|
|
GstStructure *s;
|
|
|
|
/* forward-negotiate */
|
|
if (!gst_pad_set_caps (filter->srcpad, caps))
|
|
return FALSE;
|
|
|
|
/* negotiation succeeded, so now configure ourselves */
|
|
s = gst_caps_get_structure (caps, 0);
|
|
gst_structure_get_int (s, "rate", &filter->samplerate);
|
|
gst_structure_get_int (s, "channels", &filter->channels);
|
|
|
|
return TRUE;
|
|
}
|
|
<!-- example-end forwardcaps.c b -->
|
|
<!-- example-begin forwardcaps.c c --><!--
|
|
#include "chain.func"
|
|
#include "state.func"
|
|
#include "register.func"
|
|
--><!-- example-end forwardcaps.c c --></programlisting>
|
|
<para>
|
|
There may also be cases where the filter actually is able to
|
|
<emphasis>change</emphasis> the format of the stream. In those cases,
|
|
it will negotiate a new format. Obviously, the element should first
|
|
attempt to configure <quote>pass-through</quote>, which means that
|
|
it does not change the stream's format. However, if that fails,
|
|
then it should call <function>gst_pad_get_allowed_caps ()</function>
|
|
on its sourcepad to get a list of supported formats on the outputs,
|
|
and pick the first. The return value of that function is guaranteed
|
|
to be a subset of the template caps.
|
|
</para>
|
|
<para>
|
|
Let's look at the example of an element that can convert between
|
|
samplerates, so where input and output samplerate don't have to be
|
|
the same:
|
|
</para>
|
|
<programlisting><!-- example-begin convertcaps.c a --><!--
|
|
#include "init.func"
|
|
static GstCaps *
|
|
gst_my_filter_getcaps (GstPad * pad)
|
|
{
|
|
return NULL;
|
|
}
|
|
static GstBuffer *
|
|
gst_my_filter_convert (GstMyFilter *filter, GstBuffer *in)
|
|
{
|
|
return NULL;
|
|
}
|
|
static gboolean
|
|
gst_my_filter_event (GstPad * pad, GstEvent * event)
|
|
{
|
|
return gst_pad_event_default (pad, event);
|
|
}
|
|
--><!-- example-end convertcaps.c a -->
|
|
<!-- example-begin convertcaps.c b -->
|
|
static gboolean
|
|
gst_my_filter_setcaps (GstPad *pad,
|
|
GstCaps *caps)
|
|
{
|
|
GstMyFilter *filter = GST_MY_FILTER (GST_OBJECT_PARENT (pad));
|
|
|
|
if (gst_pad_set_caps (filter->sinkpad, caps)) {
|
|
filter->passthrough = TRUE;
|
|
} else {
|
|
GstCaps *othercaps, *newcaps;
|
|
GstStructure *s = gst_caps_get_structure (caps, 0), *others;
|
|
|
|
/* no passthrough, setup internal conversion */
|
|
gst_structure_get_int (s, "channels", &filter->channels);
|
|
othercaps = gst_pad_get_allowed_caps (filter->srcpad);
|
|
others = gst_caps_get_structure (othercaps, 0);
|
|
gst_structure_set (others,
|
|
"channels", G_TYPE_INT, filter->channels, NULL);
|
|
|
|
/* now, the samplerate value can optionally have multiple values, so
|
|
* we "fixate" it, which means that one fixed value is chosen */
|
|
newcaps = gst_caps_copy_nth (othercaps, 0);
|
|
gst_caps_unref (othercaps);
|
|
gst_pad_fixate_caps (filter->srcpad, newcaps);
|
|
if (!gst_pad_set_caps (filter->srcpad, newcaps))
|
|
return FALSE;
|
|
|
|
/* we are now set up, configure internally */
|
|
filter->passthrough = FALSE;
|
|
gst_structure_get_int (s, "rate", &filter->from_samplerate);
|
|
others = gst_caps_get_structure (newcaps, 0);
|
|
gst_structure_get_int (others, "rate", &filter->to_samplerate);
|
|
}
|
|
|
|
return TRUE;
|
|
}
|
|
|
|
static GstFlowReturn
|
|
gst_my_filter_chain (GstPad *pad,
|
|
GstBuffer *buf)
|
|
{
|
|
GstMyFilter *filter = GST_MY_FILTER (GST_OBJECT_PARENT (pad));
|
|
GstBuffer *out;
|
|
|
|
/* push on if in passthrough mode */
|
|
if (filter->passthrough)
|
|
return gst_pad_push (filter->srcpad, buf);
|
|
|
|
/* convert, push */
|
|
out = gst_my_filter_convert (filter, buf);
|
|
gst_buffer_unref (buf);
|
|
|
|
return gst_pad_push (filter->srcpad, out);
|
|
}
|
|
<!-- example-end convertcaps.c b -->
|
|
<!-- example-begin convertcaps.c c --><!--
|
|
#include "state.func"
|
|
#include "register.func"
|
|
--><!-- example-end convertcaps.c c --></programlisting>
|
|
</sect2>
|
|
|
|
<sect2 id="section-nego-downstream-parse"
|
|
xreflabel="Parsing and setting caps">
|
|
<title>Parsing and setting caps</title>
|
|
<para>
|
|
Other elements, such as certain types of decoders, will not be able
|
|
to parse the caps from their input, simply because the input format
|
|
does not contain the information required to know the output format
|
|
yet; rather, the data headers need to be parsed, too. In many cases,
|
|
fixed-caps will be enough, but in some cases, particularly in cases
|
|
where such decoders are renegotiable, it is also possible to use
|
|
full caps negotiation.
|
|
</para>
|
|
<para>
|
|
Fortunately, the code required to do so is very similar to the last
|
|
code example in <xref linkend="section-nego-downstream-embed"/>, with
|
|
the difference being that the caps is selected in the <function>_chain
|
|
()</function>-function rather than in the <function>_setcaps
|
|
()</function>-function. The rest, as for getting all allowed caps from
|
|
the source pad, fixating and such, is all the same. Re-negotiation,
|
|
which will be handled in the next section, is very different for such
|
|
elements, though.
|
|
</para>
|
|
</sect2>
|
|
</sect1>
|
|
|
|
<sect1 id="section-nego-upstream" xreflabel="Upstream caps (re)negotiation">
|
|
<title>Upstream caps (re)negotiation</title>
|
|
<para>
|
|
Upstream negotiation's primary use is to renegotiate (part of) an
|
|
already-negotiated pipeline to a new format. Some practical examples
|
|
include to select a different video size because the size of the video
|
|
window changed, and the video output itself is not capable of rescaling,
|
|
or because the audio channel configuration changed.
|
|
</para>
|
|
<para>
|
|
Upstream caps renegotiation is done in the <function>gst_pad_alloc_buffer
|
|
()</function>-function. The idea here is that an element requesting a
|
|
buffer from downstream, has to specify the type of that buffer. If
|
|
renegotiation is to take place, this type will no longer apply, and the
|
|
downstream element will set a new caps on the provided buffer. The element
|
|
should then reconfigure itself to push buffers with the returned caps. The
|
|
source pad's setcaps will be called once the buffer is pushed.
|
|
</para>
|
|
<para>
|
|
It is important to note here that different elements actually have
|
|
different responsibilities here:
|
|
</para>
|
|
<itemizedlist>
|
|
<listitem>
|
|
<para>
|
|
Elements should implement a <quote>padalloc</quote>-function in
|
|
order to be able to change format on renegotiation. This is also
|
|
true for filters and converters.
|
|
</para>
|
|
</listitem>
|
|
<listitem>
|
|
<para>
|
|
Elements should allocate new buffers using
|
|
<function>gst_pad_alloc_buffer ()</function>.
|
|
</para>
|
|
</listitem>
|
|
<listitem>
|
|
<para>
|
|
Elements that are renegotiable should implement a
|
|
<quote>setcaps</quote>-function on their sourcepad as well.
|
|
</para>
|
|
</listitem>
|
|
</itemizedlist>
|
|
<para>
|
|
Unfortunately, not all details here have been worked out yet, so this
|
|
documentation is incomplete. FIXME.
|
|
</para>
|
|
</sect1>
|
|
|
|
<sect1 id="section-nego-getcaps" xreflabel="Implementing a getcaps function">
|
|
<title>Implementing a getcaps function</title>
|
|
<para>
|
|
A <function>_getcaps ()</function>-function is called when a peer
|
|
element would like to know which formats this element supports, and
|
|
in what order of preference. The return value should be all formats
|
|
that this elements supports, taking into account limitations of peer
|
|
elements further downstream or upstream, sorted by order of preference,
|
|
highest preference first.
|
|
</para>
|
|
<para>
|
|
</para>
|
|
<programlisting><!-- example-begin getcaps.c a --><!--
|
|
#include "init.func"
|
|
--><!-- example-end getcaps.c a -->
|
|
<!-- example-begin getcaps.c b -->
|
|
static GstCaps *
|
|
gst_my_filter_getcaps (GstPad *pad)
|
|
{
|
|
GstMyFilter *filter = GST_MY_FILTER (GST_OBJECT_PARENT (pad));
|
|
GstPad *otherpad = (pad == filter->srcpad) ? filter->sinkpad :
|
|
filter->srcpad;
|
|
GstCaps *othercaps = gst_pad_get_allowed_caps (otherpad), *caps;
|
|
gint i;
|
|
|
|
/* We support *any* samplerate, indifferent from the samplerate
|
|
* supported by the linked elements on both sides. */
|
|
for (i = 0; i < gst_caps_get_size (othercaps); i++) {
|
|
GstStructure *structure = gst_caps_get_structure (othercaps, i);
|
|
|
|
gst_structure_remove_field (structure, "rate");
|
|
}
|
|
caps = gst_caps_intersect (othercaps, gst_pad_get_pad_template_caps (pad));
|
|
gst_caps_unref (othercaps);
|
|
|
|
return caps;
|
|
}
|
|
<!-- example-end getcaps.c b -->
|
|
<!-- example-begin getcaps.c c --><!--
|
|
static gboolean
|
|
gst_my_filter_setcaps (GstPad * pad, GstCaps * caps)
|
|
{
|
|
return FALSE;
|
|
}
|
|
#include "chain.func"
|
|
#include "state.func"
|
|
#include "register.func"
|
|
--><!-- example-end getcaps.c c --></programlisting>
|
|
<para>
|
|
Using all the knowledge you've acquired by reading this chapter, you
|
|
should be able to write an element that does correct caps negotiation.
|
|
If in doubt, look at other elements of the same type in our git
|
|
repository to get an idea of how they do what you want to do.
|
|
</para>
|
|
</sect1>
|
|
</chapter>
|