mirror of
https://gitlab.freedesktop.org/gstreamer/gstreamer.git
synced 2024-12-04 23:46:43 +00:00
4a402c1c7d
Found via `codespell` https://bugzilla.gnome.org/show_bug.cgi?id=795610
286 lines
12 KiB
Text
286 lines
12 KiB
Text
|
|
Some notes on pad negotiation
|
|
|
|
|
|
A "pad link" is a connection between two pads. It can be in one of
|
|
two states, "negotiated" or "not negotiated". Pad links are created
|
|
by gst_pad_link().
|
|
|
|
A "pad link" is created when two pads are linked using gst_pad_link().
|
|
When initially created, the link only specifies a src pad, a sink pad,
|
|
and optionally a filter caps provided by the application.
|
|
|
|
In order to pass data through a link, the peer pads must decide on
|
|
what data format to use. This is called negotiation. Pads
|
|
describe acceptable data formats by using a combination of pad
|
|
template caps and (optionally) a pad->getcaps function.
|
|
|
|
|
|
Negotiation can happen in one of two forms, directed or undirected.
|
|
Directed negotiation happens when one element has decided (usually
|
|
during negotiation on another pad) to ask for a specific format on
|
|
a pad. This happens when a pad calls gst_pad_try_set_caps().
|
|
Undirected negotiation happens when the core decides to negotiate
|
|
a link, either due to a state change or a specific application
|
|
request.
|
|
|
|
Steps in undirected negotiation (core view):
|
|
|
|
- core checks that both pad's parent elements are in the READY or
|
|
higher state.
|
|
|
|
- core calls gst_pad_get_caps() on each pad, intersects the two caps,
|
|
and intersects again with the filter caps. If the intersection is
|
|
empty, then the pads have no formats in common, and the link fails.
|
|
|
|
- If the intersection caps is not fixed, there are multiple possible
|
|
formats that the link could use. If this is the case, fixate
|
|
functions are called until the caps are fixed. The fixate functions
|
|
are called by priority -- src application fixate function, sink
|
|
application fixate function, src and sink fixate functions, and
|
|
the default core fixate function. The application fixate functions
|
|
are implemented by the "fixate" signal on each pad. The core
|
|
loops through the fixate functions until a fixed caps is decided
|
|
on.
|
|
|
|
- Each pad may have a pad_link function, which is called with the
|
|
fixed caps. The pad_link function has the option of accepting,
|
|
rejecting, or delaying the negotiation.
|
|
|
|
- If both pads accept the caps, the link is then negotiated.
|
|
|
|
Steps in directed negotiation (gst_pad_try_set_caps):
|
|
|
|
- the originator is the pad that gst_pad_try_set_caps() is called
|
|
on.
|
|
|
|
- the elements owning both pads are assumed to be in a non-NULL state
|
|
|
|
- the caps argument of try_set_caps() must be fixed.
|
|
|
|
- gst_pad_get_caps() is called on the peer pad, and intersected with
|
|
the originating pad's pad template caps and the filter caps. The
|
|
caps argument is checked to be a subset of the intersection. (It's
|
|
important that this intersection uses the pad _template_ caps.)
|
|
|
|
- Fixate functions don't need to be called, since the caps are
|
|
already fixed.
|
|
|
|
- The peer's pad_link function is called.
|
|
|
|
- If the peer's pad_link function accepts the caps, the link is then
|
|
negotiated.
|
|
|
|
- If the peer's pad_link function refuses the caps, and the link had
|
|
already been negotiated, the peer's pad_link function is called
|
|
with the caps of the old negotiation.
|
|
|
|
- Note: the originator's pad_link function is _not_ called. The
|
|
originator must take appropriate alternative steps.
|
|
|
|
|
|
Notes about renegotiation:
|
|
|
|
- same as negotiation. Note that get_caps() functions should always
|
|
ignore the currently negotiated caps of a link.
|
|
|
|
- if renegotiation fails, the previous negotiation is still in effect.
|
|
If the renegotiation fails in the last pad_link step, the pad_link
|
|
functions are called with the previously negotiated caps.
|
|
|
|
|
|
Notes for sources and sinks:
|
|
|
|
- sources and sinks that talk to hardware may not be able to fully
|
|
describe their available formats, and thus need to rely on pad_link
|
|
functions to test a particular format. FIXME: currently, the core
|
|
completely fails negotiation if a pad_link function refuses a caps,
|
|
instead of attempting with an alternate caps.
|
|
|
|
Example: Assume osssink advertises rate=(int)[8000,48000], but
|
|
the device cannot actually handle rate=44100 (unknown to osssink).
|
|
Assume that the pad_link function is called with rate=44100 --
|
|
ideally, the pad_link function should return GST_PAD_LINK_DELAYED,
|
|
and future calls to getcaps should return {[8000,44099],[44101,
|
|
48000]}. I don't know how to make this easy and/or work well.
|
|
|
|
|
|
Notes for decoders/demuxers:
|
|
|
|
- Decoders will typically negotiate a sink pad, receive some data,
|
|
determine the output format, and call try_set_caps() with the given
|
|
format. If the output format is non-fixed, gst_pad_renegotiate()
|
|
may be used instead, in order to have the fixate functions choose
|
|
the optimal format. Note that this requires communication so that
|
|
the pad's getcaps function returns the correct caps.
|
|
|
|
Notes for converters:
|
|
|
|
- Converters change one or more properties of the format of a data
|
|
stream. A typical converter's getcaps function will call
|
|
gst_pad_get_allowed_caps() for the opposite pad in the element,
|
|
change one or more fields in the caps, and return the result.
|
|
|
|
- getcaps function:
|
|
|
|
- call gst_pad_get_allowed_caps() on the other pad in the element
|
|
|
|
- for each possible format ("A") in the allowed caps, determine all
|
|
the formats ("B") that your converter could convert the original
|
|
format (A) to. The union of all these formats (all the B's) is
|
|
the caps that should be returned. (This is how to do it
|
|
_theoretically_, but an optimal implementation will probably be
|
|
quite different.)
|
|
For example, a simple way to do this for an element that can convert
|
|
a given field of the caps is to remove the field(s) from the structure,
|
|
then intersect with the pad template.
|
|
|
|
- As an example, videoscale can convert any sized video to any other
|
|
sized video. Its getcaps function iterates over each structure in
|
|
the caps, and replaces the width and height with the range
|
|
[1,MAXINT].
|
|
|
|
- pad_link function:
|
|
|
|
- the "otherpad" is the opposite pad in the element.
|
|
|
|
- extract fields from the caps that are relevant for your converter
|
|
handling the format. Store these in _local_ variables. (E.g,
|
|
things like video size, sample rate, etc.)
|
|
|
|
- If it's possible to pass buffers through without modifying them
|
|
(passthrough), you should call gst_try_set_caps() with the caps
|
|
that was specified as the parameter to the pad_link function. If
|
|
this is successful, save the local variables to the element
|
|
structure, perform whatever other setup is necessary for your
|
|
element, and return GST_PAD_LINK_OK.
|
|
|
|
- Otherwise, you're not using passthrough, and may need to
|
|
change the caps on the otherpad to match the given format.
|
|
|
|
- If the otherpad is not negotiated (!gst_pad_is_negotiated()),
|
|
you shouldn't attempt to set a format on it. It will eventually
|
|
be negotiated. Save the local variables to the element structure,
|
|
perform whatever other setup is necessary, and return
|
|
GST_PAD_LINK_OK.
|
|
|
|
- At this point, the other pad is already negotiated, but won't
|
|
accept the passthrough format, so you should combine the existing
|
|
negotiated caps on the otherpad and the caps that was the pad link
|
|
argument. This can either be done using existing information in the
|
|
element that was saved during a previous pad_link call, or you can
|
|
get the information from the negotiated caps
|
|
(gst_pad_get_negotiated_caps()).
|
|
|
|
As an example, consider the videoscale element. Assume that
|
|
videoscale.src has already negotiated "video/x-raw-yuv,
|
|
format=(fourcc)I420, width=320, height=240", and that the sink
|
|
pad's link function is called with "video/x-raw-yuv,
|
|
format=(fourcc)YUY2, width=640, height=480". Since it's the
|
|
videoscale element, we can have different width and height
|
|
fields on the pads, but the format must be the same. So we'll
|
|
use the existing negotiated size (640x480), and the new format,
|
|
and call gst_pad_try_set_caps() with "video/x-raw-yuv,
|
|
format=(fourcc)I420, width=640, height=480".
|
|
|
|
This may seem overkill, but most of the time, you'll end up
|
|
calling try_set_caps() with the same caps that are currently
|
|
negotiated -- try_set_caps() just returns GST_PAD_LINK_OK in
|
|
this case.
|
|
|
|
- If gst_pad_try_set_caps() returns GST_PAD_LINK_OK, save the
|
|
local variables to the element structure. In any case, return
|
|
the return value of gst_pad_try_set_caps().
|
|
|
|
|
|
Notes for filters:
|
|
|
|
- Filters can almost always use gst_pad_proxy_getcaps() as the
|
|
getcaps function. This just returns gst_pad_get_allowed_caps()
|
|
on the otherpad.
|
|
|
|
- You may be able to use gst_pad_proxy_pad_link() as the pad link
|
|
function, but only if you don't need to extract parameters from
|
|
the caps.
|
|
|
|
|
|
Notes for encoders/muxers:
|
|
|
|
- Encoders and muxers should roughly work like converters. Many
|
|
converters are symmetric; encoders and muxers obvious are not,
|
|
thus it may make the code clearer to have separate src and sink
|
|
getcaps and pad_link functions.
|
|
|
|
- Encoders and muxers should handle multiple negotiations until
|
|
the first buffer has been passed. After this point, it's unlikely
|
|
that additional negotiations will happen in well-constructed
|
|
pipelines, but it may be wise to "lock" the caps after the
|
|
muxer has committed to a format. (FIXME: it's still unclear to
|
|
me when the caps should get "unlocked". Obviously at EOS or
|
|
PAUSED->READY transitions. Any others?)
|
|
|
|
- Locking caps can be done by adding (near the top) of the getcaps
|
|
function:
|
|
|
|
if (my_element->lock_caps) {
|
|
return gst_pad_get_negotiated_caps (pad);
|
|
}
|
|
|
|
|
|
Explicit caps:
|
|
|
|
- There's a hack in the core to make the code for decoder elements
|
|
a lot simpler. This hack can be used only for src pads of elements
|
|
that get their srcpad capabilities directly from the data stream,
|
|
i.e., decoders, demuxers, and typefind. This hack overrides the
|
|
pad's getcaps() and pad_link() function, so that they work correctly
|
|
in all decoder states.
|
|
|
|
- To enable this hack on a pad, call gst_pad_use_explicit_caps().
|
|
|
|
- To indicate that a decoder has found the format of the stream, call
|
|
gst_pad_set_explicit_caps(pad,caps) with the caps of the stream.
|
|
This caps must be fixed.
|
|
|
|
- To indicate that a decoder has lost the format of the stream, i.e.,
|
|
there's been a NEW_MEDIA event, call gst_pad_set_explicit_caps(pad,
|
|
NULL).
|
|
|
|
- If the explicit caps are set, the getcaps function will return that
|
|
caps, and the pad_link function will return GST_PAD_LINK_OK. If
|
|
the explicit caps are not set, the getcaps function returns the pad
|
|
template caps, and the pad_link function returns GST_PAD_LINK_DELAYED.
|
|
|
|
|
|
|
|
Other junk:
|
|
|
|
- negotiation can happen at any time
|
|
|
|
- negotiation can happen multiple times/often happens multiple times
|
|
|
|
- initial negotiation can lead to strange caps
|
|
|
|
- directed negotiation can happen in either direction (src to sink or
|
|
sink to src)
|
|
|
|
- Other considerations ignored, every pad should have a getcaps function.
|
|
|
|
- If a pad's getcaps function returns the same caps in every
|
|
circumstance, the getcaps function can be omitted.
|
|
|
|
- If you use gst_pad_use_explicit_caps(), the getcaps function must
|
|
be omitted.
|
|
|
|
- fixate functions are a method for applications to exert influence
|
|
on how a format is chosen from a caps. It's also used as a hack to
|
|
allow elements to do the same. Element fixate functions are _not_
|
|
intended to give good results for applications -- they're intended
|
|
to give non-disgusting results in gst-launch. Don't attempt to
|
|
make them do more than they're capable of.
|
|
|
|
- Fixate functions should not be implemented on anything except source
|
|
and sink elements.
|
|
|
|
|
|
|