Caps Negotiation ================ Definitions ----------- - GstCaps is a structure that holds a mimetype and a set of properties. The properties are either fixed or non-fixed, they contain ranges or lists. - filter caps: caps set by the application to constrain a link to a specific GstCaps. - allowed caps: caps calculated when connecting elements, these are the types that are possible on this connection. If no types are possible, the link is refused. - pad template caps: caps put on padtemplates to describe the possible media types of the pad. - pad caps: the caps of the pad as it is currently negotiated. General overview ---------------- Caps negotiation is the process of two pads agreeing on a common fixed GstCaps structure that describes the media type that will be exchanged between those pads. We define that caps negotiation only happens in the PLAYING state, when there is actual dataflow and elements thus know what kind of data they are dealing with, or when linking elements. Caps negotiation is first performed when linking elements. If the elements can agree on a media type at link time then negotiation will not have to be performed at runtime. Usually this is not possible, so the pads remain unnegotiated until at runtime. Caps negotiation is initiated by an element that wants to send data on a non-negotiated pad, thus the source pad. The core only provides a policy and convenience methods to aid the element in performing the negotiation. The core does not keep track of what pads are negotiated or not, this is a task of the pads. Caps negotiation is normally started by the source element right before it sends out data (case 1). Caps negotiation can also be redone by a sink element that wants to receive another format from a downstream element (case 2). There are two functions handling the negotiation, the link function and the do_link function. The link function is always called first and must eventually call do_link on the peer pad. The link function must figure out a compatible format for the connection and then call the do_link function on the peer pad. The do_link function can only accept or refuse the provided caps. For autopluggers it is important to know when the pad is ready to start the negotiation. It is also important to know when the negotiation failed and it must be possible to restart the negotiation with another element. This functionality will be provided with signals. Pad functions ------------- ! ! const GstCaps* gst_pad_iterate_caps (GstPad *pad, gint position); ! Returns the nth possible caps that describes the media type that can flow over this pad. This function should not return static caps but caps that are dependent of the media type and the peer connections of the element. This function can be called at any time so that an autoplugger can know the exact types of the pads at any time. ! ! gboolean gst_pad_accept_caps (GstPad *pad, GstCaps *caps); ! Checks that a given caps is acceptable for this pad. Returns FALSE if the pad cannot handle the caps. ! ! gboolean gst_pad_link (GstPad *pad, GstPad *peer, GstCaps *caps); ! Tells the pad to start negotiation with the given filtercaps. The caps need not be fixed and serves as a filter for performing the negotiation with the peerpad. ! ! gboolean gst_pad_do_link (GstPad *pad, GstPad *peer, GstCaps *caps); ! Configures the pad to accept data with the given caps. The caps must be fixed. ! ! const GstCaps* gst_pad_get_negotiated_caps (GstPad *pad); ! Get the negotiated caps of a pad or NULL if the pad is not negotiated. Linking Elements ---------------- When linking elements with the gst_pad_link function, the core will call the link function on the srcpad. If that pad returns GST_PAD_LINK_OK, the link is established, else the link fails. Since the link function of the pad will call the do_link function of the peerpad, the caps will be set on both pads. It is not required to decide on a caps at link time, a plugin can choose to dynamically renegotiate at runtime. When a link fails, the core emits a signal link_failed, which the application can catch to try to establish a new link with another element, for example. ! ! def gst_pad_link_filtered (pad1, pad2, filtercaps): ! ! ... get srcpad and sinkpad ... ! srcpad = (pad1 == GST_PAD_SRC ? pad1 : pad2); ! sinkpad = (pad1 == GST_PAD_SINK ? pad1 : pad2); ! ! ... more checks to see if the pads are of different types and ! ... that they live in the same scheduler etc... ! ! res = gst_pad_link (srcpad, sinkpad, filtercaps) ! if (res == GST_PAD_LINK_OK) ! ... perform other setup ... ! else ! res = signal_emit (srcpad, "link_failed") ! ! return res ! Pad link is just a convenience function that passes a NULL filtercaps: ! ! def gst_pad_link (pad1, pad2): ! gst_pad_link_filtered (pad1, pad2, NULL) ! Dynamic renegotiation --------------------- Dynamic renegotiation happens at runtime when the element knows the exact media type it is handling. The element that wants to redecide on the data type of a connection just calls gst_pad_relink on one of it's pads. The core will call unlink and link on the pads again as if this were a new connection. Since the iterate_caps function returns other values while linking, the new link will renegotiate to a new format or the link will fail. Prototype of the relink function: ! ! def gst_pad_relink_filtered (pad, filtercaps): ! gst_pad_unlink (pad, peerpad) ! gst_pad_link_filtered (pad, peerpad, filtercaps) ! Again the relink function is a convenience function for not having to pass filtercaps. ! ! def gst_pad_relink (pad): ! gst_pad_relink_filtered (pad, NULL) ! The core, however, should optimize the relink function so that it does a minimal amount of work, like not informing the scheduler about the unlink/link call in case of success. Error recovery -------------- When a pad is ready to negotiate and it has no peer pad, it fires the "need-link" signal. Autopluggers can use this signal to start plugging elements to the pad. When a link fails because of a renegotiation, the "link-failed" signal is fired so that autopluggers can try other elements. Default implementations ----------------------- ! ! gst_pad_src_link_func (pad, peerpad, filtercaps) ! { ! srcpad->negotiated_caps = NULL; ! gboolean possible = FALSE; ! ! for (i=0; peercaps = gst_pad_iterate_caps (peerpad, i); i++) ! { ! for (j=0; srccaps = gst_pad_iterate_caps (srcpad, j); j++) ! { ! test = gst_caps_intersect (srccaps, peercaps, filtercaps); ! if (test == GST_CAPS_EMPTY) ! continue; ! ! /* non empty caps, something is possible */ ! possible = TRUE; ! ! if (!gst_caps_is_fixed (caps)) ! continue; ! ! if (gst_pad_accepts_caps (peerpad, test)) ! { ! if (gst_pad_do_link (peerpad, srcpad, test)) ! { ! srcpad->negotiated_caps = test; ! goto done; ! } ! else ! { ! /* weird, it accepted but did not want to link */ ! } ! } ! else ! { ! /* caps are not accepted by peer, try next */ ! } ! } ! } ! done: ! if (!possible) ! return GST_PAD_LINK_FAILED; ! else ! return GST_PAD_LINK_OK; ! } ! gst_pad_iterate_caps returns caps in the following order: 1) preferred caps (if any) 2) negotiated caps (if any) 3) profile caps (if any, filtered against caps of current media type) 4) padtemplate caps (filtered against caps of current media type) 1 = a caps describing the media type that would result in optimal processing of the current media type 2 = a caps that it is currently handling 3 = a caps describing the user configured default values 4 = a caps describing all other possibilities that this pad can produce or consume given the current media type if any. generic flow 1: Linkage src sink | | | <--- link(sink, A) | check if | | A is ok | | or filter | | against | | allowed | | src caps | | | iterate_caps | if A not +-------------------------> | fixed | | | <-------------------------+ filter | | against A | | to get A' | | | can_accept(A') | if A' fix +-------------------------> | | | check if A' is ok | yes | | <-------------------------+ | | if A' not | | fixed, | | A'=null | | | do_link(src, A') | +-------------------------> | | ok | store src, A' | <-------------------------+ store | | sink, A' | | Unlink src sink | | | <--- unlink() | | | | unlink() | +-------------------------> | unref | | unref src, format sink, fmt | | Dynamic negotiation of B src sink | can_accept(B) | +-------------------------> | | | check if B is ok | yes | | <-------------------------+ | | | do_link(B) | +-------------------------> | | ok | store B | <-------------------------+ store | | B | | TODO ----- Write test objects to simulate different behaviour. tests (in python): vts -> colorspace -> ximagsink relink with different colorspaces while running vts -> videoscale -> ximagesink relink to different sizes while running, scaling vts or ximagesink. sinesrc -> audiorate -> audiosink where audiosink only has one possible fixed caps.