OUTDATED -------- caps negotiation ================ 1) purpose ---------- The pads expose the media types they can handle using a mime type and a set of properties. Before the pad is created or used to pass buffers, we only know the global 'range' of media data this pad can accept. When the element has had a chance to look at the media data, only then it knows the exact values of the properties. example1: ! ! The mp3 decoder exposes the capabilities of its src pad ! with the following caps: ! ! 'mpg123_src': ! MIME type: 'audio/raw': ! format: Integer: 16 ! depth: Integer: 16 ! rate: Integer range: 11025 - 48000 ! channels: Integer range: 1 - 2 as you can see in example1, the padtemplate has both a range (for the audio rate) and a list (for the number of channels) for its properties. only when the mpg123 element has decoded the first mpeg audio header, it knows the exact values of the rate and channels properties. suppose that we want to connect this src pad to the sink pad of an audiosink with the properties given in example2: example2: ! ! 'audiosink_sink': ! MIME type: 'audio/raw': ! format: Integer: 16 ! depth: List: ! Integer: 8 ! Integer: 16 ! rate: Integer range: 8000 - 44000 ! channels: Integer range: 1 - 2 we can see that connecting the mpg123 src pad with the audiosinks sink pad can cause a potential problem with the rate property. When the mpg123 decoder decides to output raw audio with a 48000Hz samplerate, the audiosink will not be able to handle it. The conservative approach would be to disallow the connection between the two incompatible pads. This rules out any potential problems but severely limits the amount of possible connections between the elements. Another approach would be to allow the connection (and mark it as dangerous) and let the two elements figure out a suitable media type at runtime. This procedure is called caps negotiation. 2) a bit of history ------------------- The typing of the data that was attached to a buffer used to be done using GstMeta* (and it still is as of 11 feb 2001). With the new GstCaps and GstProps system this typing is gradually moved to the pads and to the padtemplates. This has several advantages: - the typing of the data tends to be static. The type of media doesn't change for every buffer. - Moving the typing up to the pad(templates) allows us to save them into the registry and allows us to figure out what pads are compatible. - the current metadata implementation needs header files. this may change when we also use properties for metadata. example3: ! ! This is the current GstMeta structure that travels with audio buffers ! ! struct _MetaAudioRaw { ! GstMeta meta; ! ! /* formatting information */ ! gint format; ! gint channels; ! gint frequency; ! gint bps; ! }; The question still remains what purpose the metadata will serve now that we expose the media type in the pads. Some possibilities: - interesting information, not describing the data itself but the context in which the data was generated (suggested buffer size, timestamps, etc...) - user app metadata. In this proposal we also assume that the current use of metadata using GstMeta is deprecated and that we move this information to the properties of the pads. 3) the pad/padtemplates caps ---------------------------- All elements have to provide a padtemplate for their pads. The padtemplates provide a range of possible media types this pad can src/sink. the main purpose for the padtemplates is to allow a rough guess at which pads are compatible before even a single buffer has been processed by the element. pads are usually created from the templates. When the pad is created it has no GstCaps* attached to it yet. The possible caps this pad can have is exposed in the padtemplate. The caps are filled in by the element when it knows the values for the caps. 4) the connect function ----------------------- when two pads are connected the following steps will take place (not sure, FIXME): - if both pads have caps, the caps are checked. If the caps are incompatible, the padtemplates are checked, if they are compatible, caps negotiation is performed. - if one of the pads has caps, the caps is checked against the padtemplate of the peer pad. If they are incompatible, the padtemplates are compared, if they are incompatible, caps negotiation is performed. - if none of the pads have caps, the padtemplates are checked, if they are incompatible, a warning is issued. 5) when the element knows the media type it is handling ------------------------------------------------------- When the element has received its first buffer it will know the media type it is handling by inspecting the buffer. before pushing the data out to its peer element(s), the element will set its src pad with the appropriate caps and properties. These caps must follow the following rules: - the caps must be compatible with the padtemplates of this pad. - the caps cannot contain ranges or lists. when the element wants to change the caps of a pad, it has to perform gst_pad_renegotiate (GstPad *pad). this will trigger the caps negotiation procedure. this will trigger the class method of the pad and calls the pads gst_pad_negotiate function: GstCaps *gst_pad_negotiate (GstPad *pad, GstCaps *caps, guint count); This function takes a GstCaps *structure as an argument (typically the current caps of the pad) and a negotiation counter. this counter can be used to keep track of the negotiation process. The pad then creates a new caps structure with the desired caps. If the caps are accepted, it just returns the provided input caps. the _renegotiate function will set the caps of both pads whenever the input caps are the same (pointer wise) as the input caps. the caps structure is checked against the padtemplate of the peer pad, if it is incompatible the gst_pad_negotiate function is called again and the element is supposed to create another caps structure. the gst_pad_renegotiate function then calls the gst_pad_negotiate function of the peer pad with the new caps as the argument. The peer pad can adjust or create a new caps if it doesn't accept it. the caps structure keeps on bouncing between the two pads until one of the pads negotiation functions returns the caps unmodified. The element can also return a NULL pointer if it has run out of options for the caps structure. When this happens, both pads are set the NULL caps again and the pad connection is broken. The negotiation process is stopped after a fixed number of tries, when the counter has reached some limit. This limit is typically checked by the pads negotiate function. 6) caps negotiation function ---------------------------- the negotiate function of a pad is called whenever the pad or peer pad has performed _renegotiate. example5: ! ! this is the caps negotiation function implemented by an element on ! one of its sink pads. ! ! static GstCaps* ! gst_pad_negotiate (GstPad *pad, GstCaps *caps, guint counter) ! { ! /* we don't accept anything else than audio/raw */ ! if (strcmp (gst_caps_get_mime (caps), "audio/raw")) ! return NULL; ! ! if (gst_caps_get_int_prop (caps, "format") != AFMT_S16_LE) ! return NULL; ! ! /* we accept everything else */ ! return caps; ! } When the negotiate function returns NULL (it does not accept the specified caps of the peer pad), the negotiation process is stopped. APPENDIX A: use cases ===================== 1) mpg123 src!sink audiosink ---------------------------- When the pads are connected the padtemplates are checked and it turns out that the pads might be incompatible (mpg123 can do 48000Hz while audiosink can do 44000Hz). Nothing happens at connect time except for the user app that can mark this connection as possibly dangerous and keep some spare elements ready for when the pads turn out to be incompatible. both elements start out with no caps at all (NULL). mpg123 wants to output a buffer with specific properties. It calls gst_pad_renegotiate (mpg123->srcpad). The _renegotiate functions calls the negotiate function of the mpg123->srcpad. the negotiate function would look like this: /* * The mpg123 element cannot convert the decoded type into something * else so it has to force the caps of the src pad into the specific * type as defined by the mp3. */ static GstCaps* gst_mpeg123_src_negotiate (GstPad *pad, GstCaps *caps, guint counter) { GstMpg123 *mpg123; mpg123 = GST_MPG123 (gst_pad_get_parent (pad)); /* we got caps in, check them */ if (caps != NULL) { if (!strcmp (gst_caps_get_mime (caps), "audio/raw") && (gst_caps_get_int_prop (caps, "format") == AFMT_S16_LE) && (gst_caps_get_int_prop (caps, "depth") == 16) && (gst_caps_get_int_prop (caps, "rate") == mpg123->rate) && (gst_caps_get_int_prop (caps, "channels") == mpg123->channels)) { return caps; } } /* we didn't get caps, so we decide */ else if (counter != 2) { GstCaps *new; /* fill in our desired caps */ new = gst_caps_new_with_props ( "src_caps", /* name */ "audio/raw", /* mime */ gst_props_new ( "format", GST_PROPS_INT (AFMT_S16_LE), "depth", GST_PROPS_INT (16), "rate", GST_PROPS_INT (mpg123->rate), "channels", GST_PROPS_INT (mpg123->channels), NULL ) ); return caps; } /* too many attempts at nogotiation, bail out */ return NULL; } The audiosink pad negotiate function would look like this: /* * The audiosink has a wide range of possible parameters for * its sink pad, based on the audio card capabilities and * possibly the element configuration. * we assume the audiosink element can be both the initiator of * the negotiations and the negotiated one. */ static GstCaps* gst_audiosink_sink_negotiate (GstPad *pad, GstCaps *caps, guint counter) { GstAudiosink *audiosink; gboolean accepted = TRUE; audiosink = GST_AUDIOSINK (gst_pad_get_parent (pad)); /* we got caps in, we know they will match the padtemplate */ if (caps != NULL) { return caps; } /* we didn't get caps, so we decide */ else if (counter != 2) { GstCaps *new; /* fill in our desired caps */ new = gst_caps_new_with_props ( "sink_caps", /* name */ "audio/raw", /* mime */ gst_props_new ( "format", GST_PROPS_INT (audiosink->format), "depth", GST_PROPS_INT (audiosink->depth), "rate", GST_PROPS_INT (audiosink->rate), "channels", GST_PROPS_INT (audiosink->channels), NULL ) ); return caps; } /* too many attempts at nogotiation, bail out */ return NULL; }