Negotiation ----------- Capabilities negotiation is the process of deciding on an adequate format for dataflow within a GStreamer pipeline. Ideally, negotiation (also known as "capsnego") transfers information from those parts of the pipeline that have information to those parts of the pipeline that are flexible, constrained by those parts of the pipeline that are not flexible. GStreamer's two scheduling modes, push mode and pull mode, lend themselves to different mechanisms to acheive this goal. As it is more common we describe push mode negotiation first. Push-mode negotiation --------------------- Pussh-mode negotiation happens when elements want to push buffers and need to decide on the format. This is called downstream negotiation because the upstream element decides the format for the downstream element. This is the most common case. Negotiation can also happen when a downstream element wants to receive another data format from an upstream element. This is called upstream negotiation. The basics of negotiation are as follows: - GstCaps (see part-caps.txt) are refcounted before they are attached to a buffer to describe the contents of the buffer. It is possible to add a NULL caps to a buffer, this means that the buffer type did not change relative to the previous buffer. If no previous buffer was received by a downstream element, it is free to discard the buffer. - Before receiving a buffer, an element must check if the datatype of the buffer has changed. The element should reconfigure itself to the new format before processing the buffer data. If the data type on the buffer is not acceptable, the element should refuse the buffer by returning an appropriate return value from the chain function. The core will automatically call the set_caps function for this purpose when it is installed on the sink or source pad. - When requesting a buffer from a bufferpool, the prefered type should be passed to the buffer allocation function. After receiving a buffer from a bufferpool, the datatype should be checked again. - A bufferpool allocation function should try to allocate a buffer of the prefered type. If there is a good reason to choose another type, the alloc function should see if that other type is accepted by the other element, then allocate a buffer of that type and attach the type to the buffer before returning it. The general flow for a source pad starting the negotiation. src sink | | | accepts? | type A |---------------->| | yes | |<----------------| | | get buffer | alloc_buf | from pool |---------------->| with type A | | Create buffer of type A. | | check type |<----------------| and use A | | | push | push buffer |---------------->| Receive type A, reconfigure to with new type| | process type A. | | One possible implementation in pseudo code: [element wants to create a buffer] if not format # see what the peer can do peercaps = gst_pad_peer_get_caps (srcpad) # see what we can do ourcaps = gst_pad_get_caps (srcpad) # get common formats candidates = gst_caps_intersect (peercaps, ourcaps) foreach candidate in candidates # make sure the caps is fixed fixedcaps = gst_pad_fixate_caps (srcpad, candidate) # see if the peer accepts it if gst_pad_peer_accept_caps (srcpad, fixedcaps) # store the caps as the negotiated caps, this will # call the setcaps function on the pad gst_pad_set_caps (srcpad, fixedcaps) break endif done endif # if the type is different, the buffer will have different caps from # the src pad -- setcaps will get called on the pad_push buffer = gst_pad_alloc_buffer (srcpad, 0, size, GST_PAD_CAPS (fixedcaps)); if buffer [fill buffer and push] elseif [no buffer, either no peer or no acceptable format found] endif The general flow for a sink pad starting a renegotiation. src sink | | | accepts? | |<----------------| type B | yes | |---------------->| | | get buffer | alloc_buf | from pool |---------------->| with type A | | Create buffer of new type B. | | check type |<----------------| and | | reconfigure | | | push | push buffer |---------------->| Receive type B, reconfigure to with new type| | process type B. | | Use case: videotestsrc ! xvimagesink 1) Who decides what format to use? - src pad always decides, by convention. sinkpad can suggest a format by putting it high in the getcaps function GstCaps. - since the src decides, it can always choose something that it can do, so this step can only fail if the sinkpad stated it could accept something while later on it couldn't. 2) When does negotiation happen? - before srcpad does a push, it figures out a type as stated in 1), then it calls the pad alloc function with the type. The sinkpad has to create a buffer of that type, src fills the buffer and sends it to sink. - since the sink stated in 1) it could accept the type, it will be able to create a buffer of the type and handle it. - sink checks media type of buffer and configures itself for this type. 3) How can sink request another format? - sink asks if new format is possible for the source. - sink returns buffer with new type in allocfunction. - src receives buffer with new type, reconfigures and pushes. - sink can always select something it can create and handle since it takes the initiative. src should be able to handle the new type since it said it could accept it. videotestsrc ! queue ! xvimagesink - queue implements an allocfunction, proxying all calls to its srcpad peer. - queue proxies all accept and getcaps to the other peer pad. - queue contains buffers with different types. Pull-mode negotiation --------------------- A pipeline in pull mode has different negotiation needs than one activated in push mode. Push mode is optimized for two use cases: * Playback of media files, in which the demuxers and the decoders are the points from which format information should disseminate to the rest of the pipeline; and * Recording from live sources, in which users are accustomed to putting a capsfilter directly after the source element; thus the caps information flow proceeds from the user, through the potential caps of the source, to the sinks of the pipeline. In contrast, pull mode has other typical use cases: * Playback from a lossy source, such as RTP, in which more knowledge about the latency of the pipeline can increase quality; or * Audio synthesis, in which audio APIs are tuned to producing only the necessary number of samples, typically driven by a hardware interrupt to fill a DMA buffer or a Jack[0] port buffer. * Low-latency effects processing, whereby filters should be applied as data is transferred from a ring buffer to a sink instead of beforehand. For example, instead of using the internal alsasink ringbuffer thread in push-mode wavsrc ! volume ! alsasink, placing the volume inside the sound card writer thread via wavsrc ! audioringbuffer ! volume ! alsasink. [0] http://jackit.sf.net The problem with push mode is that the sink has to know the format in order to know how many bytes to pull via gst_pad_pull_range(). This means that before pulling, the sink must initiate negotation to decide on a format. Recalling the principles of capsnego, whereby information must flow from those that have it to those that do not, we see that the two named use cases have different negotiation requirements: * RTP and low-latency playback are both like the normal playback case, in which information flows downstream. * In audio synthesis, the part of the pipeline that has the most information is the sink, constrained by the capabilities of the graph that feeds it. However the caps are not completely specified; at some point the user has to intervene to choose the sample rate, at least. This can be done externally to gstreamer, as in the jack elements, or internally via a capsfilter, as is customary with live sources. Given that sinks potentially need the input of sources, as in the RTP case and at least as a filter in the synthesis case, there must be a negotiation phase before the pull thread is activated. [ok at this point i'm a bit at a loss about how it should go] This negotiation phase is initiated by the sink, after it succeeds in calling gst_pad_activate_pull(), but before it spawns a thread to start pulling. The sink will call gst_pad_get_allowed_caps() on the its sink pad and fixate if necessary to determine the flow caps. It then calls gst_pad_set_caps on its sink pad's peer, to configure the upstream elements. A typical element will implement a setcaps() function on its src pads that proxies the setcaps() to all peers of its sink pads. This way, when getrange() is called on a pad, it knows what format it is being asked to produce. If the sink element could not set caps on its peer(s), it should post an error message on the bus indicating that negotiation was not possible. In this way all src pads are negotiated before data starts flowing, so all getrange() requests have a defined meaning for the number of bytes being pulled. (Now we get to questions: set caps on the sink pads at the same time or no? Does it matter? Does gst_pad_get_range() set caps? Does pull_range()? What happens when caps changes? Can caps change?)