mirror of
https://gitlab.freedesktop.org/gstreamer/gstreamer.git
synced 2024-12-25 09:40:37 +00:00
258 lines
10 KiB
Text
258 lines
10 KiB
Text
Negotiation
|
|
-----------
|
|
|
|
Capabilities negotiation is the process of deciding on an adequate
|
|
format for dataflow within a GStreamer pipeline. Ideally, negotiation
|
|
(also known as "capsnego") transfers information from those parts of the
|
|
pipeline that have information to those parts of the pipeline that are
|
|
flexible, constrained by those parts of the pipeline that are not
|
|
flexible.
|
|
|
|
GStreamer's two scheduling modes, push mode and pull mode, lend
|
|
themselves to different mechanisms to achieve this goal. As it is more
|
|
common we describe push mode negotiation first.
|
|
|
|
Push-mode negotiation
|
|
~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
Push-mode negotiation happens when elements want to push buffers and
|
|
need to decide on the format. This is called downstream negotiation
|
|
because the upstream element decides the format for the downstream
|
|
element. This is the most common case.
|
|
|
|
Negotiation can also happen when a downstream element wants to receive
|
|
another data format from an upstream element. This is called upstream
|
|
negotiation.
|
|
|
|
The basics of negotiation are as follows:
|
|
|
|
- GstCaps (see part-caps.txt) are refcounted before they are pushed as
|
|
an event to describe the contents of the following buffer.
|
|
|
|
- An element should reconfigure itself to the new format received as a CAPS
|
|
event before processing the following buffers. If the data type in the
|
|
caps event is not acceptable, the element should refuse the buffer by
|
|
returning an appropriate GST_FLOW_NOT_NEGOTIATED return value from the
|
|
chain function.
|
|
|
|
- Downstream elements can request a format change of the stream by sending a
|
|
RECONFIGURE event upstream. Upstream elements will renegotiate a new format
|
|
when they receive a RECONFIGURE event.
|
|
|
|
The general flow for a source pad starting the negotiation.
|
|
|
|
src sink
|
|
| |
|
|
| accepts? |
|
|
type A |---------------->|
|
|
| yes |
|
|
|< - - - - - - - -|
|
|
| |
|
|
| send_event() |
|
|
send CAPS |---------------->| Receive type A, reconfigure to
|
|
event A | | process type A.
|
|
| |
|
|
| push |
|
|
push buffer |---------------->| Process buffer of type A
|
|
| |
|
|
|
|
One possible implementation in pseudo code:
|
|
|
|
[element wants to create a buffer]
|
|
if not format
|
|
# see what the peer can do
|
|
peercaps = gst_pad_peer_get_caps (srcpad)
|
|
# see what we can do
|
|
ourcaps = gst_pad_get_caps (srcpad)
|
|
|
|
# get common formats
|
|
candidates = gst_caps_intersect (peercaps, ourcaps)
|
|
|
|
foreach candidate in candidates
|
|
# make sure the caps is fixed
|
|
fixedcaps = gst_pad_fixate_caps (srcpad, candidate)
|
|
|
|
# see if the peer accepts it
|
|
if gst_pad_peer_accept_caps (srcpad, fixedcaps)
|
|
# store the caps as the negotiated caps, this will
|
|
# call the setcaps function on the pad
|
|
gst_pad_push_event (srcpad, gst_event_new_caps (fixedcaps))
|
|
break
|
|
endif
|
|
done
|
|
endif
|
|
|
|
buffer = gst_buffer_new_and_alloc (size);
|
|
[fill buffer and push]
|
|
|
|
|
|
The general flow for a sink pad starting a renegotiation.
|
|
|
|
src sink
|
|
| |
|
|
| accepts? |
|
|
|<----------------| type B
|
|
| yes |
|
|
|- - - - - - - - >|-.
|
|
| | | suggest B caps next
|
|
| |<'
|
|
| |
|
|
| push_event() |
|
|
mark .-|<----------------| send RECONFIGURE event
|
|
renegotiate| | |
|
|
'>| |
|
|
| get_caps() |
|
|
renegotiate |---------------->|
|
|
| suggest B |
|
|
|< - - - - - - - -|
|
|
| |
|
|
| send_event() |
|
|
send CAPS |---------------->| Receive type B, reconfigure to
|
|
event B | | process type B.
|
|
| |
|
|
| push |
|
|
push buffer |---------------->| Process buffer of type B
|
|
| |
|
|
|
|
|
|
Use case:
|
|
|
|
|
|
videotestsrc ! xvimagesink
|
|
|
|
1) Who decides what format to use?
|
|
- src pad always decides, by convention. sinkpad can suggest a format
|
|
by putting it high in the getcaps function GstCaps.
|
|
- since the src decides, it can always choose something that it can do,
|
|
so this step can only fail if the sinkpad stated it could accept
|
|
something while later on it couldn't.
|
|
|
|
2) When does negotiation happen?
|
|
- before srcpad does a push, it figures out a type as stated in 1), then
|
|
it pushes a caps event with the type. The sink checks the media type and
|
|
configures itself for this type.
|
|
- the source then usually does an ALLOCATION query to negotiate a bufferpool
|
|
with the sink. It then allocates a buffer from the pool and pushes it to
|
|
the sink. since the sink accepted the caps, it can create a pool for the
|
|
format.
|
|
- since the sink stated in 1) it could accept the type, it will be able to
|
|
handle it.
|
|
|
|
3) How can sink request another format?
|
|
- sink asks if new format is possible for the source.
|
|
- sink pushes RECONFIGURE event upstream
|
|
- src receives the RECONFIGURE event and marks renegotiation
|
|
- On the next buffer push, the source renegotiates the caps and the
|
|
bufferpool. The sink will put the new new prefered format high in the list
|
|
of caps it returns from its getcaps function.
|
|
|
|
videotestsrc ! queue ! xvimagesink
|
|
|
|
- queue proxies all accept and getcaps to the other peer pad.
|
|
- queue proxies the bufferpool
|
|
- queue proxies the RECONFIGURE event
|
|
- queue stores CAPS event in the queue. This means that the queue can contain
|
|
buffers with different types.
|
|
|
|
|
|
Pull-mode negotiation
|
|
~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
Rationale
|
|
^^^^^^^^^
|
|
|
|
A pipeline in pull mode has different negotiation needs than one
|
|
activated in push mode. Push mode is optimized for two use cases:
|
|
|
|
* Playback of media files, in which the demuxers and the decoders are
|
|
the points from which format information should disseminate to the
|
|
rest of the pipeline; and
|
|
|
|
* Recording from live sources, in which users are accustomed to putting
|
|
a capsfilter directly after the source element; thus the caps
|
|
information flow proceeds from the user, through the potential caps
|
|
of the source, to the sinks of the pipeline.
|
|
|
|
In contrast, pull mode has other typical use cases:
|
|
|
|
* Playback from a lossy source, such as RTP, in which more knowledge
|
|
about the latency of the pipeline can increase quality; or
|
|
|
|
* Audio synthesis, in which audio APIs are tuned to producing only the
|
|
necessary number of samples, typically driven by a hardware interrupt
|
|
to fill a DMA buffer or a Jack[0] port buffer.
|
|
|
|
* Low-latency effects processing, whereby filters should be applied as
|
|
data is transferred from a ring buffer to a sink instead of
|
|
beforehand. For example, instead of using the internal alsasink
|
|
ringbuffer thread in push-mode wavsrc ! volume ! alsasink, placing
|
|
the volume inside the sound card writer thread via wavsrc !
|
|
audioringbuffer ! volume ! alsasink.
|
|
|
|
[0] http://jackit.sf.net
|
|
|
|
The problem with pull mode is that the sink has to know the format in
|
|
order to know how many bytes to pull via gst_pad_pull_range(). This
|
|
means that before pulling, the sink must initiate negotation to decide
|
|
on a format.
|
|
|
|
Recalling the principles of capsnego, whereby information must flow from
|
|
those that have it to those that do not, we see that the two named use
|
|
cases have different negotiation requirements:
|
|
|
|
* RTP and low-latency playback are both like the normal playback case,
|
|
in which information flows downstream.
|
|
|
|
* In audio synthesis, the part of the pipeline that has the most
|
|
information is the sink, constrained by the capabilities of the graph
|
|
that feeds it. However the caps are not completely specified; at some
|
|
point the user has to intervene to choose the sample rate, at least.
|
|
This can be done externally to gstreamer, as in the jack elements, or
|
|
internally via a capsfilter, as is customary with live sources.
|
|
|
|
Given that sinks potentially need the input of sources, as in the RTP
|
|
case and at least as a filter in the synthesis case, there must be a
|
|
negotiation phase before the pull thread is activated. Also, given the
|
|
low latency offered by pull mode, we want to avoid capsnego from within
|
|
the pulling thread, in case it causes us to miss our scheduling
|
|
deadlines.
|
|
|
|
The pull thread is usually started in the PAUSED->PLAYING state change. We must
|
|
be able to complete the negotiation before this state change happens.
|
|
|
|
The time to do capsnego, then, is after the SCHEDULING query has succeeded,
|
|
but before the sink has spawned the pulling thread.
|
|
|
|
|
|
Mechanism
|
|
^^^^^^^^^
|
|
|
|
The sink determines that the upstream elements support pull based scheduling by
|
|
doing a SCHEDULING query.
|
|
|
|
The sink initiates the negotiation process by intersecting the results
|
|
of gst_pad_get_caps() on its sink pad and its peer src pad. This is the
|
|
operation performed by gst_pad_get_allowed_caps(). In the simple
|
|
passthrough case, the peer pad's getcaps() function should return the
|
|
intersection of calling get_allowed_caps() on all of its sink pads. In
|
|
this way the sink element knows the capabilities of the entire pipeline.
|
|
|
|
The sink element then fixates the resulting caps, if necessary,
|
|
resulting in the flow caps. From now on, the getcaps function
|
|
of the sinkpad will only return these fixed caps meaning that upstream elements
|
|
will only be able to produce this format.
|
|
|
|
If the sink element could not set caps on its sink pad, it should post
|
|
an error message on the bus indicating that negotiation was not
|
|
possible.
|
|
|
|
When negotiation succeeded, the sinkpad and all upstream internally linked pads
|
|
are activated in pull mode. Typically, this operation will trigger negotiation
|
|
on the downstream elements, which will now be forced to negotiation to the
|
|
final fixed desired caps of the sinkpad.
|
|
|
|
After these steps, the sink element returns ASYNC from the state change
|
|
function. The state will commit to PAUSED when the first buffer is received in
|
|
the sink. This is needed to provide a consistent API to the applications that
|
|
expect ASYNC return values from sinks but it also allows us to perform the
|
|
remainder of the negotiation outside of the context of the pulling thread.
|