mirrors/gstreamer
mirrors/gstreamer
1
1
Fork 0
mirror of https://gitlab.freedesktop.org/gstreamer/gstreamer.git synced 2024-06-26 01:40:38 +00:00
Code Issues 1 Projects Releases Wiki Activity

Added docs with proposals for major rewrite for 0.9.

Browse source 
Original commit message from CVS:
Added docs with proposals for major rewrite for 0.9.
This commit is contained in:
Wim Taymans 2004-11-29 11:27:26 +00:00
parent a9bad8aa6e
commit 1d4aeab404
3 changed files with 1089 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

328
docs/random/wtay/negotiation3 Normal file
View file

@ -0,0 +1,328 @@
Caps Negotiation
================
Definitions
-----------
- GstCaps is a structure that holds a mimetype and a set of properties. The properties are
either fixed or non-fixed, they contain ranges or lists.
- filter caps: caps set by the application to constrain a link to a specific GstCaps.
- allowed caps: caps calculated when connecting elements, these are the types that are
possible on this connection. If no types are possible, the link is refused.
- pad template caps: caps put on padtemplates to describe the possible media types of
the pad.
- pad caps: the caps of the pad as it is currently negotiated.
General overview
----------------
Caps negotiation is the process of two pads agreeing on a common fixed GstCaps structure
that describes the media type that will be exchanged between those pads.
We define that caps negotiation only happens in the PLAYING state, when there is actual
dataflow and elements thus know what kind of data they are dealing with, or when
linking elements.
Caps negotiation is first performed when linking elements. If the elements can agree
on a media type at link time then negotiation will not have to be performed at runtime.
Usually this is not possible, so the pads remain unnegotiated until at runtime.
Caps negotiation is initiated by an element that wants to send data on a non-negotiated
pad, thus the source pad. The core only provides a policy and convenience methods to aid
the element in performing the negotiation. The core does not keep track of what pads
are negotiated or not, this is a task of the pads.
Caps negotiation is normally started by the source element right before it sends out
data (case 1).
Caps negotiation can also be redone by a sink element that wants to receive another
format from a downstream element (case 2).
There are two functions handling the negotiation, the link function and the do_link
function. The link function is always called first and must eventually call do_link on
the peer pad. The link function must figure out a compatible format for the connection
and then call the do_link function on the peer pad. The do_link function can only
accept or refuse the provided caps.
For autopluggers it is important to know when the pad is ready to start the negotiation.
It is also inportant to know when the negotiation failed and it must be possible to
restart the negotiation with another element. This functionality will be provided
with signals.
Pad functions
-------------
!
! const GstCaps* gst_pad_iterate_caps (GstPad *pad, gint position);
!
Returns the nth possible caps that describes the media type that can flow over
this pad. This function should not return static caps but caps that are
dependent of the media type and the peer connections of the element.
This function can be called at any time so that an autoplugger can know the
exact types of the pads at any time.
!
! gboolean gst_pad_accept_caps (GstPad *pad, GstCaps *caps);
!
Checks that a given caps is acceptable for this pad. Returns FALSE if the
pad cannot handle the caps.
!
! gboolean gst_pad_link (GstPad *pad, GstPad *peer, GstCaps *caps);
!
Tells the pad to start negotiation with the given filtercaps. The
caps need not be fixed and serves as a filter for performing the negotiation
with the peerpad.
!
! gboolean gst_pad_do_link (GstPad *pad, GstPad *peer, GstCaps *caps);
!
Configures the pad to accept data with the given caps. The caps must be fixed.
!
! const GstCaps* gst_pad_get_negotiated_caps (GstPad *pad);
!
Get the negotiated caps of a pad or NULL if the pad is not negotiated.
Linking Elements
----------------
When linking elements with the gst_pad_link function, the core will call
the link function on the srcpad. If that pad returns GST_PAD_LINK_OK,
the link is established, else the link fails.
Since the link function of the pad will call the do_link function of
the peerpad, the caps will be set on both pads.
It is not required to decide on a caps at link time, a plugin can choose to
dynamically renegotiate at runtime.
When a link fails, the core emits a signal link_failed, which the application
can catch to try to establish a new link with another element, for example.
!
! def gst_pad_link_filtered (pad1, pad2, filtercaps):
!
! ... get srcpad and sinkpad ...
! srcpad = (pad1 == GST_PAD_SRC ? pad1 : pad2);
! sinkpad = (pad1 == GST_PAD_SINK ? pad1 : pad2);
!
! ... more checks to see if the pads are of different types and
! ... that they live in the same scheduler etc...
!
! res = gst_pad_link (srcpad, sinkpad, filtercaps)
! if (res == GST_PAD_LINK_OK)
! ... perform other setup ...
! else
! res = signal_emit (srcpad, "link_failed")
!
! return res
!
Pad link is just a convenience function that passes a NULL filtercaps:
!
! def gst_pad_link (pad1, pad2):
! gst_pad_link_filtered (pad1, pad2, NULL)
!
Dynamic renegotiation
---------------------
Dynamic renegotiation happens at runtime when the element knows the exact media
type it is handling.
The element that wants to redecide on the data type of a connection just calls
gst_pad_relink on one of it's pads. The core will call unlink and link on the
pads again as if this were a new connection. Since the iterate_caps function
returns other values while linking, the new link will renegotiate to a new
format or the link will fail.
Prototype of the relink function:
!
! def gst_pad_relink_filtered (pad, filtercaps):
! gst_pad_unlink (pad, peerpad)
! gst_pad_link_filtered (pad, peerpad, filtercaps)
!
Again the relink function is a convenience function for not having to pass
filtercaps.
!
! def gst_pad_relink (pad):
! gst_pad_relink_filtered (pad, NULL)
!
The core, however, should optimize the relink function so that it does a minimal
amount of work, like not informing the scheduler about the unlink/link call in
case of success.
Error recovery
--------------
When a pad is ready to negotiate and it has no peer pad, it fires the "need-link"
signal. Autopluggers can use this signal to start plugging elements to the pad.
When a link fails because of a renegotiation, the "link-failed" signal is fired
so that autopluggers can try other elements.
Default implementations
-----------------------
!
! gst_pad_src_link_func (pad, peerpad, filtercaps)
! {
! srcpad->negotiated_caps = NULL;
! gboolean possible = FALSE;
!
! for (i=0; peercaps = gst_pad_iterate_caps (peerpad, i); i++)
! {
! for (j=0; srccaps = gst_pad_iterate_caps (srcpad, j); j++)
! {
! test = gst_caps_intersect (srccaps, peercaps, filtercaps);
! if (test == GST_CAPS_EMPTY)
! continue;
!
! /* non empty caps, something is possible */
! possible = TRUE;
!
! if (!gst_caps_is_fixed (caps))
! continue;
!
! if (gst_pad_accepts_caps (peerpad, test))
! {
! if (gst_pad_do_link (peerpad, srcpad, test))
! {
! srcpad->negotiated_caps = test;
! goto done;
! }
! else
! {
! /* weird, it accepted but did not want to link */
! }
! }
! else
! {
! /* caps are not accepted by peer, try next */
! }
! }
! }
! done:
! if (!possible)
! return GST_PAD_LINK_FAILED;
! else
! return GST_PAD_LINK_OK;
! }
!
gst_pad_iterate_caps returns caps in the following order:
1) prefered caps (if any)
2) negotiated caps (if any)
3) profile caps (if any, filtered against caps of current media type)
4) padtemplate caps (filtered against caps of current media type)
1 = a caps describing the media type that would result in optimal
processing of the current media type
2 = a caps that it is currently handling
3 = a caps describing the user configured default values
4 = a caps describing all other possibilities that this pad can
produce or consume given the current media type if any.
generic flow 1:
Linkage
src sink
| |
| <--- link(sink, A) |
check if | |
A is ok | |
or filter | |
against | |
allowed | |
src caps | |
| iterate_caps |
if A not +-------------------------> |
fixed | |
| <-------------------------+
filter | |
against A | |
to get A' | |
| can_accept(A') |
if A' fix +-------------------------> |
| | check if A' is ok
| yes |
| <-------------------------+
| |
if A' not | |
fixed, | |
A'=null | |
| do_link(src, A') |
+-------------------------> |
| ok | store src, A'
| <-------------------------+
store | |
sink, A' | |
Unlink
src sink
| |
| <--- unlink() |
| |
| unlink() |
+-------------------------> |
unref | | unref src, format
sink, fmt | |
Dynamic negotiation of B
src sink
| can_accept(B) |
+-------------------------> |
| | check if B is ok
| yes |
| <-------------------------+
| |
| do_link(B) |
+-------------------------> |
| ok | store B
| <-------------------------+
store | |
B | |
TODO
-----
Write test objects to simulate different behaviour.
tests (in python):
vts -> colorspace -> ximagsink
relink with different colorspaces while running
vts -> videoscale -> ximagesink
relink to different sizes while running, scaling vts or ximagesink.
sinesrc -> audiorate -> audiosink
where audiosink only has one possible fixed caps.

85
docs/random/wtay/performance Normal file
View file

@ -0,0 +1,85 @@
Fixing Performance
------------------
1) Observations
The following observations are made when considering the current (17/11/2004)
problems in gstreamer.
- startup time.
Most of the time is spend in reading the xml registry. Several methods exist
for not using a registry (see testsuite/plugin/).
Since registries are pluggable, one can also write a binary registry which
should significantly reduce the size and the load time.
- Performance in state changes
State changes are pretty heavy currently because negotiation happens at
each state change. The PAUSED->PLAYING state change in particular is
too heavy and should be made more 'snappy'.
- Performance in data passing.
Too much checking is done in the push/pull functions. Most of these checks
can be performed more efficiently in the plugins, like checking if the
element is sufficiently negotiated. The discont event 'invent' hack used
for fixing the synchronisation has to go away.
We also propose a get_range based method for pads so that random-access
in file-based sources can happen more efficiently (typical for muxers).
- Scheduling overhead.
The current default opt scheduler has extensive data-structures and does
uses fairly complicated algorithms for grouping and scheduling elements.
This has performance impact on pipeline constructions and pipeline
execution.
- Events in dataflow.
Because events are put inside the dataflow, a typical chain function has
to check if the GstData object is an event of a buffer.
2) Proposed solutions
- startup time.
Volunteers to implement a binary registry?
- The changes listed in the separate document 'threading' will greatly
reduce the number of negotiation steps during state changes. It will
also reduce the latency between PAUSED<->PLAYING since it basically
involves unlocking a mutex in the sinks.
- adding return values to push/pull will offload some checks to the plugins
that can more efficiently and more accuratly handle most error cases.
With the new sync model in 'threading' it is also not needed to have
the 'invented' events in the pull function.
- The following scheduling policies should be made possible for pads:
1) get_range <-> pull-range-based
2) get <-> pull based
3) push <-> chain-based
The scheduling policies are listed in order of preference. A pad should
export what scheduling it supports and the core will select what
scheduling method to use. It is possible for a pad to support more than
one scheduling method.
It is possible that two elements cannot connect when they do not support
compatible scheduling policies. We require that pull-based pads also
support the chain based methods, at minimal, using a helper function to
queue a buffer and that get-based pads also support a push based implementation.
- With the changes listed in 'threading' the scheduler will be a lot more
simple since it does not have to keep track of groups and connected
elements. Starting and scheduling the pipeline will be a matter of
starting the source/loop GstTasks and they will from there on take
over.
- move events out of the dataflow. Different methods will be used to
send/receive events. This will also remove some of the checks in
the push/pull functions. Note that plugins are still able to serialize
buffers and events because they hold the streaming lock.
3) detailed explanation
TO BE WRITTEN

676
docs/random/wtay/threading Normal file
View file

@ -0,0 +1,676 @@
Fixing Threading
----------------
1) Observations
The following observations are made when considering the current (17/11/2004)
problems in gstreamer.
- Bin state changes.
Currently the state of a bin is determined by the highest state of the
children, This is in particular a problem for GstThread because a thread
should start/stop spinning at any time depending on the state of a child.
ex 1:
+-------------------------------------+
| GstThread |
| +--------+ +---------+ +------+ |
| | src | | decoder | | sink | |
| | src-sink src-sink | |
| +--------+ +---------+ +------+ |
+-------------------------------------+
When performing the state change on the GstThread to PLAYING, one of the
children (at random) will go to PLAYING first, this will trigger a method
in GstThread that will start spinning the thread. Some elements are not yet
in the PLAYING state when the scheduler starts iterating elements. This
is not a clean way to start the data passing.
State changes also trigger negotiation and scheduling (in the other thread)
can do too. This creates races in negotiation.
- ERROR and EOS conditions triggering a state change
A typical problem is also that since scheduling starts while the state change
happens, it is possible that the elements go to EOS or ERROR before the
state change completes. Currently this makes the elements go to PAUSED again,
creating races with the state change in progress. This also gives the
impression to the core that the state change failed.
- no locking whatsoever
When an element does a state change, it is possible for another thread to
perform a conflicting state change.
- negotiation is not designed to work over multithread boundaries.
negotiation over a queue is not possible. There is no method or policy of
discovering a media type and then commiting it. It is also not possible to
tie the negotiated media to the relevant buffer.
ex1:
it Should be possible to queue the old and the new formats in a queue.
The element connected to the sinkpad of the queue should be able to
find out that the new format will be accepted by the element connected
on the srcpad of the queue, even if that element is streaming the old
format.
+------------------------------+
| GstQueue |
| +++++++++++++++++++++++++ |
-sink |B|B|B|B|B|B|A|A|A|A|A|A| src-
| +++++++++++++++++++++++++ |
+------------------------------+
+----------+ +----------+
buffers in buffers in
new format old format
- element properties are not threadsafe
When setting an element property while streaming, the element does no
locking whatsoever to guarantee its internal consistency.
- No control over streaming.
When some GstThread is iterating and you want to reconnect a pad, there
is no way to block the pad, perform the actions and then unblock it
again. This leads to thread problems where a pad is negotiation at the
same time that it is passing data.
This is currently solved by PAUSING the pipeline or performing the actions
in the same threadcontext as the iterate loop.
- race conditions in synchronizing the clocks and spinning up the pipeline.
Currently the clock is started as soon as the pipeline is set to playing.
Because some time elaspes before the elements are negotiated, autoplugged
and streaming, the first frame/sample almost always arrives late at the
sinks. Hacks exist to adjust the element base time to compensate for the
delay but this clearly is not clean.
- race conditions when performing seeks in the pipeline. Since the elements
have no control over the streaming threads, they cannot block them or
resync them to the new seek position. It is also hard to synchronize them
with the clock.
- race conditions when sending tags and error messages up the pipeline
hierarchy. These races are either caused by glib refcounting problems and
by not properly locking.
- more as changes are implemented and testcases are written
2) possible solutions
- not allowing threading at all
Run the complete pipeline in a single thread. Complications to solve include
handling of blocking operations like source elements blocking in kernel
space, sink elements blocking on the clock or kernel space, etc.. In practice,
all operations should be made non-blocking because a blocking element can
cause the rest of the pipeline to block as well and cause it to miss a deadline.
A non-blocking model needs cooperation from the kernel (with callbacks) or
requires the use of a polling mechanism, both of which are either impractical
or too CPU intensive and in general not achievable for a general purpose
Multimedia framework. For this reason we will not go further with this
solution.
- Allow threading.
To make this work, We propose the following changes:
- Remove GstThread, it does not add anything useful in a sense that you cannot
arbitrarily place the thread element, it needs decoupled elements around the
borders.
- Simplify the state changes of bins elements. A bin or element never changes
state automatically on EOS and ERROR.
- Introduce the concept of the application and the streaming thread. All data
passing is done in the streaming thread. This also means that all operations
either are performed in the application thread or streaming thread and that
they should be protected against competing operations in other threads.
This would define a policy for adding appropriate locking.
- Move the creation of threads into source and loop-based elements. This will
make it possible for the elements in control of the threads to perform the
locking when needed. One particular instance is for example the state changes,
by creating the threads in the element, it is possible to sync the streaming
and the application thread (which does the state change).
- Remove negotiation from state changes. This will remove the conflict between
streaming and negotiating elements.
- add locks around pad operations like negotiation, streaming, linking, etc. This
will remove races between these conflicting operations. This will also make it
possible to un/block dataflow.
- add locks around bin operations like add/removing elements.
- add locks around element operations like state changes and property changes.
- add a 2-phase directed negotiation process. The source pad queries and figures
out what the sinkpad can take in the first phase. In the second phase it sends
the new format change as an event to the peer element. This event can be
interleaved with the buffers and can travel over queues inbetween the buffers.
Need to rethink this wrt bufferpools (see DShow and old bufferpool implementation)
- add a preroll phase that will be used to spin up the pipeline and align frames/samples
in the sinks. This phase will happen in the PAUSED state. This also means that
dataflow will happen in the PAUSED state. Sinks will not sink samples in the PAUSED
state but will complete their state change asynchronously. This will allow
us to have perfect synchronisation with the clock.
- a two phase seek policy. First the event travels upstream, putting all elements in
the seeking phase and making them synchronize to the new position. In the
second phase the DISCONT event signals the end of the seek and all filters can
continue with the new position.
- Error messages, EOS, tags and other events in the pipeline should be sent to a
mainloop. The app then has an in-thread mechanism for getting information about
the pipeline. It should also be possible to get the messages directly from the
elements itself, like signals. The application programmer has to know that
working these events come from another thread and should handle them accordingly.
- Add return values to push/pull so that errors upstream or downstream can be noted
by other elements so that they can disable themselves or propagate the error.
3) detailed explanation
a) Pipeline construction
Pipeline construction includes:
- adding/removing elements to the bin
- finding elements in a bin by name and interface
- setting the clock on a pipeline.
- setting properties on objects
- linking/unlinking pads
These operations should take the object lock to make sure it can be
executed from different threads.
When connecting pads to other pads from elements inside another bin,
we require that the bin has a ghostpad for the pad. This is needed so
that the bin looks like a self-contained element.
not allowed:
+---------------------+
| GstBin |
+---------+ | +--------+ |
| element | | | src | |
sink src------sink src- ... |
+---------+ | +--------+ |
+---------------------+
allowed:
+-----------------------+
| GstBin |
| +--------+ |
+---------+ | | src | |
| element | | sink src- ... |
sink src---sink/ +--------+ |
+---------+ +-----------------------+
This requirement is important when we need to sort the elements in the
bin to perfrom the state change.
testcases:
- create a bin, add/remove elements from it
- add/remove from different threads and check the bin integrity.
b) state changes
An element can be in one of the four states NULL, READY, PAUSED, PLAYING.
NULL: starting state of the element
READY: element is ready to start running.
PAUSED: element is streaming data, has opened devices etc.
PLAYING: element is streaming data and clock is running
Note that data starts streaming even in the PAUSED state. The only difference
between the PAUSED and PLAYING state is that the clock is running in the
PLAYING state. This mostly has an effect on the renderers which will block on
the first sample they receive when in PAUSED mode. The transition from
READY->PAUSED is called the preroll state. During that transition, media is
queued in the pipeline and autoplugging is done.
Elements are put in a new state using the _set_state function. This function
can return the following return values:
typedef enum {
GST_STATE_FAILURE = 0,
GST_STATE_PARTIAL = 1,
GST_STATE_SUCCESS = 2,
GST_STATE_ASYNC = 3
} GstElementStateReturn;
GST_STATE_FAILURE is returned when the element failed to go to the
required state. When dealing with a bin, this is returned when one
of the elements failed to go to the required state. The other elements
in the bin might have changed their states succesfully. This return
value means that the element did _not_ change state, for bins this
means that not all children have changed their state.
GST_STATE_PARTIAL is returned when some elements in a bin where in the
locked state and therefore did not change their state. Note that the
state of the bin will be changed regardless of this PARTIAL return value.
GST_STATE_SUCCES is returned when all the elements successfully changed their
states.
GST_STATE_ASYNC is returned when an element is going to report the success
or failure of a state change later.
The state of a bin is not related to the state of its children but only to
the last state change directly performed on the bin or on a parent bin. This
means that changing the state of an element inside the bin does not affect
the state of the bin.
Setting the state on a bin that is already in the correct state will
perform the requested state change on the children.
Elements are not allowed to change their own state. For bins, it is allowed
to change the state of its children. This means that the application
can only know about the states of the elements it has explicitly set.
There is a difference in the way a pipeline and a bin handles the state
change of its children:
- a bin returns GST_STATE_ASYNC when one of its children returns an
ASYNC reply.
- a pipeline never returns GST_STATE_ASYNC but returns from the state
change function after all ASYNC elements completed the state change.
This is done by polling the ASYNC elements until they return their
final state.
The state change function must be fast an cannot block. If a blocking behaviour
is unavoidable, the state change function must perform an async state change.
Sink elements therefore always use async state changes since they need to
wait before the first buffer arrives at the sink.
A bin has to change the state of its children elements from the sink to the
source elements. This makes sure that a sink element is always ready to
receive data from a source element in the case of a READY->PAUSED state change.
In the case of a PAUSED->READY state, the sink element will be set to READY
first so that the source element will receive an error when it tries to push
data to this element so that it will shut down as well.
For loop based elements we have to be careful since they can pull a buffer
from the peer element before it has been put in the right state.
The state of a loop based element is therefore only changed after the source
element has been put in the new state.
c) Element state change functions
The core will call the change_state function of an element with the element
lock held. The element is responsible for starting any streaming tasks/threads
and making sure that it synchronizes them to the state change function if
needed.
This means that no other thread is allowed to change the state of the element
at that time and for bins, it is not possible to add/remove elements.
When an element is busy doing the ASYNC state change, it is possible that another
state change happens. The elements should be prepared for this.
An element can receive a state change for the same state it is in. This
is not a problem, some elements (like bins) use this to resynchronize their
children. Other elements should ignore this state change and return SUCCESS.
When performing a state change on an element that returns ASYNC on one of
the state changes, ASYNC is returned and you can only proceed to the next
state change change when this ASYNC state change completed. Use the
gst_element_get_state function to know when the state change completed.
An example of this behaviour is setting a videosink to PLAYING, it will
return ASYNC in the state change from READY->PAUSED. You can only set
it to PLAYING when this state change completes.
Bins will perform the state change code listed in d).
For performing the state change, two variables are used: the current state
of the element and the pending state. When the element is not performing a
state change, the pending state == None. The state change variables are
protected by the element lock. The pending state != None as long as the
state change is performed or when an ASYNC state change is running.
The core provides the following function for applications and bins to
get the current state of an element:
bool gst_element_get_state(&state, &pending, timeout);
This function will block while the state change function is running inside
the element because it grabs the element lock.
When the element did not perform an async state change, this function returns
TRUE immediatly with the state updated to reflect the current state of the
element and pending set to None.
When the element performed an async state change, this function will block
for the value of timeout and will return TRUE if the element completed the
async state change within that timeout, otherwise it returns FALSE, with
the current and pending state filled in.
The algorithm is like this:
bool gst_element_get_state(elem, &state, &pending, timeout)
{
g_mutex_lock (ELEMENT_LOCK);
if (elem->pending != none) {
if (!g_mutex_cond_wait(STATE, ELEMENT_LOCK, timeout) {
/* timeout triggered */
*state = elem->state;
*pending = elem->pending;
ret = FALSE;
}
}
if (elem->pending == none) {
*state = elem->state;
*pending = none;
ret = TRUE;
}
g_mutex_unlock (ELEMENT_LOCK);
return ret;
}
For plugins the following function is provided to commit the pending state,
the ELEMENT_LOCK should be held when calling this function:
gst_element_commit_state(element)
{
if (pending != none) {
state = pending;
pending = none;
}
g_cond_broadcast (STATE);
}
For bins the gst_element_get_state() works slightly different. It will run
the function on all of its children, as soon as one of the children returns
FALSE, the method returns FALSE with the state set to the current bin state
and the pending set to pending state.
For bins with elements that did an ASYNC state change, the _commit_state()
is only executed when actively calling _get_state(). The reason for this is
that when a child of the bin commits its state, this is not automatically
reported to the bin. This is not a problem since the _get_state() function
is the only way to get the current and pending state of the bin and is always
consistent.
d) bin state change algorithm
In order to perform the sink to source state change a bin must be able to sort
the elements. To make this easier we require that elements are connected to
bins using ghostpads on the bin.
The algoritm goes like this:
d = [ ] # list of delayed elements
p = [ ] # list of pending async elements
q = [ elements without srcpads ] # sinks
while q not empty do
e = dequeue q
s = [ all elements connected to e on the sinkpads ]
q = q append s
if e is entry point
d = d append e
else
r = state change e
if r is ASYNC
p = p append e
done
while d not empty do
e = dequeue d
r = state change e
if r is ASYNC
p = p append e
done
# a bin would return ASYNC here if p is not empty
# this last part is only performed by a pipeline
while p not empty do
e = peek p
if state completed e
dequeue e from p
done
The algorithm first tries to find the sink elements, ie. ones without
sinkpads. Then it changes the state of each sink elements and queues
the elements connected to the sinkpads.
The entry points (loopbased and getbased elements) are delayed as we
first need to change the state of the other elements before we can activate
the entry points in the pipeline.
The pipeline will poll the async children before returning.
e) The GstTask infrastructure
A new component: GstTask is added to the core. A task is created by
an instance of the abstract GstScheduler class.
Each schedulable element (when added to a pipeline) is handed a
reference to a GstScheduler. It can use this object to create
a GstTask, which is basically a managed wrapper around a threading
library like GThread. It should be possible to write a GstScheduler
instance that uses other means of scheduling, like one that does not
use threads but implements task switching based on mutex locking.
When source and loopbased elements want to create the streaming thread
they create an instance of a GstTask, which they pass a pointer to
a loop-function. This function will be called as soon as the element
performs GstTask.start(). The element can stop and uses mutexes to
pause the GstTask from, for example, the state change function or the
event functions.
The GstTasks implement the streaming threads.
f) the preroll phase
Element start the streaming threads in the READY->PAUSED state. Since
the elements that start the threads are put in the PAUSED state last,
after their connected elements, they will be able to deliver data to
their peers without problems.
Sink elements like audio and videosinks will return an async state change
reply and will only commit the state change after receiving the first
buffer. This will implement the preroll phase.
The following pseudo code shows an algorithm for commiting the state
change in the streaming method.
GST_LOCK (element);
/* if we are going to PAUSED, we can commit the state change */
if (GST_STATE_TRANSITION (element) == GST_STATE_READY_TO_PAUSED) {
gst_element_commit_state (element);
}
/* if we are paused we need to wait for playing to continue */
if (GST_STATE (element) == GST_STATE_PAUSED) {
/* here we wait for the next state change */
do {
g_cond_wait (element->state_cond, GST_GET_LOCK (element));
} while (GST_STATE (element) == GST_STATE_PAUSED);
/* check if we got playing */
if (GST_STATE (element) != GST_STATE_PLAYING) {
/* not playing, we can't accept the buffer */
GST_UNLOCK (element);
gst_buffer_unref (buf);
return GST_FLOW_WRONG_STATE;
}
}
GST_UNLOCK (element);
g) return values for push/pull
To recover from pipeline errors in a more elegant manner than just
shutting down the pipeline, we need more finegrained error messages
in the data transport. The plugins should be able to know what goes
wrong when interacting with their outside environment. This means
that gst_pad_push/gst_pad_pull and gst_event_send should return a
result code.
Possible return values include:
- GST_OK
- GST_ERROR
- GST_NOT_CONNECTED
- GST_NOT_NEGOTIATED
- GST_WRONG_STATE
- GST_UNEXPECTED
- GST_NOT_SUPPORTED
GST_OK
Data transport was successful
GST_ERROR
An error occured during transport, such as a fatal decoding error,
the pad should not be used again.
GST_NOT_CONNECTED
The pad was not connected
GST_NOT_NEGOTIATED
The peer does not know what datatype is going over the pipeline.
GST_WRONG_STATE
The peer pad is not in the correct state.
GST_UNEXPECTED
The peer pad did not expect the data because it was flushing or
received an eos.
GST_NOT_SUPPORTED
The operation is not supported.
The signatures of the functions will become:
GstFlowReturn gst_pad_push (GstPad *pad, GstBuffer *buffer);
GstFlowReturn gst_pad_pull (GstPad *pad, GstBuffer **buffer);
GstResult gst_pad_push_event (GstPad *pad, GstEvent *event);
- push_event will send the event to the connected pad.
For sending events from the application:
GstResult gst_pad_send_event (GstPad *pad, GstEvent *event);
h) Negotiation
Implement a simple two phase negotiation. First the source queries the
sink if it accepts a certain format, then it sends the new format
as an event. Sink pads can also trigger a state change by requesting
a renegotiation.
i) Mainloop integration/GstBus
All error, warning and EOS messages from the plugins are sent to an event
queue. The pipeline reads the messages from the queue and will either
handle them or forward them to the main event queue that is read by the
application.
Specific pipelines can be written that deal with negotiation messages and
errors in the pipeline intelligently. The basic pipeline will stop the
pipeline when an error occurs.
Whenever an element posts a message on the event queue, a signal is also
fired that can be catched by the application. When dealing with those
signals the application has to be aware that they come from the streaming
threads and need to make sure they use proper locking to protect their
own data structures.
The messages will be implemented using a GstBus object that allows
plugins to post messages and allows the application to read messages either
synchronous or asynchronous. It is also possible to integrate the bus in
the mainloop.
The messages will derive from GstData to make them a lightweight refcounted
object. Need to figure out how we can extend this method to encapsulate
generic signals in messages too.
This decouples the streaming thread from the application thread and should
avoid race conditions and pipeline stalling due to application interaction.
It is still possible to receive the messages in the streaming thread context
if an application wants to. When doing this, special care has to be taken
when performing state changes.
j) EOS
When an element goes to EOS, it sends the EOS event to the peer plugin
and stops sending data on that pad. The peer element that received an EOS
event on a pad can refuse any buffers on that pad.
All elements without source pads must post the EOS message on the message
queue. When the pipeline receives an EOS event from all sinks, it will
post the EOS message on the application message queue so that the application
knows the pipeline is in EOS. Elements without any connected sourcepads
should also post the EOS message. This makes sure that all "dead-ends"
signalled the EOS.
No state change happens when elements go to EOS but the elements with the
GstTask will stop their tasks and so stop producing data.
An application can issue a seek operation which makes all tasks running
again so that they can start streaming from the new location.
A) threads and lowlatency
People often think it is a sin to use threads in low latency applications. This is true
when using the data has to pass thread boundaries but false when it doesn't. Since
source and loop based elements create a thread, it is possible to construct a pipeline
where data passing has to cross thread boundaries, consider this case:
+-----------------------------------+
| +--------+ +--------+ |
| |element1| |element2| |
| .. -sink src-sink src- .. |
| +--------+ +--------+ |
+-----------------------------------+
The two elements are loop base and thus create a thread to drive the pipeline. At the
border between the two elements there is a mutex to pass the data between the two
threads. When using these kinds of element in a pipeline, low-latency will not be
possible. For low-latency apps, don't use these constructs!
Note that in a typical pipeline with one get-based element and two chain-based
elements (decoder/sink) there is only one thread, no data is crossing thread
boundaries and thus this pipeline can be low-latency. Also note that while this
pipeline is streaming no interaction or locking is done between it and the main
application.
+-------------------------------------+
| +--------+ +---------+ +------+ |
| | src | | decoder | | sink | |
| | src-sink src-sink | |
| +--------+ +---------+ +------+ |
+-------------------------------------+
B) howto make non-threaded pipelines
For low latency it is required to not have datapassing cross any thread
borders. Here are some pointers for making sure this requirement is met:
- never connect a loop or chain based element to a loop based element, this
will create a new thread for the sink loop element.
- do not use queues or any other decoupled element, as they implicitly
create a thread boundary.
- At least one thread will be created for any source element (either in the
connected loop-based element or in the source itself) unless the source
elements are connected to the same loop based element.
- when designing sinks, make them non-blocking, use the async clock callbacks
to schedule media rendering in the same thread (if any) as the clock. Sinks that
provide the clock can be made blocking.