mirror of
https://gitlab.freedesktop.org/gstreamer/gstreamer.git
synced 2024-09-09 05:38:58 +00:00
537 lines
14 KiB
Text
537 lines
14 KiB
Text
Element types
|
|
-------------
|
|
|
|
SOURCES
|
|
-------
|
|
|
|
* never chain-based
|
|
* have no sinkpads
|
|
|
|
1) get based src
|
|
|
|
(----------)
|
|
! fakesrc !
|
|
! src- (get based)
|
|
(----------)
|
|
|
|
* no sinkpads
|
|
* srcpad(s) that are get-based
|
|
|
|
2) loop based src
|
|
|
|
(----------)
|
|
! fakesrc !
|
|
! src- (loop based)
|
|
(----------)
|
|
|
|
* no sinkpads
|
|
* element is loop-based
|
|
* data is pushed on the srcpad(s)
|
|
|
|
FILTERS
|
|
-------
|
|
|
|
3) chain based filter
|
|
|
|
(----------)
|
|
! identity !
|
|
-sink src-
|
|
(----------)
|
|
|
|
* sinkpad(s) have a chain function
|
|
* srcpad(s) push data
|
|
|
|
4) loop-based filter
|
|
|
|
(----------)
|
|
! identity !
|
|
-sink src-
|
|
(----------)
|
|
|
|
* element is loop-based
|
|
* data is pushed on the srcpads
|
|
* data is pulled from sinkpad(s)
|
|
|
|
|
|
SINKS
|
|
-----
|
|
|
|
5) chain based sink
|
|
|
|
(----------)
|
|
! fakesink !
|
|
-sink !
|
|
(----------)
|
|
|
|
* sinkpad(s) have a chain function
|
|
* no srcpads
|
|
|
|
6) loop-based sink
|
|
|
|
(----------)
|
|
! fakesink !
|
|
-sink !
|
|
(----------)
|
|
|
|
* element is loop-based
|
|
* data is pulled from sinkpad(s)
|
|
|
|
|
|
DECOUPLED
|
|
---------
|
|
|
|
7) decoupled element
|
|
|
|
|
|
(----------)
|
|
! queue !
|
|
-sink src-
|
|
(----------)
|
|
|
|
* sinkpad(s) have chain function
|
|
* srcpad(s) have get function
|
|
* never loop-based
|
|
* always acts like a chain-based sink for upstream elements
|
|
* always acts like a get-based src for downstream elements
|
|
* are not added to a group, but marked as an entry point in
|
|
case it acts as a src element
|
|
|
|
|
|
Connection types
|
|
----------------
|
|
|
|
1) get based src
|
|
2) loop based src
|
|
3) chain based filter
|
|
4) loop-based filter
|
|
5) chain based sink
|
|
6) loop-based sink
|
|
7) decoupled element
|
|
|
|
|
|
! 1 ! 2 ! 3 ! 4 ! 5 ! 6 ! 7 !
|
|
---+-----+-----+-----+-----+-----+-----+-----+
|
|
1 ! X ! X ! A ! C ! A ! C ! A !
|
|
! ! ! ! ! ! ! !
|
|
2 ! X ! X ! B ! F ! B ! F ! B !
|
|
! ! ! ! ! ! ! !
|
|
3 ! X ! X ! D ! E ! D ! E ! D !
|
|
! ! ! ! ! ! ! !
|
|
4 ! X ! X ! B ! F ! B ! F ! B !
|
|
! ! ! ! ! ! ! !
|
|
5 ! X ! X ! X ! X ! X ! X ! X !
|
|
! ! ! ! ! ! ! !
|
|
6 ! X ! X ! X ! X ! X ! X ! X !
|
|
! ! ! ! ! ! ! !
|
|
7 ! X ! X ! A ! C ! A ! C ! X !
|
|
! ! ! ! ! ! ! !
|
|
|
|
|
|
A)
|
|
|
|
src -> sink
|
|
src -> filter
|
|
src -> decoupled
|
|
decoupled -> sink
|
|
decoupled -> filter
|
|
|
|
- get based source
|
|
- chain based sink
|
|
|
|
* one group
|
|
* src at start of group and entry point
|
|
* _get from src, push to sink
|
|
|
|
(-group1---------------)
|
|
! !
|
|
! *fakesrc -> fakesink !
|
|
(----------------------)
|
|
|
|
|
|
B)
|
|
|
|
src -> sink
|
|
src -> filter
|
|
src -> decoupled
|
|
filter -> sink
|
|
filter -> filter
|
|
filter -> decoupled
|
|
|
|
- loop based source/filter
|
|
- chain based sink/filter/decoupled
|
|
|
|
* one group
|
|
* src/filter at start of group and entry point
|
|
* loop on src, chainhandler set to chain function
|
|
|
|
(-group1----------------)
|
|
! !
|
|
! %*fakesrc -> fakesink !
|
|
(-----------------------)
|
|
|
|
|
|
C)
|
|
|
|
src -> sink
|
|
src -> filter
|
|
decoupled -> sink
|
|
decoupled -> filter
|
|
|
|
- get based source/decoupled
|
|
- loop based sink/filter
|
|
|
|
* one group
|
|
* loop based element is entry point
|
|
* loop on sink/filter, gethandler set to getfunction
|
|
|
|
(-group1----------------)
|
|
! !
|
|
! fakesrc -> %*fakesink !
|
|
(-----------------------)
|
|
|
|
D)
|
|
|
|
filter -> filter
|
|
filter -> sink
|
|
filter -> decoupled
|
|
|
|
- chain based filter
|
|
- chain based filter/sink/decoupled
|
|
|
|
* one group is created to hold the two elements
|
|
* no entry point
|
|
* chainhandler set to peer chainfunction
|
|
|
|
(-group1----------------)
|
|
! !
|
|
! identity -> identity !
|
|
(-----------------------)
|
|
|
|
E)
|
|
|
|
filter -> filter
|
|
filter -> sink
|
|
|
|
- chain based filter
|
|
- loop based filter/sink
|
|
|
|
* two groups
|
|
* group is created for src element if needed
|
|
* chainhandler of loop based element set to loop wrapper, control is
|
|
handed to the peer group
|
|
* gethandler of loop based element set to get wrapper
|
|
|
|
(-group1---) (-group2------)
|
|
! ! ! !
|
|
! identity ---> %*identity !
|
|
(----------) (-------------)
|
|
|
|
|
|
F)
|
|
|
|
src -> filter
|
|
src -> sink
|
|
filter -> filter
|
|
filter -> sink
|
|
|
|
- loop based filter/src
|
|
- loop based filter/sink
|
|
|
|
* two groups
|
|
* two entry points
|
|
* chainhandler set to loop wrapper
|
|
* gethandler set to get wrapper
|
|
|
|
(-group1-----) (-group2------)
|
|
! ! ! !
|
|
! %*identity ---> %*identity !
|
|
(------------) (-------------)
|
|
|
|
|
|
Grouping
|
|
--------
|
|
|
|
* a group has at most one loop based element
|
|
* elements in a group are sorted, src elements first (not mandatory)
|
|
* a group has one cothread
|
|
* a group is created immediatly for loop based elements, all other elements
|
|
are added to a group when a pad connection is made
|
|
* get-based plugins are put in the same group as a peer loop based element
|
|
* chain based elements are put in the same group as sink peer elements
|
|
* entry point in the group is:
|
|
- loopbased element
|
|
- first src element if no loopbased element exists in the group
|
|
|
|
Result: you end up with a group of connected elements with either:
|
|
- a loop based plugin as the entry point
|
|
- a get based plugin as the entry point
|
|
|
|
Scheduling the group is a matter of starting the cothread and calling
|
|
the loop function or doing a _get/_push on a srcpad.
|
|
|
|
|
|
other examples of groups:
|
|
-------------------------
|
|
|
|
% = loop based
|
|
* = entry point of group
|
|
|
|
.
|
|
(-group1---------------)
|
|
! !
|
|
! *fakesrc -> fakesink !
|
|
(----------------------)
|
|
|
|
.
|
|
(-group1---------------------------------------)
|
|
! !
|
|
! *fakesrc -> identity -> identity -> fakesink !
|
|
(----------------------------------------------)
|
|
|
|
.
|
|
(-group1-----------------------------------------)
|
|
! !
|
|
! fakesrc -> %*identity -> identity -> fakesink !
|
|
(------------------------------------------------)
|
|
|
|
.
|
|
(-group1---------------) (-group2-----------------)
|
|
! ! ! !
|
|
! *fakesrc -> identity --> *%identity -> fakesink !
|
|
(----------------------) (------------------------)
|
|
|
|
.
|
|
(-group1------------------------------------)
|
|
! !
|
|
! *fakesrc -> tee --> identity -> fakesink !
|
|
! --> identity -> fakesink !
|
|
(-------------------------------------------)
|
|
|
|
.
|
|
(-group1------------------------------------)
|
|
! !
|
|
! *fakesrc -> tee --> identity -> fakesink !
|
|
(--------------!----------------------------)
|
|
v
|
|
(-group2-----------------)
|
|
! !
|
|
! *%identity -> fakesink !
|
|
(------------------------)
|
|
|
|
.
|
|
(-group1----------) (-group2-----------------)
|
|
! ! ! !
|
|
! *fakesrc -> tee --> *%identity -> fakesink !
|
|
(--------------!--) (------------------------)
|
|
v
|
|
(-group3-----------------)
|
|
! !
|
|
! *%identity -> fakesink !
|
|
(------------------------)
|
|
|
|
.
|
|
(-group1-----------------------) (-group2----------------------)
|
|
! ! ! !
|
|
! filesrc -> *%mpegdemux --> queue* -> mpeg2dec -> xvideosink !
|
|
! ! (-----------------------------)
|
|
! ! (-group3----------------------)
|
|
! ! ! !
|
|
! --> queue* -> mad -> osssink !
|
|
(------------------------------) (-----------------------------)
|
|
|
|
|
|
Chaining
|
|
--------
|
|
|
|
* groups that are connected end up in the same chain
|
|
* a group always belongs to a chain
|
|
* updating the chain is only needed when two groups are
|
|
connected with a connection of type E/F. for other
|
|
connection types, the group itself is updated.
|
|
* a chain is scheduled by scheduling a random group in the chain.
|
|
|
|
|
|
Wrapper functions
|
|
-----------------
|
|
|
|
|
|
|
|
iterating without cothreads
|
|
---------------------------
|
|
|
|
A cothread for each group is the easiest way to schedule most of
|
|
the pipelines. Some pipelines are however schedulable without
|
|
any cothreads.
|
|
|
|
Each group is schedulable without cothreads, one can call the
|
|
group schedule function and be done with it. Problems arise
|
|
one the group boundaries of connected elements, which are always
|
|
of type E and F (chain->loop, loop->loop)
|
|
|
|
We always have a producer group and a provider group in this case.
|
|
|
|
chain->loop
|
|
-----------
|
|
|
|
|
|
|
|
Scheduler algorithm
|
|
-------------------
|
|
|
|
1: select (random?) group in chain
|
|
2: schedule group
|
|
3: on E/F connections, the get/chain wrapper is called
|
|
- get wrapper puts the peer element on the runqueue and
|
|
recursively invokes the scheduler.
|
|
- chain wrapper puts the buffer in the bufpen and puts
|
|
the peer element in the runqueue
|
|
4: when the group is scheduled, take group from the runqueue
|
|
and goto 2:
|
|
5: no more groups on the runqueue, iteration ends
|
|
|
|
|
|
NOTES:
|
|
|
|
- We need a GList instead of a single bufpen to hold buffers
|
|
for multi-out elements.
|
|
|
|
- We probably need to set a limit on the maximum number of
|
|
recursions and size of the bufpen list.
|
|
|
|
- elements run non-preemptively, a group is done scheduling when all
|
|
elements in that group complete their chain/loop function.
|
|
|
|
- can we only have a stack overflow when there is a loop in the
|
|
pipeline? I think so.
|
|
|
|
- putting groups twice on the runqueue is not a good idea, we
|
|
need to check a flag or something, maybe give the group a
|
|
higher priority?
|
|
|
|
- what about starvation? We'll probably have to put the group
|
|
at the end of the runqueue.
|
|
|
|
- multi-out elements can introduce significant latency. consider:
|
|
|
|
(-group1----------) (-group2-----------------)
|
|
! ! ! !
|
|
! *filesrc -> mad --> *%identity -> osssink !
|
|
(-----------------) (------------------------)
|
|
|
|
mad produces N output buffers, they are queued in the bufpen, when
|
|
group1 is done, group2 is scheduled to empty the bufpen queue.
|
|
|
|
The time it takes for the first buffer to arrive at osssink equals
|
|
the time needed to decode the N-1 other buffers.
|
|
|
|
Of course, given the right buffersize for filesrc, N can be reduced
|
|
to 1. mad can also suggest a buffersize to filesrc with the
|
|
BUFFER_SIZE event.
|
|
|
|
|
|
|
|
Ungrouping
|
|
----------
|
|
|
|
Ungrouping might be the hardest part. especially in the case where
|
|
an element is still running (state changes and pipeline modifications
|
|
in element callbacks).
|
|
|
|
Ungrouping involves two tasks:
|
|
|
|
- breaking up groups. This can happen when a pad connection is broken
|
|
so that the group contains two clusters of unconnected elements.
|
|
- breaking up chains. This happens when a pad connection is broken
|
|
so that the chains contains two clusters of unconnected groups.
|
|
|
|
|
|
case1
|
|
-----
|
|
|
|
The most simple case is where two elements are disconnected and one
|
|
of the elements does not belong to a group. In this case, we don't need
|
|
to do anything at all.
|
|
|
|
case2
|
|
-----
|
|
|
|
if the elements are part of different groups, we need to check if the
|
|
chain needs to be broken. A chain is broken if it contains two sets
|
|
of unconnected groups.
|
|
|
|
To test this case, we will create a new chain and recursively move
|
|
one of the groups with all of its connected groups to it. If the
|
|
origial chain is empty after this operation, there was still a connection
|
|
and the new chain replaces the old one, else we end up with two chains.
|
|
|
|
case3
|
|
-----
|
|
|
|
When the elements are part of the same group we check if both elements
|
|
still have a connection to some other element in that group. The elements
|
|
without connections are removed from the group.
|
|
|
|
It is possible that when an element still has a connection with some other
|
|
element in the group, the group has to be split up anyway. This can happen
|
|
in fakesrc ! indentity ! identity ! fakesink when we break the connection
|
|
between the two identity elements. We have to be careful here in the cothread
|
|
case that we don't take away the running cothread from under the elements.
|
|
In the non-cothread case we can just move the elements to another new group.
|
|
|
|
|
|
Interrupt with cothreadless optimal scheduler
|
|
---------------------------------------------
|
|
|
|
Interrupts are usually performed when a blocking _get based source or
|
|
decoupled element is unlocked for a state change. The idea of the
|
|
interrupt scheduler call is to return to the main execution stack frame
|
|
ASAP so that the state change can take place. For cothread based
|
|
implementations of the scheduler this is not a problem as one can jump
|
|
to the main cothread context without problems. For non cothread based
|
|
schedulers we need to follow this scheme:
|
|
|
|
get <-> chain based connection.
|
|
|
|
- enter get group scheduler
|
|
- call _get function on source, this can block internally
|
|
- app performs state change, element is told to unlock itself
|
|
- lock inside the get based function unlocks and _get function
|
|
return GST_EVENT_INTERRUPT
|
|
- group scheduler notices the event, unrefs it and jumps out of the
|
|
get_group_scheduler function, a scheduler INTERRUPTED flag is set.
|
|
|
|
get <-> loop based connection
|
|
|
|
- enter loop group scheduler
|
|
- loop based element does pull on sinkpad
|
|
- _get blocks
|
|
- app performs state change, _get is unlocked and returns
|
|
GST_EVENT_INTERRUPT.
|
|
- loop based function receives INTERRUPT event and exits its loop
|
|
ASAP using gst_element_interrupt.
|
|
|
|
loop/chain <-> loop based connection
|
|
|
|
- when returning from the recursive call to the scheduler, the state
|
|
of the scheduler is checked, if it was interrupted, a
|
|
GST_EVENT_INTERRUPTED event is returned to the loop based element.
|
|
- the loop based element exits its loop ASAP.
|
|
|
|
|
|
This technique will unwind the stack of scheduled groups ASAP and returns
|
|
to the main execution stack frame where the iterate() function can return
|
|
and the state change can take place.
|
|
|
|
Another alternative could be implemented when the _push and _pull functions
|
|
would return a result code...
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|