gstreamer/docs/random/wtay/scheduling_ideas

Element types
-------------

SOURCES
-------

* never chain-based
* have no sinkpads

1) get based src

   (----------)
   ! fakesrc  !
   !         src-   (get based)
   (----------)

  * no sinkpads
  * srcpad(s) that are get-based

2) loop based src

   (----------)
   ! fakesrc  !
   !         src-   (loop based)
   (----------)

  * no sinkpads
  * element is loop-based
  * data is pushed on the srcpad(s)

FILTERS
-------

3) chain based filter

   (----------)
   ! identity !
 -sink       src- 
   (----------)

  * sinkpad(s) have a chain function
  * srcpad(s) push data

4) loop-based filter

   (----------)
   ! identity !
 -sink       src- 
   (----------)

  * element is loop-based
  * data is pushed on the srcpads
  * data is pulled from sinkpad(s)


SINKS
-----

5) chain based sink

   (----------)
   ! fakesink !
 -sink        !
   (----------)

  * sinkpad(s) have a chain function
  * no srcpads

6) loop-based sink

   (----------)
   ! fakesink !
 -sink        !
   (----------)

  * element is loop-based
  * data is pulled from sinkpad(s)


DECOUPLED
---------

7) decoupled element


   (----------)
   ! queue    !
 -sink       src- 
   (----------)

  * sinkpad(s) have chain function
  * srcpad(s) have get function
  * never loop-based
  * always acts like a chain-based sink for upstream elements
  * always acts like a get-based src for downstream elements
  * are not added to a group, but marked as an entry point in 
    case it acts as a src element


Connection types
----------------

1) get based src
2) loop based src
3) chain based filter
4) loop-based filter
5) chain based sink
6) loop-based sink
7) decoupled element


   !  1  !  2  !  3  !  4  !  5  !  6  !  7  !
---+-----+-----+-----+-----+-----+-----+-----+
 1 !  X  !  X  !  A  !  C  !  A  !  C  !  A  !
   !     !     !     !     !     !     !     !
 2 !  X  !  X  !  B  !  F  !  B  !  F  !  B  !
   !     !     !     !     !     !     !     !
 3 !  X  !  X  !  D  !  E  !  D  !  E  !  D  !
   !     !     !     !     !     !     !     !
 4 !  X  !  X  !  B  !  F  !  B  !  F  !  B  !
   !     !     !     !     !     !     !     !
 5 !  X  !  X  !  X  !  X  !  X  !  X  !  X  !
   !     !     !     !     !     !     !     !
 6 !  X  !  X  !  X  !  X  !  X  !  X  !  X  !
   !     !     !     !     !     !     !     !
 7 !  X  !  X  !  A  !  C  !  A  !  C  !  X  !
   !     !     !     !     !     !     !     !


A)

  src       -> sink
  src       -> filter
  src       -> decoupled
  decoupled -> sink
  decoupled -> filter

 - get based source
 - chain based sink

  * one group
  * src at start of group and entry point
  * _get from src, push to sink

  (-group1---------------)
  !                      !
  ! *fakesrc -> fakesink !
  (----------------------)


B)

  src    -> sink
  src    -> filter
  src    -> decoupled
  filter -> sink
  filter -> filter
  filter -> decoupled

 - loop based source/filter
 - chain based sink/filter/decoupled

 * one group
 * src/filter at start of group and entry point
 * loop on src, chainhandler set to chain function

  (-group1----------------)
  !                       !
  ! %*fakesrc -> fakesink !
  (-----------------------)


C) 

  src       -> sink
  src       -> filter
  decoupled -> sink
  decoupled -> filter

 - get based source/decoupled
 - loop based sink/filter

 * one group
 * loop based element is entry point
 * loop on sink/filter, gethandler set to getfunction
 
  (-group1----------------)
  !                       !
  ! fakesrc -> %*fakesink !
  (-----------------------)

D)

  filter -> filter
  filter -> sink
  filter -> decoupled

  - chain based filter
  - chain based filter/sink/decoupled

  * one group is created to hold the two elements
  * no entry point
  * chainhandler set to peer chainfunction
  
  (-group1----------------)
  !                       !
  ! identity -> identity  !
  (-----------------------)

E)

  filter  -> filter
  filter  -> sink

  - chain based filter
  - loop based filter/sink

  * two groups
  * group is created for src element if needed
  * chainhandler of loop based element set to loop wrapper, control is
    handed to the peer group
  * gethandler of loop based element set to get wrapper

  (-group1---)  (-group2------)
  !          !  !             !
  ! identity ---> %*identity  !
  (----------)  (-------------)


F)

  src     -> filter
  src     -> sink
  filter  -> filter
  filter  -> sink

  - loop based filter/src
  - loop based filter/sink

  * two groups
  * two entry points
  * chainhandler set to loop wrapper
  * gethandler set to get wrapper
  
  (-group1-----)  (-group2------)
  !            !  !             !
  ! %*identity ---> %*identity  !
  (------------)  (-------------)


Grouping
--------

 * a group has at most one loop based element
 * elements in a group are sorted, src elements first (not mandatory)
 * a group has one cothread
 * a group is created immediately for loop based elements, all other elements
   are added to a group when a pad connection is made
 * get-based plugins are put in the same group as a peer loop based element
 * chain based elements are put in the same group as sink peer elements
 * entry point in the group is:
   - loopbased element 
   - first src element if no loopbased element exists in the group

Result: you end up with a group of connected elements with either:
    - a loop based plugin as the entry point
    - a get based plugin as the entry point

Scheduling the group is a matter of starting the cothread and calling
the loop function or doing a _get/_push on a srcpad.


other examples of groups:
-------------------------

 % = loop based
 * = entry point of group
   
.
  (-group1---------------)
  !                      !
  ! *fakesrc -> fakesink !
  (----------------------)

.
  (-group1---------------------------------------)
  !                                              !
  ! *fakesrc -> identity -> identity -> fakesink !
  (----------------------------------------------)

.
  (-group1-----------------------------------------)
  !                                                !
  ! fakesrc -> %*identity -> identity -> fakesink  !
  (------------------------------------------------)

.
  (-group1---------------) (-group2-----------------)
  !                      ! !                        !
  ! *fakesrc -> identity --> *%identity -> fakesink !
  (----------------------) (------------------------)

.
  (-group1------------------------------------)
  !                                           ! 
  ! *fakesrc -> tee  --> identity -> fakesink !
  !                  --> identity -> fakesink !  
  (-------------------------------------------)

.
  (-group1------------------------------------)
  !                                           ! 
  ! *fakesrc -> tee  --> identity -> fakesink !
  (--------------!----------------------------)
                 v
              (-group2-----------------)
              !                        !
              ! *%identity -> fakesink !
              (------------------------)

.
  (-group1----------) (-group2-----------------)
  !                 ! !                        !
  ! *fakesrc -> tee --> *%identity -> fakesink !
  (--------------!--) (------------------------)
                 v
              (-group3-----------------)
              !                        !
              ! *%identity -> fakesink !
              (------------------------)

.
  (-group1-----------------------) (-group2----------------------)
  !                              ! !                             !
  ! filesrc -> *%mpegdemux  --> queue* -> mpeg2dec -> xvideosink !
  !                              ! (-----------------------------)
  !                              ! (-group3----------------------)
  !                              ! !                             !
  !                         --> queue* -> mad -> osssink         !  
  (------------------------------) (-----------------------------)


Chaining
--------

 * groups that are connected end up in the same chain
 * a group always belongs to a chain
 * updating the chain is only needed when two groups are
   connected with a connection of type E/F. for other
   connection types, the group itself is updated.
 * a chain is scheduled by scheduling a random group in the chain.


Wrapper functions
-----------------



iterating without cothreads
---------------------------

A cothread for each group is the easiest way to schedule most of
the pipelines. Some pipelines are however schedulable without
any cothreads.

Each group is schedulable without cothreads, one can call the
group schedule function and be done with it. Problems arise
one the group boundaries of connected elements, which are always 
of type E and F (chain->loop, loop->loop)

We always have a producer group and a provider group in this case.

chain->loop
-----------



Scheduler algorithm
-------------------

1: select (random?) group in chain
2: schedule group
3: on E/F connections, the get/chain wrapper is called
   - get wrapper puts the peer element on the runqueue and
     recursively invokes the scheduler.
   - chain wrapper puts the buffer in the bufpen and puts
     the peer element in the runqueue
4: when the group is scheduled, take group from the runqueue
   and goto 2:
5: no more groups on the runqueue, iteration ends


NOTES:

- We need a GList instead of a single bufpen to hold buffers
  for multi-out elements.
  
- We probably need to set a limit on the maximum number of
  recursions and size of the bufpen list.

- elements run non-preemptively, a group is done scheduling when all
  elements in that group complete their chain/loop function.

- can we only have a stack overflow when there is a loop in the
  pipeline? I think so.

- putting groups twice on the runqueue is not a good idea, we
  need to check a flag or something, maybe give the group a 
  higher priority?

- what about starvation? We'll probably have to put the group
  at the end of the runqueue.

- multi-out elements can introduce significant latency. consider:

  (-group1----------) (-group2-----------------)
  !                 ! !                        !
  ! *filesrc -> mad --> *%identity -> osssink  !
  (-----------------) (------------------------)

  mad produces N output buffers, they are queued in the bufpen, when
  group1 is done, group2 is scheduled to empty the bufpen queue.

  The time it takes for the first buffer to arrive at osssink equals
  the time needed to decode the N-1 other buffers.

  Of course, given the right buffersize for filesrc, N can be reduced
  to 1. mad can also suggest a buffersize to filesrc with the 
  BUFFER_SIZE event.



Ungrouping
----------

Ungrouping might be the hardest part. especially in the case where
an element is still running (state changes and pipeline modifications
in element callbacks).

Ungrouping involves two tasks:

 - breaking up groups. This can happen when a pad connection is broken
   so that the group contains two clusters of unconnected elements.
 - breaking up chains. This happens when a pad connection is broken
   so that the chains contains two clusters of unconnected groups.


case1
-----

The most simple case is where two elements are disconnected and one
of the elements does not belong to a group. In this case, we don't need
to do anything at all.

case2
-----

if the elements are part of different groups, we need to check if the
chain needs to be broken. A chain is broken if it contains two sets
of unconnected groups.

To test this case, we will create a new chain and recursively move
one of the groups with all of its connected groups to it. If the
origial chain is empty after this operation, there was still a connection
and the new chain replaces the old one, else we end up with two chains.

case3
-----

When the elements are part of the same group we check if both elements
still have a connection to some other element in that group. The elements
without connections are removed from the group. 

It is possible that when an element still has a connection with some other
element in the group, the group has to be split up anyway. This can happen
in fakesrc ! indentity ! identity ! fakesink when we break the connection
between the two identity elements. We have to be careful here in the cothread
case that we don't take away the running cothread from under the elements.
In the non-cothread case we can just move the elements to another new group.


Interrupt with cothreadless optimal scheduler
---------------------------------------------

Interrupts are usually performed when a blocking _get based source or
decoupled element is unlocked for a state change. The idea of the 
interrupt scheduler call is to return to the main execution stack frame
ASAP so that the state change can take place. For cothread based
implementations of the scheduler this is not a problem as one can jump
to the main cothread context without problems. For non cothread based
schedulers we need to follow this scheme:

get <-> chain based connection.

 - enter get group scheduler
   - call _get function on source, this can block internally
   - app performs state change, element is told to unlock itself
   - lock inside the get based function unlocks and _get function
     return GST_EVENT_INTERRUPT
   - group scheduler notices the event, unrefs it and jumps out of the
     get_group_scheduler function, a scheduler INTERRUPTED flag is set.

get <-> loop based connection

  - enter loop group scheduler
    - loop based element does pull on sinkpad
    - _get blocks
    - app performs state change, _get is unlocked and returns 
      GST_EVENT_INTERRUPT.
    - loop based function receives INTERRUPT event and exits its loop
      ASAP using gst_element_interrupt.

loop/chain <-> loop based connection

  - when returning from the recursive call to the scheduler, the state
    of the scheduler is checked, if it was interrupted, a
    GST_EVENT_INTERRUPTED event is returned to the loop based element.
  - the loop based element exits its loop ASAP.
      

This technique will unwind the stack of scheduled groups ASAP and returns
to the main execution stack frame where the iterate() function can return
and the state change can take place.

Another alternative could be implemented when the _push and _pull functions
would return a result code...