gstreamer/docs/pwg/advanced-scheduling.xml

400 lines
15 KiB
XML
Raw Normal View History

<chapter id="chapter-loopbased-sched">
<title>How scheduling works</title>
<para>
Scheduling is, in short, a method for making sure that every element gets
called once in a while to process data and prepare data for the next
element. Likewise, a kernel has a scheduler to for processes, and your
brain is a very complex scheduler too in a way.
Randomly calling elements' chain functions won't bring us far, however, so
you'll understand that the schedulers in &GStreamer; are a bit more complex
than this. However, as a start, it's a nice picture.
&GStreamer; currently provides two schedulers: a <emphasis>basic</emphasis>
scheduler and an <emphasis>optimal</emphasis> scheduler. As the name says,
the basic scheduler (<quote>basic</quote>) is an unoptimized, but very
complete and simple scheduler. The optimal scheduler (<quote>opt</quote>),
on the other hand, is optimized for media processing, but therefore also
more complex.
</para>
<para>
Note that schedulers only operate on one thread. If your pipeline contains
multiple threads, each thread will run with a separate scheduler. That is
the reason why two elements running in different threads need a queue-like
element (a <classname>DECOUPLED</classname> element) in between them.
</para>
<sect1 id="section-sched-basic" xreflabel="The Basic Scheduler">
<title>The Basic Scheduler</title>
<para>
The <emphasis>basic</emphasis> scheduler assumes that each element is its
own process. We don't use UNIX processes or POSIX threads for this,
however; instead, we use so-called <emphasis>co-threads</emphasis>.
Co-threads are threads that run besides each other, but only one is active
at a time. The advantage of co-threads over normal threads is that they're
lightweight. The disadvantage is that UNIX or POSIX do not provide such a
thing, so we need to include our own co-threads stack for this to run.
</para>
<para>
The task of the scheduler here is to control which co-thread runs at what
time. A well-written scheduler based on co-threads will let an element run
until it outputs one piece of data. Upon pushing one piece of data to the
next element, it will let the next element run, and so on. Whenever a
running element requires data from the previous element, the scheduler will
switch to that previous element and run that element until it has provided
data for use in the next element.
</para>
<para>
This method of running elements as needed has the disadvantage that a lot
of data will often be queued in between two elements, as the one element
has provided data but the other element hasn't actually used it yet. These
storages of in-between-data are called <emphasis>bufpens</emphasis>, and
they can be visualized as a light <quote>queue</quote>.
</para>
<para>
Note that since every element runs in its own (co-)thread, this scheduler
is rather heavy on your system for larger pipelines.
</para>
</sect1>
<sect1 id="section-sched-opt" xreflabel="The Optimal Scheduler">
<title>The Optimal Scheduler</title>
<para>
The <emphasis>optimal</emphasis> scheduler takes advantage of the fact that
several elements can be linked together in one thread, with one element
controlling the other. This works as follows: in a series of chain-based
elements, each element has a function that accepts one piece of data, and
it calls a function that provides one piece of data to the next element.
The optimal scheduler will make sure that the <function>gst_pad_push ()</function>
function of the first element <emphasis>directly</emphasis> calls the
chain-function of the second element. This significantly decreases the
latency in a pipeline. It takes similar advantage of other possibilities
of short-cutting the data path from one element to the next.
</para>
<para>
The disadvantage of the optimal scheduler is that it is not fully
implemented. Also it is badly documented; for most developers, the opt
scheduler is one big black box. Features that are not implemented
include pad-unlinking within a group while running, pad-selecting
(i.e. waiting for data to arrive on a list of pads), and it can't really
cope with multi-input/-output elements (with the elements linked to each
of these in-/outputs running in the same thread) right now.
</para>
<para>
Some of our developers are intending to write a new scheduler, similar to
the optimal scheduler (but better documented and more completely
implemented).
</para>
</sect1>
</chapter>
<chapter id="chapter-loopbased-loopfn">
<title>How a loopfunc works</title>
<para>
A <function>_loop ()</function> function is a function that is called by
the scheduler, but without providing data to the element. Instead, the
element will become responsible for acquiring its own data, and it will
still be responsible of sending data over to its source pads. This method
noticeably complicates scheduling; you should only write loop-based
elements when you need to. Normally, chain-based elements are preferred.
Examples of elements that <emphasis>have</emphasis> to be loop-based are
elements with multiple sink pads. Since the scheduler will push data into
the pads as it comes (and this might not be synchronous), you will easily
get asynchronous data on both pads, which means that the data that arrives
on the first pad has a different display timestamp than the data arriving
on the second pad at the same time. To get over these issues, you should
write such elements in a loop-based form. Other elements that are
<emphasis>easier</emphasis> to write in a loop-based form than in a
chain-based form are demuxers and parsers. It is not required to write such
elements in a loop-based form, though.
</para>
<para>
Below is an example of the easiest loop-function that one can write:
</para>
<programlisting>
static void gst_my_filter_loopfunc (GstElement *element);
static void
gst_my_filter_init (GstMyFilter *filter)
{
[..]
gst_element_set_loopfunc (GST_ELEMENT (filter), gst_my_filter_loopfunc);
[..]
}
static void
gst_my_filter_loopfunc (GstElement *element)
{
GstMyFilter *filter = GST_MY_FILTER (element);
GstData *data;
/* acquire data */
data = gst_pad_pull (filter->sinkpad);
/* send data */
gst_pad_push (filter->srcpad, data);
}
</programlisting>
<para>
Obviously, this specific example has no single advantage over a chain-based
element, so you should never write such elements. However, it's a good
introduction to the concept.
</para>
<sect1 id="section-loopfn-multiinput" xreflabel="Multi-Input Elements">
<title>Multi-Input Elements</title>
<para>
Elements with multiple sink pads need to take manual control over their
input to assure that the input is synchronized. The following example
code could (should) be used in an aggregator, i.e. an element that takes
input from multiple streams and sends it out intermangled. Not really
useful in practice, but a good example, again.
</para>
<programlisting>
<![CDATA[
typedef struct _GstMyFilterInputContext {
gboolean eos;
GstBuffer *lastbuf;
} GstMyFilterInputContext;
[..]
static void
gst_my_filter_init (GstMyFilter *filter)
{
GstElementClass *klass = GST_ELEMENT_GET_CLASS (filter);
GstMyFilterInputContext *context;
filter->sinkpad1 = gst_pad_new_from_template (
gst_element_class_get_pad_template (klass, "sink"), "sink_1");
context = g_new0 (GstMyFilterInputContext, 1);
gst_pad_set_private_data (filter->sinkpad1, context);
[..]
filter->sinkpad2 = gst_pad_new_from_template (
gst_element_class_get_pad_template (klass, "sink"), "sink_2");
context = g_new0 (GstMyFilterInputContext, 1);
gst_pad_set_private_data (filter->sinkpad2, context);
[..]
gst_element_set_loopfunc (GST_ELEMENT (filter),
gst_my_filter_loopfunc);
}
[..]
static void
gst_my_filter_loopfunc (GstElement *element)
{
GstMyFilter *filter = GST_MY_FILTER (element);
GList *padlist;
GstMyFilterInputContext *first_context = NULL;
/* Go over each sink pad, update the cache if needed, handle EOS
* or non-responding streams and see which data we should handle
* next. */
for (padlist = gst_element_get_padlist (element);
padlist != NULL; padlist = g_list_next (padlist)) {
GstPad *pad = GST_PAD (padlist->data);
GstMyFilterInputContext *context = gst_pad_get_private_data (pad);
if (GST_PAD_IS_SRC (pad))
continue;
while (GST_PAD_IS_USABLE (pad) &&
!context->eos && !context->lastbuf) {
GstData *data = gst_pad_pull (pad);
if (GST_IS_EVENT (data)) {
/* We handle events immediately */
GstEvent *event = GST_EVENT (data);
switch (GST_EVENT_TYPE (event)) {
case GST_EVENT_EOS:
context->eos = TRUE;
gst_event_unref (event);
break;
case GST_EVENT_DISCONTINUOUS:
g_warning ("HELP! How do I handle this?");
/* fall-through */
default:
gst_pad_event_default (pad, event);
break;
}
} else {
/* We store the buffer to handle synchronization below */
context->lastbuf = GST_BUFFER (data);
}
}
/* synchronize streams by always using the earliest buffer */
if (context->lastbuf) {
if (!first_context) {
first_context = context;
} else {
if (GST_BUFFER_TIMESTAMP (context->lastbuf) <
GST_BUFFER_TIMESTAMP (first_context->lastbuf))
first_context = context;
}
}
}
/* If we handle no data at all, we're at the end-of-stream, so
* we should signal EOS. */
if (!first_context) {
gst_pad_push (filter->srcpad, GST_DATA (gst_event_new (GST_EVENT_EOS)));
gst_element_set_eos (element);
return;
}
/* So we do have data! Let's forward that to our source pad. */
gst_pad_push (filter->srcpad, GST_DATA (first_context->lastbuf));
first_context->lastbuf = NULL;
}
]]>
</programlisting>
<para>
Note that a loop-function is allowed to return. Better yet, a loop
function <emphasis>has to</emphasis> return so the scheduler can
let other elements run (this is particularly true for the optimal
scheduler). Whenever the scheduler feels right, it will call the
loop-function of the element again.
</para>
</sect1>
<sect1 id="section-loopfn-bytestream" xreflabel="The Bytestream Object">
<title>The Bytestream Object</title>
<para>
A second type of elements that wants to be loop-based, are the so-called
bytestream-elements. Until now, we've only dealt with elements that
receive or pull full buffers of a random size from other elements. Often,
however, it is wanted to have control over the stream at a byte-level,
such as in stream parsers or demuxers. It is possible to manually pull
buffers and merge them until a certain size; it is easier, however, to
use bytestream, which wraps this behaviour.
</para>
<para>
To use bytestream, you need to load the bytestream when your plugin is
loaded; you should do this before registering the element, which you
learned previously in <xref linkend="section-boiler-plugininit"/>.
After that, all functions of the bytestream plugin are available in
your plugin as well.
</para>
<programlisting>
#include &lt;gst/bytestream/bytestream.h&gt;
static gboolean
plugin_init (GstPlugin *plugin)
{
if (!gst_library_load ("gstbytestream"))
return FALSE;
/* and now, actually register the element */
[..]
}
</programlisting>
<para>
Bytestream-using elements are usually stream parsers or demuxers. For
now, we will take a parser as an example. Demuxers require some more
magic that will be dealt with later in this guide:
<xref linkend="chapter-advanced-request"/>. The goal of this parser will be
to parse a text-file and to push each line of text as a separate buffer
over its source pad.
</para>
<programlisting>
<![CDATA[
static void
gst_my_filter_loopfunc (GstElement *element)
{
GstMyFilter *filter = GST_MY_FILTER (element);
gint n, num;
guint8 *data;
for (n = 0; ; n++) {
num = gst_bytestream_peek_bytes (filter->bs, &data, n + 1);
if (num != n + 1) {
GstEvent *event = NULL;
guint remaining;
gst_bytestream_get_status (filter->bs, &remaining, &event);
if (event) {
if (GST_EVENT_TYPE (event) == GST_EVENT_EOS)) {
/* end-of-file */
gst_pad_push (filter->srcpad, GST_DATA (event));
gst_element_set_eos (element);
return;
}
gst_event_unref (event);
}
/* failed to read - throw error and bail out */
gst_element_error (element, STREAM, READ, (NULL), (NULL));
return;
}
/* check if the last character is a newline */
if (data[n] == '\n') {
GstBuffer *buf = gst_buffer_new_and_alloc (n + 1);
/* read the line of text without newline - then flush the newline */
gst_bytestream_peek_data (filter->bs, &data, n);
memcpy (GST_BUFFER_DATA (buf), data, n);
GST_BUFFER_DATA (buf)[n] = '\0';
gst_bytestream_flush_fast (filter->bs, n + 1);
g_print ("Pushing '%s'\n", GST_BUFFER_DATA (buf));
gst_pad_push (filter->srcpad, GST_DATA (buf));
return;
}
}
}
static void
gst_my_filter_change_state (GstElement *element)
{
GstMyFilter *filter = GST_MY_FILTER (element);
switch (GST_STATE_TRANSITION (element)) {
case GST_STATE_READY_TO_PAUSED:
filter->bs = gst_bytestream_new (filter->sinkpad);
break;
case GST_STATE_PAUSED_TO_READY:
gst_bytestream_destroy (filter->bs);
break;
default:
break;
}
if (GST_ELEMENT_CLASS (parent_class)->change_state)
return GST_ELEMENT_CLASS (parent_class)->change_state (element);
return GST_STATE_SUCCESS;
}
]]>
</programlisting>
<para>
In the above example, you'll notice how bytestream handles buffering of
data for you. The result is that you can handle the same data multiple
times. Event handling in bytestream is currently sort of
<emphasis>wacky</emphasis>, but it works quite well. The one big
disadvantage of bytestream is that it <emphasis>requires</emphasis>
the element to be loop-based. Long-term, we hope to have a chain-based
usable version of bytestream, too.
</para>
</sect1>
<sect1 id="section-loopbased-secnd">
<title>Adding a second output</title>
<para>
WRITEME
</para>
</sect1>
<sect1 id="section-loopbased-modappl">
<title>Modifying the test application</title>
<para>
WRITEME
</para>
</sect1>
</chapter>