From 723c2a48b9d22e5120d04c8a862ed1d7ef06379d Mon Sep 17 00:00:00 2001 From: Wim Taymans Date: Fri, 28 Sep 2012 13:25:30 +0200 Subject: [PATCH] pwg: rework scheduling docs --- docs/pwg/advanced-scheduling.xml | 326 ++++++++++++++++--------------- 1 file changed, 167 insertions(+), 159 deletions(-) diff --git a/docs/pwg/advanced-scheduling.xml b/docs/pwg/advanced-scheduling.xml index bf7932c470..cd88288939 100644 --- a/docs/pwg/advanced-scheduling.xml +++ b/docs/pwg/advanced-scheduling.xml @@ -1,85 +1,93 @@ Different scheduling modes - Scheduling is, in short, a method for making sure that every element gets - called once in a while to process data and prepare data for the next - element. Likewise, a kernel has a scheduler for processes, and your - brain is a very complex scheduler too in a way. - Randomly calling elements' chain functions won't bring us far, however, so - you'll understand that the schedulers in &GStreamer; are a bit more complex - than this. However, as a start, it's a nice picture. + The scheduling mode of a pad defines how data is retrieved from (source) + or given to (sink) pads. &GStreamer; can operate in two scheduling + mode, called push- and pull-mode. &GStreamer; supports elements with pads + in any of the scheduling modes where not all pads need to be operating + in the same mode. So far, we have only discussed _chain ()-operating elements, i.e. elements that have a chain-function set on their sink pad - and push buffers on their source pad(s). Pads (or elements) can also operate - in two other scheduling modes, however. In this chapter, we will discuss - what those scheduling modes are, how they can be enabled and in what - cases they are useful. The other two scheduling modes are random access - (_getrange ()-based) or task-runner (which means - that this element is the driving force in the pipeline) mode. + and push buffers on their source pad(s). We call this the push-mode + because a peer element will use gst_pad_push () on + a srcpad, which will cause our _chain ()-function + to be called, which in turn causes our element to push out a buffer on + the source pad. The initiative to start the dataflow happens somewhere + upstream when it pushes out a buffer and all downstream elements get + scheduled when their _chain ()-functions are + called in turn. + + + Before we explain pull-mode scheduling, let's first understand how the + different scheduling modes are selected and activated on a pad. + xreflabel="The pad activation stage"> The pad activation stage - The stage in which &GStreamer; decides in what scheduling mode the - various elements will operate, is called the pad-activation stage. In - this stage, &GStreamer; will query the scheduling capabilities (i.e. - it will see in what modes each particular element/pad can operate) and - decide on the optimal scheduling composition for the pipeline. Next, - each pad will be notified of the scheduling mode that was assigned to - it, and after that the pipeline will start running. + During the element state change of READY->PAUSED, the pads of an + element will be activated. This happens first on the source pads and + then on the sink pads of the element. &GStreamer; calls the + _activate () of a pad. By default this function + will activate the pad in push-mode by calling + gst_pad_activate_mode () with the GST_PAD_MODE_PUSH + scheduling mode. + It is possible to override the _activate () of a pad + and decide on a different scheduling mode. You can know in what + scheduling mode a pad is activated by overriding the + _activate_mode ()-function. - Pads can be assigned one of three modes, each mode putting several - prerequisites on the pads. Pads should implement a notification - function (gst_pad_set_activatepull_function () and - gst_pad_set_activatepush_function ()) to be - notified of the scheduling mode assignment. Also, sinkpads assigned - to do pull-based scheduling mode should start and stop their task - in this function. + &GStreamer; allows the different pads of an element to operate in + different scheduling modes. This allows for many different possible + use-cases. What follows is an overview of some typical use-cases. - If all pads of an element are assigned to do - push-based scheduling, then this means that data - will be pushed by upstream elements to this element using the - sinkpads _chain ()-function. Prerequisites - for this scheduling mode are that a chain-function was set for - each sinkpad usinggst_pad_set_chain_function () - and that all downstream elements operate in the same mode. Pads are - assigned to do push-based scheduling in sink-to-source element - order, and within an element first sourcepads and then sinkpads. - Sink elements can operate in this mode if their sinkpad is activated - for push-based scheduling. Source elements cannot be chain-based. + If all pads of an element are activated in push-mode scheduling, + the element as a whole is operating in push-mode. + For source elements this means that they will have to start a + task that pushes out buffers on the source pad to the downstream + elements. + Downstream elements will have data pushed to them by upstream elements + using the sinkpads _chain ()-function which will + push out buffers on the source pads. + Prerequisites for this scheduling mode are that a chain-function was + set for each sinkpad using gst_pad_set_chain_function () + and that all downstream elements operate in the same mode. Alternatively, sinkpads can be the driving force behind a pipeline - by operating in pull-based mode, while the sourcepads - of the element still operate in push-based mode. In order to be the + by operating in pull-mode, while the sourcepads + of the element still operate in push-mode. In order to be the driving force, those pads start a GstTask when they are activated. This task is a thread, which will call a function specified by the element. When called, this function will have random data access (through - gst_pad_get_range ()) over all sinkpads, and + gst_pad_pull_range ()) over all sinkpads, and can push data over the sourcepads, which effectively means that this element controls data flow in the pipeline. Prerequisites for - this mode are that all downstream elements can act in chain-based - mode, and that all upstream elements allow random access (see below). - Source elements can be told to act in this mode if their sourcepads - are activated in push-based fashion. Sink elements can be told to - act in this mode when their sinkpads are activated in pull-mode. + this mode are that all downstream elements can act in push + mode, and that all upstream elements operate in pull-mode (see below). + + + Source pads can be activated in PULL mode by a downstream element + when they return GST_PAD_MODE_PULL from the GST_QUERY_SCHEDULING + query. Prerequisites for this scheduling mode are that a + getrange-function was set for the source pad using + gst_pad_set_getrange_function (). - lastly, all pads in an element can be assigned to act in pull-mode. - too. However, contrary to the above, this does not mean that they + Lastly, all pads in an element can be activated in PULL-mode. + However, contrary to the above, this does not mean that they start a task on their own. Rather, it means that they are pull slave for the downstream element, and have to provide random data access to it from their _get_range ()-function. @@ -87,14 +95,18 @@ ()-function was set on this pad using the function gst_pad_set_getrange_function (). Also, if the element has any sinkpads, all those pads (and thereby their - peers) need to operate in random access mode, too. Note that the - element is supposed to activate those elements itself! &GStreamer; - will not do that for you. + peers) need to operate in PULL access mode, too. + + + When a sink element is activated in PULL mode, it should start a + task that calls gst_pad_pull_range () on its + sinkpad. It can only do this when the upstream SCHEDULING query + returns support for the GST_PAD_MODE_PULL scheduling mode. - In the next two sections, we will go closer into pull-based scheduling + In the next two sections, we will go closer into pull-mode scheduling (elements/pads driving the pipeline, and elements/pads providing random access), and some specific use cases will be given. @@ -103,12 +115,12 @@ Pads driving the pipeline - Sinkpads assigned to operate in pull-based mode, while none of its - sourcepads operate in pull-based mode (or it has no sourcepads), can - start a task that will drive the pipeline data flow. Within this - function, those elements have random access over all of their sinkpads, - and push data over their sourcepads. This can come in useful for - several different kinds of elements: + Sinkpads operating in pull-mode, with the sourcepads operating in + push-mode (or it has no sourcepads when it is a sink), can start a task + that will drive the pipeline data flow. + Within this task function, you have random access over all of the sinkpads, + and push data over the sourcepads. + This can come in useful for several different kinds of elements: @@ -116,7 +128,7 @@ Demuxers, parsers and certain kinds of decoders where data comes in unparsed (such as MPEG-audio or video streams), since those will prefer byte-exact (random) access from their input. If possible, - however, such elements should be prepared to operate in chain-based + however, such elements should be prepared to operate in push-mode mode, too. @@ -128,85 +140,40 @@ - In order to start this task, you will need to create it in the - activation function. + First you need to perform a SCHEDULING query to check if the upstream + element(s) support pull-mode scheduling. If that is possible, you + can activate the sinkpad in pull-mode. Inside the activate_mode + function you can then start the task. #include "filter.h" #include <string.h> -static gboolean gst_my_filter_activate (GstPad * pad); -static gboolean gst_my_filter_activate_pull (GstPad * pad, - gboolean active); -static void gst_my_filter_loop (GstMyFilter * filter); +static gboolean gst_my_filter_activate (GstPad * pad, + GstObject * parent); +static gboolean gst_my_filter_activate_mode (GstPad * pad, + GstObject * parent, + GstPadMode mode + gboolean active); +static void gst_my_filter_loop (GstMyFilter * filter); -GST_BOILERPLATE (GstMyFilter, gst_my_filter, GstElement, GST_TYPE_ELEMENT); +G_DEFINE_TYPE (GstMyFilter, gst_my_filter, GST_TYPE_ELEMENT); - static void gst_my_filter_init (GstMyFilter * filter) { [..] gst_pad_set_activate_function (filter->sinkpad, gst_my_filter_activate); - gst_pad_set_activatepull_function (filter->sinkpad, - gst_my_filter_activate_pull); + gst_pad_set_activatemode_function (filter->sinkpad, + gst_my_filter_activate_mode); [..] @@ -217,35 +184,73 @@ gst_my_filter_init (GstMyFilter * filter) --> static gboolean -gst_my_filter_activate (GstPad * pad) +gst_my_filter_activate (GstPad * pad, GstObject * parent) { - if (gst_pad_check_pull_range (pad)) { - return gst_pad_activate_pull (pad, TRUE); - } else { - return FALSE; + GstQuery *query; + gboolean pull_mode; + + /* first check what upstream scheduling is supported */ + query = gst_query_new_scheduling (); + + if (!gst_pad_peer_query (pad, query)) { + gst_query_unref (query); + goto activate_push; + } + + /* see if pull-mode is supported */ + pull_mode = gst_query_has_scheduling_mode_with_flags (query, + GST_PAD_MODE_PULL, GST_SCHEDULING_FLAG_SEEKABLE); + gst_query_unref (query); + + if (!pull_mode) + goto activate_push; + + /* now we can activate in pull-mode. GStreamer will also + * activate the upstream peer in pull-mode */ + return gst_pad_activate_mode (pad, GST_PAD_MODE_PULL, TRUE); + +activate_push: + { + /* something not right, we fallback to push-mode */ + return gst_pad_activate_mode (pad, GST_PAD_MODE_PUSH, TRUE); } } static gboolean -gst_my_filter_activate_pull (GstPad *pad, - gboolean active) +gst_my_filter_activate_pull (GstPad * pad, + GstObject * parent, + GstPadMode mode, + gboolean active) { - GstMyFilter *filter = GST_MY_FILTER (GST_OBJECT_PARENT (pad)); + gboolean res; + GstMyFilter *filter = GST_MY_FILTER (parent); - if (active) { - filter->offset = 0; - return gst_pad_start_task (pad, - (GstTaskFunction) gst_my_filter_loop, filter); - } else { - return gst_pad_stop_task (pad); + switch (mode) { + case GST_PAD_MODE_PUSH: + res = TRUE; + break; + case GST_PAD_MODE_PULL: + if (active) { + filter->offset = 0; + res = gst_pad_start_task (pad, + (GstTaskFunction) gst_my_filter_loop, filter, NULL); + } else { + res = gst_pad_stop_task (pad); + } + break; + default: + /* unknown scheduling mode */ + res = FALSE; + break; } + return res; } Once started, your task has full control over input and output. The most simple case of a task function is one that reads input and pushes that over its source pad. It's not all that useful, but provides some - more flexibility than the old chain-based case that we've been looking + more flexibility than the old push-mode case that we've been looking at so far. @@ -259,7 +264,7 @@ gst_my_filter_loop (GstMyFilter * filter) GstFormat fmt = GST_FORMAT_BYTES; GstBuffer *buf = NULL; - if (!gst_pad_query_duration (filter->sinkpad, &fmt, &len)) { + if (!gst_pad_query_duration (filter->sinkpad, fmt, &len)) { GST_DEBUG_OBJECT (filter, "failed to query duration, pausing"); goto stop; } @@ -307,16 +312,15 @@ stop: Providing random access In the previous section, we have talked about how elements (or pads) - that are assigned to drive the pipeline using their own task, have - random access over their sinkpads. This means that all elements linked - to those pads (recursively) need to provide random access functions. - Requesting random access is done using the function - gst_pad_pull_range (), which requests a buffer of - a specified size and offset. Source pads implementing and assigned to - do random access will have a _get_range ()-function - set using gst_pad_set_getrange_function (), and - that function will be called when the peer pad requests some data. The - element is then responsible for seeking to the right offset and + that are activated to drive the pipeline using their own task, must use + pull-mode scheduling on their sinkpads. This means that all pads linked + to those pads need to be activated in pull-mode. + Source pads activated in pull-mode must implement a + _get_range ()-function set using + gst_pad_set_getrange_function (), and + that function will be called when the peer pad requests some data with + gst_pad_pull_range (). + The element is then responsible for seeking to the right offset and providing the requested data. Several elements can implement random access: @@ -329,17 +333,14 @@ stop: - Filters that would like to provide a pull-based-like scheduling - mode over the whole pipeline. Note that elements assigned to do - random access-based scheduling are themselves responsible for - assigning this scheduling mode to their upstream peers! &GStreamer; - will not do that for you. + Filters that would like to provide a pull-mode scheduling + over the whole pipeline. Parsers who can easily provide this by skipping a small part of - their input and are thus essentially "forwarding" random access + their input and are thus essentially "forwarding" getrange requests literally without any own processing involved. Examples include tag readers (e.g. ID3) or single output parsers, such as a WAVE parser. @@ -354,15 +355,16 @@ stop: #include "filter.h" static GstFlowReturn gst_my_filter_get_range (GstPad * pad, + GstObject * parent, guint64 offset, guint length, GstBuffer ** buf); -GST_BOILERPLATE (GstMyFilter, gst_my_filter, GstElement, GST_TYPE_ELEMENT); +G_DEFINE_TYPE (GstMyFilter, gst_my_filter, GST_TYPE_ELEMENT); +[..] + gst_pad_set_getrange_function (filter->srcpad, gst_my_filter_get_range); + +--> [..] } -static gboolean +static GstFlowReturn gst_my_filter_get_range (GstPad * pad, + GstObject * parent, guint64 offset, guint length, GstBuffer ** buf) { - GstMyFilter *filter = GST_MY_FILTER (GST_OBJECT_PARENT (pad)); + GstMyFilter *filter = GST_MY_FILTER (parent); [.. here, you would fill *buf ..] @@ -423,15 +432,14 @@ gst_my_filter_get_range (GstPad * pad, --> In practice, many elements that could theoretically do random access, - may in practice often be assigned to do push-based scheduling anyway, + may in practice often be activated in push-mode scheduling anyway, since there is no downstream element able to start its own task. Therefore, in practice, those elements should implement both a _get_range ()-function and a _chain ()-function (for filters and parsers) or a _get_range ()-function and be prepared to start their own task by providing _activate_* ()-functions (for - source elements), so that &GStreamer; can decide for the optimal - scheduling mode and have it just work fine in practice. + source elements).