docs/pwg/: Documentation on how to write source and sink elements. Other stuff in chapter 4 (1-to-n/demuxer, n-to-1/m...

Original commit message from CVS:
* docs/pwg/other-sink.xml:
* docs/pwg/other-source.xml:
Documentation on how to write source and sink elements. Other
stuff in chapter 4 (1-to-n/demuxer, n-to-1/muxer, n-to-n,
manager, autoplugger) are all still pending.

ChangeLog

@@ -1,3 +1,11 @@
+2004-03-24  Ronald Bultje  <rbultje@ronald.bitfreak.net>
+
+	* docs/pwg/other-sink.xml:
+	* docs/pwg/other-source.xml:
+	  Documentation on how to write source and sink elements. Other
+	  stuff in chapter 4 (1-to-n/demuxer, n-to-1/muxer, n-to-n,
+	  manager, autoplugger) are all still pending.
+
 2004-03-25  Benjamin Otte  <otte@gnome.org>
 
 	* testsuite/elements/Makefile.am:

docs/pwg/other-sink.xml

@@ -4,6 +4,164 @@
 <chapter id="chapter-other-sink" xreflabel="Writing a Sink">
   <title>Writing a Sink</title>
   <para>
-    FIXME: write.
+    Sinks are output elements that, opposite to sources, have no source
+    pads and one or more (usually one) sink pad. They can be sound card
+    outputs, disk writers, etc. This chapter will discuss the basic
+    implementation of sink elements.
   </para>
+
+  <sect1 id="other-sink-processing" xreflabel="Data processing, events, synchronization and clocks">
+    <title>Data processing, events, synchronization and clocks</title>
+    <para>
+      Except for corner cases, sink elements will be <function>_chain
+      ()</function>-based elements. The concept of such elements has
+      been discussed before in detail, so that will be skipped here. What
+      is very important in sink elements, specifically in real-time audio
+      and video sources (such as <classname>osssink</classname> or
+      <classname>ximagesink</classname>), is event handling in the
+      <function>_chain ()</function>-function, because most elements rely
+      on EOS-handling of the sink element, and because A/V synchronization
+      can only be perfect if the element takes this into account.
+    </para>
+    <para>
+      How to achieve synchronization between streams depends on whether
+      you're a clock-providing or a clock-receiving element. If you're
+      the clock provider, you can do with time whatever you want. Correct
+      handling would mean that you check whether the end of the previous
+      buffer (if any) and the start of the current buffer are the same.
+      If so, there's no gap between the two and you can continue playing
+      right away. If there is a gap, then you'll need to wait for your
+      clock to reach that time. How to do that depends on the element
+      type. In the case of audio output elements, you would output silence
+      for a while. In the case of video, you would show background color.
+      In case of subtitles, show no subtitles at all.
+    </para>
+    <para>
+      In the case that the provided clock and the received clock are not
+      the same (or in the case where your element provides no clock, which
+      is the same), you simply wait for the clock to reach the timestamp of
+      the current buffer and then you handle the data in it.
+    </para>
+    <para>
+      A simple data handling function would look like this:
+    </para>
+    <programlisting>
+static void
+gst_my_sink_chain (GstPad  *pad,
+		   GstData *data)
+{
+  GstMySink *sink = GST_MY_SINK (gst_pad_get_parent (pad));
+  GstBuffer *buf;
+  GstClockTime time;
+
+  /* only needed if the element is GST_EVENT_AWARE */
+  if (GST_IS_EVENT (data)) {
+    GstEvent *event = GST_EVENT (data);
+
+    switch (GST_EVENT_TYPE (event)) {
+      case GST_EVENT_EOS:
+        [ if your element provides a clock, disable (inactivate) it here ]
+        /* pass-through */
+
+      default:
+        /* the default handler handles discontinuities, even if your
+         * element provides a clock! */
+        gst_pad_event_default (pad, event);
+        break;
+    }
+
+    return;
+  }
+
+  buf = GST_BUFFER (data);
+  if (GST_BUFFER_TIME_IS_VALID (buf))
+    time = GST_BUFFER_TIMESTAMP (buf);
+  else
+    time = sink->expected_next_time;
+
+  /* Synchronization - the property is only useful in case the
+   * element has the option of not syncing. So it is not useful
+   * for hardware-sync (clock-providing) elements. */
+  if (sink->sync) {
+    /* This check is only needed if you provide a clock. Else,
+     * you can always execute the 'else' clause. */
+    if (sink->provided_clock == sink->received_clock) {
+      /* GST_SECOND / 10 is 0,1 sec, it's an arbitrary value. The
+       * casts are needed because else it'll be unsigned and we
+       * won't detect negative values. */
+      if (llabs ((gint64) sink->expected_next_time - (gint64) time) >
+            (GST_SECOND / 10)) {
+        /* so are we ahead or behind? */
+        if (time > sink->expected_time) {
+          /* we need to wait a while... In case of audio, output
+           * silence. In case of video, output background color.
+           * In case of subtitles, display nothing. */
+          [..]
+        } else {
+          /* Drop data. */
+          [..]
+        }
+      }
+    } else {
+      /* You could do more sophisticated things here, but we'll
+       * keep it simple for the purpose of the example. */
+      gst_element_wait (GST_ELEMENT (sink), time);
+    }
+  }
+
+  /* And now handle the data. */
+[..]
+}
+    </programlisting>
+  </sect1>
+
+  <sect1 id="other-sink-buffers" xreflabel="Special memory">
+    <title>Special memory</title>
+    <para>
+      Like source elements, sink elements can sometimes provide externally
+      allocated (such as X-provided or DMA'able) memory to elements earlier
+      in the pipeline, and thereby prevent the need for
+      <function>memcpy ()</function> for incoming data. We do this by
+      providing a pad-allocate-buffer function.
+    </para>
+    <programlisting>
+static GstBuffer *	gst_my_sink_buffer_allocate	(GstPad *pad,
+							 guint64 offset,
+							 guint   size);
+
+static void
+gst_my_sink_init (GstMySink *sink)
+{
+[..]
+  gst_pad_set_bufferalloc_function (sink->sinkpad,
+                                    gst_my_sink_buffer_allocate);
+}
+
+static void
+gst_my_sink_buffer_free (GstBuffer *buf)
+{
+  GstMySink *sink = GST_MY_SINK (GST_BUFFER_PRIVATE (buf));
+
+  /* Do whatever is needed here. */
+[..]
+}
+
+static GstBuffer *
+gst_my_sink_buffer_allocate (GstPad *pad,
+			     guint64 offset,
+			     guint   size)
+{
+  GstBuffer *buf = gst_buffer_new ();
+
+  /* So here it's up to you to wrap your private buffers and
+   * return that. */
+  GST_BUFFER_FREE_DATA_FUNC (buf) = gst_my_sink_buffer_free;
+  GST_BUFFER_PRIVATE (buf) = sink;
+  GST_BUFFER_FLAG_SET (buf, GST_BUFFER_DONTFREE);
+[..]
+
+  return buf;
+}
+    </programlisting>
+  </sect1>
 </chapter>

docs/pwg/other-source.xml

@@ -4,6 +4,459 @@
 <chapter id="chapter-other-source" xreflabel="Writing a Source">
   <title>Writing a Source</title>
   <para>
-    FIXME: write.
+    Source elements are the start of a data streaming pipeline. Source
+    elements have no sink pads and have one or more source pads. We will
+    focus on single-sourcepad elements here, but the concepts apply equally
+    well to multi-sourcepad elements. This chapter will explain the essentials
+    of source elements, which features it should implement and which it
+    doesn't have to, and how source elements will interact with other
+    elements in a pipeline.
   </para>
+
+  <sect1 id="section-source-getfn" xreflabel="The get()-function">
+    <title>The get()-function</title>
+    <para>
+      Source elements have the special option of having a
+      <function>_get ()</function>-function rather than a
+      <function>_loop ()</function>- or <function>_chain
+      ()</function>-function. A <function>_get ()</function>-function is
+      called by the scheduler every time the next elements needs data. Apart
+      from corner cases, every source element will want to be <function>_get
+      ()</function>-based.
+    </para>
+    <programlisting>
+static GstData *	gst_my_source_get	(GstPad *pad);
+
+static void
+gst_my_source_init (GstMySource *src)
+{
+[..]
+  gst_pad_set_get_function (src->srcpad, gst_my_source_get);
+}
+
+static GstData *
+gst_my_source_get (GstPad *pad)
+{
+  GstBuffer *buffer;
+
+  buffer = gst_buffer_new ();
+  GST_BUFFER_DATA (buf) = g_strdup ("hello pipeline!");
+  GST_BUFFER_SIZE (buf) = strlen (GST_BUFFER_DATA (buf));
+  /* terminating '/0' */
+  GST_BUFFER_MAZSIZE (buf) = GST_BUFFER_SIZE (buf) + 1;
+
+  return GST_DATA (buffer);
+}
+    </programlisting>
+  </sect1>
+
+  <sect1 id="section-source-padfn" xreflabel="Events, querying and converting">
+    <title>Events, querying and converting</title>
+    <para>
+      One of the most important functions of source elements is to
+      implement correct query, convert and event handling functions.
+      Those will continuously describe the current state of the stream.
+      Query functions can be used to get stream properties such as current
+      position and length. This can be used by fellow elements to convert
+      this same value into a different unit, or by appliations to provide
+      information about the length/position of the stream to the user.
+      Conversion functions are used to convert such values from one unit
+      to another. Lastly, events are mostly used to seek to positions
+      inside the stream. Any function is essentially optional, but the
+      element should try to provide as much information as it knows. Note
+      that elements providing an event function should also list their
+      supported events in an <function>_get_event_mask ()</function>
+      function. Elements supporting query operations should list the
+      supported operations in a <function>_get_query_types
+      ()</function> function. Elements supporting either conversion
+      or query operations should also implement a <function>_get_formats
+      ()</function> function.
+    </para>
+    <para>
+      An example source element could, for example, be an element that
+      continuously generates a wave tone at 44,1 kHz, mono, 16-bit. This
+      element will generate 44100 audio samples per second or 88,2 kB/s.
+      This information can be used to implement such functions:
+    </para>
+    <programlisting>
+static GstFormat *	gst_my_source_format_list	(GstPad      *pad);
+static GstQueryType *	gst_my_source_query_list	(GstPad      *pad);
+
+static gboolean		gst_my_source_convert		(GstPad      *pad,
+							 GstFormat    from_fmt,
+							 gint64       from_val,
+							 GstFormat   *to_fmt,
+							 gint64      *to_val);
+static gboolean		gst_my_source_query		(GstPad      *pad,
+							 GstQueryType type,
+							 GstFormat   *to_fmt,
+							 gint64      *to_val);
+
+static void
+gst_my_source_init (GstMySource *src)
+{
+[..]
+  gst_pad_set_convert_function (src->srcpad, gst_my_source_convert);
+  gst_pad_set_formats_function (src->srcpad, gst_my_source_format_list);
+  gst_pad_set_query_function (src->srcpad, gst_my_source_query);
+  gst_pad_set_query_type_function (src->srcpad, gst_my_source_query_list);
+}
+
+/*
+ * This function returns an enumeration of supported GstFormat
+ * types in the query() or convert() functions. See gst/gstformat.h
+ * for a full list.
+ */
+
+static GstFormat *
+gst_my_source_format_list (GstPad *pad)
+{
+  static const GstFormat formats[] = {
+    GST_FORMAT_TIME,
+    GST_FORMAT_DEFAULT, /* means "audio samples" */
+    GST_FORMAT_BYTES,
+    0
+  };
+
+  return formats;
+}
+
+/*
+ * This function returns an enumeration of the supported query()
+ * operations. Since we generate audio internally, we only provide
+ * an indication of how many samples we've played so far. File sources
+ * or such elements could also provide GST_QUERY_TOTAL for the total
+ * stream length, or other things. See gst/gstquery.h for details.
+ */
+
+static GstQueryType *
+gst_my_source_query_list (GstPad *pad)
+{
+  static const GstQueryType query_types[] = {
+    GST_QUERY_POSITION,
+    0,
+  };
+
+  return query_types;
+}
+
+/*
+ * And below are the logical implementations.
+ */
+
+static gboolean
+gst_my_source_convert (GstPad    *pad,
+		       GstFormat  from_fmt,
+		       gint64     from_val,
+		       GstFormat *to_fmt,
+		       gint64    *to_val)
+{
+  gboolean res = TRUE;
+  GstMySource *src = GST_MY_SOURCE (gst_pad_get_parent (pad));
+
+  switch (from_fmt) {
+    case GST_FORMAT_TIME:
+      switch (*to_fmt) {
+        case GST_FORMAT_TIME:
+          /* nothing */
+          break;
+
+        case GST_FORMAT_BYTES:
+          *to_val = from_val / (GST_SECOND / (44100 * 2));
+          break;
+
+        case GST_FORMAT_DEFAULT:
+          *to_val = from_val / (GST_SECOND / 44100);
+          break;
+
+        default:
+          res = FALSE;
+          break;
+      }
+      break;
+
+    case GST_FORMAT_BYTES:
+      switch (*to_fmt) {
+        case GST_FORMAT_TIME:
+          *to_val = from_val * (GST_SECOND / (44100 * 2));
+          break;
+
+        case GST_FORMAT_BYTES:
+          /* nothing */
+          break;
+
+        case GST_FORMAT_DEFAULT:
+          *to_val = from_val / 2;
+          break;
+
+        default:
+          res = FALSE;
+          break;
+      }
+      break;
+
+    case GST_FORMAT_DEFAULT:
+      switch (*to_fmt) {
+        case GST_FORMAT_TIME:
+          *to_val = from_val * (GST_SECOND / 44100);
+          break;
+
+        case GST_FORMAT_BYTES:
+          *to_val = from_val * 2;
+          break;
+
+        case GST_FORMAT_DEFAULT:
+          /* nothing */
+          break;
+
+        default:
+          res = FALSE;
+          break;
+      }
+      break;
+
+    default:
+      res = FALSE;
+      break;
+  }
+
+  return res;
+}
+
+static gboolean
+gst_my_source_query (GstPad      *pad,
+		     GstQueryType type,
+		     GstFormat   *to_fmt,
+		     gint64      *to_val)
+{
+  GstMySource *src = GST_MY_SOURCE (gst_pad_get_parent (pad));
+  gboolean res = TRUE;
+
+  switch (type) {
+    case GST_QUERY_POSITION:
+      res = gst_pad_convert (pad, GST_FORMAT_BYTES, src->total_bytes,
+                             to_fmt, to_val);
+      break;
+
+    default:
+      res = FALSE;
+      break;
+  }
+
+  return res;
+}
+    </programlisting>
+    <para>
+      Be sure to increase src->total_bytes after each call to your
+      <function>_get ()</function> function.
+    </para>
+    <para>
+      Event handling has already been explained previously in the events
+      chapter.
+    </para>
+  </sect1>
+
+  <sect1 id="section-source-sync" xreflabel="Time, clocking and synchronization">
+    <title>Time, clocking and synchronization</title>
+    <para>
+      The above example does not provide any timing info, but will suffice
+      for elementary data sources such as a file source or network data
+      source element. Things become slightly more complicated, but still
+      very simple, if we create artificial video or audio data sources,
+      such as a video test image source or an artificial audio source (e.g.
+      <classname>sinesrc</classname> or <classname>silence</classname>).
+      It will become more complicated if we want the element to be a
+      realtime capture source, such as a video4linux source (for reading
+      video frames from a TV card) or an ALSA source (for reading data
+      from soundcards supported by an ALSA-driver). Here, we will need to
+      make the element aware of timing and clocking.
+    </para>
+    <para>
+      Timestamps can essentially be generated from all the information
+      given above without any difficulty. We could add a very small amount
+      of code to generate perfectly timestamped buffers from our
+      <function>_get ()</function>-function:
+    </para>
+    <programlisting>
+static void
+gst_my_source_init (GstMySource *src)
+{
+[..]
+  src->total_bytes = 0;
+}
+
+static GstData *
+gst_my_source_get (GstPad *pad)
+{
+  GstMySource *src = GST_MY_SOURCE (gst_pad_get_parent (pad));
+  GstBuffer *buf;
+  GstFormat fmt = GST_FORMAT_TIME;
+[..]
+  GST_BUFFER_DURATION (buf) = GST_BUFFER_SIZE (buf) * (GST_SECOND / (44100 * 2));
+  GST_BUFFER_TIMESTAMP (buf) = src->total_bytes * (GST_SECOND / (44100 * 2));
+  src->total_bytes += GST_BUFFER_SIZE (buf);
+
+  return GST_DATA (buf);
+}
+
+static GstStateReturn
+gst_my_source_change_state (GstElement *element)
+{
+  GstMySource *src = GST_MY_SOURCE (element);
+
+  switch (GST_STATE_PENDING (element)) {
+    case GT_STATE_PAUSED_TO_READY:
+      src->total_bytes = 0;
+      break;
+
+    default:
+      break;
+  }
+
+  if (GST_ELEMENT_CLASS (parent_class)->change_state)
+    return GST_ELEMENT_CLASS (parent_class)->change_state (element);
+
+  return GST_STATE_SUCCESS;
+}
+    </programlisting>
+    <para>
+      That wasn't too hard. Now, let's assume real-time elements. Those
+      can either have hardware-timing, in which case we can rely on backends
+      to provide sync for us (in which case you probably want to provide a
+      clock), or we will have to emulate that internally (e.g. to acquire
+      sync in artificial data elements such as <classname>sinesrc</classname>).
+      Let's first look at the second option (software sync). The first option
+      (hardware sync + providing a clock) does not require any special code
+      with respect to timing, and the clocking section already explained how
+      to provide a clock.
+    </para>
+    <programlisting>
+enum {
+  ARG_0,
+[..]
+  ARG_SYNC,
+[..]
+};
+
+static void
+gst_my_source_class_init (GstMySourceClass *klass)
+{
+  GObjectClass *object_class = G_OBJECT_CLASS (klass);
+[..]
+  g_object_class_install_property (object_class, ARG_SYNC,
+      g_param_spec_boolean ("sync", "Sync", "Synchronize to clock",
+                            FALSE, G_PARAM_READWRITE));
+[..]
+}
+
+static void
+gst_my_source_init (GstMySource *src)
+{
+[..]
+  src->sync = FALSE;
+}
+
+static void
+gst_my_source_get (GstPad *pad)
+{
+  GstMySource *src = GST_MY_SOURCE (gst_pad_get_parent (pad));
+  GstBuffer *buf;
+[..]
+  if (src->sync) {
+    /* wait on clock */
+    gst_element_wait (GST_ELEMENT (src), GST_BUFFER_TIMESTAMP (buf));
+  }
+
+  return GST_DATA (buf);
+}
+
+static void
+gst_my_source_get_property (GObject    *object,
+			    guint       prop_id,
+			    GParamSpec *pspec,
+			    GValue     *value)
+{
+  GstMySource *src = GST_MY_SOURCE (gst_pad_get_parent (pad));
+
+  switch (prop_id) {
+[..]
+    case ARG_SYNC:
+      g_value_set_boolean (value, src->sync);
+      break;
+[..]
+  }
+}
+
+static void
+gst_my_source_get_property (GObject      *object,
+			    guint         prop_id,
+			    GParamSpec   *pspec,
+			    const GValue *value)
+{
+  GstMySource *src = GST_MY_SOURCE (gst_pad_get_parent (pad));
+
+  switch (prop_id) {
+[..]
+    case ARG_SYNC:
+      src->sync = g_value_get_boolean (value);
+      break;
+[..]
+  }
+}
+    </programlisting>
+    <para>
+      Most of this is GObject wrapping code. The actual code to do
+      software-sync (in the <function>_get ()</function>-function)
+      is relatively small.
+    </para>
+  </sect1>
+  <sect1 id="section-source-buffers" xreflabel="Using special memory">
+    <title>Using special memory</title>
+    <para>
+      In some cases, it might be useful to use specially allocated memory
+      (e.g. <function>mmap ()</function>'ed DMA'able memory) in
+      your buffers, and those will require special handling when they are
+      being dereferenced. For this, &GStreamer; uses the concept of
+      buffer-free functions. Those are special functions pointers that an
+      element can set on buffers that it created itself. The given function
+      will be called when the buffer has been dereferenced, so that the
+      element can clean up or re-use memory internally rather than using
+      the default implementation (which simply calls
+      <function>g_free ()</function> on the data pointer).
+    </para>
+    <programlisting>
+static void
+gst_my_source_buffer_free (GstBuffer *buf)
+{
+  GstMySource *src = GST_MY_SOURCE (GST_BUFFER_PRIVATE (buf));
+
+  /* do useful things here, like re-queueing the buffer which
+   * makes it available for DMA again. The default handler will
+   * not free this buffer because of the GST_BUFFER_DONTFREE
+   * flag. */
+}
+
+static void
+gst_my_source_get (GstPad *pad)
+{
+  GstMySource *src = GST_MY_SOURCE (gst_pad_get_parent (pad));
+  GstBuffer *buf;
+[..]
+  buf = gst_buffer_new ();
+  GST_BUFFER_FREE_DATA_FUNC (buf) = gst_my_source_buffer_free;
+  GST_BUFFER_PRIVATE (buf) = src;
+  GST_BUFFER_FLAG_SET (buf, GST_BUFFER_READONLY | GST_BUFFER_DONTFREE);
+[..]
+
+  return GST_DATA (buf);
+}
+    </programlisting>
+    <para>
+      Note that this concept should <emphasis>not</emphasis> be used to
+      decrease the number of calls made to functions such as
+      <function>g_malloc ()</function> inside your element. We
+      have better ways of doing that elsewhere (&GStreamer; core, Glib,
+      Glibc, Linux kernel, etc.).
+    </para>
+  </sect1>
 </chapter>