gstreamer/docs/manual/advanced-autoplugging.xml

<chapter id="chapter-autoplugging">
  <title>Putting together a pipeline</title>
  <para> 
    The small application we created in the previous chapter used the
    concept of a factory to create the elements. In this chapter we will
    show you how to use the factory concepts to create elements based
    on what they do instead of what they are called.
  </para>

  <para> 
    We will first explain the concepts involved before we move on
    to the reworked helloworld example using autoplugging.
  </para>
  <sect1 id="section-factories-helloworld-problems">
    <title>The problems with the helloworld example</title>
    <para> 
      If we take a look at how the elements were created in the previous
      example we used a rather crude mechanism:
    </para>

    <programlisting>
  ...    
  /* now it's time to get the parser */
  decoder = gst_element_factory_make ("mad", "decoder");
  ...
    </programlisting>

    <para>
      While this mechanism is quite effective it also has some big problems:
      The elements are created based on their name. Indeed, we create an
      element, mad, by explicitly stating the mad element's name.  Our little
      program therefore always uses the mad decoder element to decode
      the MP3 audio stream, even if there are three other MP3 decoders in the
      system. We will see how we can use a more general way to create an
      MP3 decoder element.
    </para>
    <para>
      We have to introduce the concept of MIME types and capabilities 
      added to the source and sink pads.
    </para>
  </sect1>

  <sect1 id="section-factories-mime">
    <title>More on MIME Types</title>
    <para> 
      GStreamer uses MIME types to identify the different types of data
      that can be handled by the elements. They are the high level
      mechanisms to make sure that everyone is talking about the right
      kind of data. 
    </para> 
    <para> 
      A MIME (Multipurpose Internet Mail Extension) type is a pair of
      strings that denote a certain type of data. Examples include:
      <itemizedlist>
        <listitem>
          <para>
	    audio/x-raw-int : raw audio samples
          </para>
        </listitem>
        <listitem>
          <para>
	    audio/mpeg : MPEG audio
          </para>
        </listitem>
        <listitem>
          <para>
	    video/mpeg : MPEG video
          </para>
        </listitem>
      </itemizedlist>
    </para> 
    <para> 
      An element must associate a MIME type to its source and sink pads
      when it is loaded into the system. GStreamer knows about the
      different elements and what type of data they expect and emit.
      This allows for very dynamic and extensible element creation as we
      will see.
    </para> 
    <para> 
      As we have seen in the previous chapter, MIME types are added
      to the Capability structure of a pad.
    </para> 

    <para>
      <xref linkend="section-mime-img"/> shows the MIME types associated with
      each pad from the "hello world" example.
    </para>
    <figure float="1" id="section-mime-img">
      <title>The Hello world pipeline with MIME types</title>
      <mediaobject>
        <imageobject>
          <imagedata fileref="images/mime-world.&image;" format="&IMAGE;" />
        </imageobject>
      </mediaobject>  

    </figure>
    <para>
      We will see how you can create an element based on the MIME types
      of its source and sink pads. This way the end-user will have the
      ability to choose his/her favorite audio/mpeg decoder without
      you even having to care about it.
    </para>
    <para>
      The typing of the source and sink pads also makes it possible to
      'autoplug' a pipeline. We will have the ability to say: "construct
      a pipeline that does an audio/mpeg to audio/x-raw-int conversion".
    </para>
    <note>
      <para>
        The basic GStreamer library does not try to solve all of your 
	autoplug problems. It leaves the hard decisions to the application
	programmer, where they belong. 
      </para>
    </note>
    
  </sect1>

  <sect1 id="section-factories-gstreamer-types">
    <title>GStreamer types</title>
    <para> 
      GStreamer assigns a unique number to all registered MIME types.
      GStreamer also keeps a reference to
      a function that can be used to determine if a given buffer is of
      the given MIME type.
    </para>
    <para>
      There is also an association between a MIME type and a file extension,
      but the use of typefind functions (similar to file(1)) is preferred.
    </para>
    <para> 
      The type information is maintained in a list of 
      <classname>GstType</classname>. The definition of a 
      <classname>GstType</classname> is like:
    </para>
    <para>
      <programlisting>
typedef GstCaps (*GstTypeFindFunc) (GstBuffer *buf,gpointer *priv);
 
typedef struct _GstType GstType;

struct _GstType {
  guint16 id;                   /* type id (assigned) */

  gchar *mime;                  /* MIME type */
  gchar *exts;                  /* space-delimited list of extensions */

  GstTypeFindFunc typefindfunc; /* typefind function */
};
      </programlisting>
    </para> 
    <para> 
      All operations on <classname>GstType</classname> occur
      via their <classname>guint16 id</classname> numbers, with
      the <classname>GstType</classname> structure private to the GStreamer
      library.
    </para> 

    <sect2>
    <title>MIME type to id conversion</title>

      <para> 
        We can obtain the id for a given MIME type
        with the following piece of code:
      </para> 
      <programlisting>
  guint16 id;
  
  id = gst_type_find_by_mime ("audio/mpeg");
      </programlisting>
      <para> 
        This function will return 0 if the type was not known.
      </para> 
    </sect2>

    <sect2>
    <title>id to <classname>GstType</classname> conversion</title>
      <para> 
        We can obtain the <classname>GstType</classname> for a given id 
        with the following piece of code:
      </para> 
      <programlisting>
  GstType *type;
  
  type = gst_type_find_by_id (id);
      </programlisting>
      <para> 
        This function will return NULL if the id was not associated with
        any known <classname>GstType</classname>
      </para> 
    </sect2>

    <sect2>
    <title>extension to id conversion</title>
      <para> 
        We can obtain the id for a given file extension
        with the following piece of code:
      </para> 
      <programlisting>
  guint16 id;
  
  id = gst_type_find_by_ext (".mp3");
      </programlisting>
      <para> 
        This function will return 0 if the extension was not known.
      </para> 
    </sect2>
  </sect1>

  <sect1 id="section-factories-create">
    <title>Creating elements with the factory</title>
    <para>
      In the previous section we described how you could obtain
      an element factory using MIME types. One the factory has been
      obtained, you can create an element using:
    </para>
    <programlisting>
  GstElementFactory *factory;
  GstElement *element;

  // obtain the factory
  factory = ... 

  element = gst_element_factory_create (factory, "name");
    </programlisting>
    <para>
      This way, you do not have to create elements by name which
      allows the end-user to select the elements he/she prefers for the
      given MIME types.
    </para>
  </sect1>

  <sect1 id="section-factories-basic-types">
    <title>GStreamer basic types</title>
    <para>
      GStreamer only has two builtin types:
    </para>
    <itemizedlist>
      <listitem>
        <para>
          audio/raw : raw audio samples
        </para>
      </listitem>
      <listitem>
        <para>
          video/raw and image/raw : raw video data
        </para>
      </listitem>
    </itemizedlist>
    <para>
      All other MIME types are maintained by the plugin elements.
    </para>

  </sect1>

  <sect1 id="chapter-dynamic">
  <title>Dynamic pipelines</title>
  <para> 
    In this chapter we will see how you can create a dynamic pipeline. A
    dynamic pipeline is a pipeline that is updated or created while data
    is flowing through it. We will create a partial pipeline first and add
    more elements while the pipeline is playing. Dynamic pipelines cause
    all sorts of scheduling issues and will remain a topic of research for
    a long time in GStreamer.
  </para>
  <para> 
    We will show how to create an MPEG1 video player using dynamic pipelines.
    As you have seen in the pad section, we can attach a signal to an element
    when a pad is created. We will use this to create our MPEG1 player.
  </para>

  <para> 
    We'll start with a simple main function:
  </para>
  <programlisting>

/* example-begin dynamic.c */
#include &lt;string.h&gt;
#include &lt;gst/gst.h&gt;

void 
eof (GstElement *src) 
{
  g_print ("have eos, quitting\n");
  exit (0);
}

gboolean 
idle_func (gpointer data) 
{
  gst_bin_iterate (GST_BIN (data));
  return TRUE;
}

void 
new_pad_created (GstElement *parse, GstPad *pad, GstElement *pipeline) 
{
  GstElement *decode_video = NULL;
  GstElement *decode_audio, *play, *color, *show;
  GstElement *audio_queue, *video_queue;
  GstElement *audio_thread, *video_thread;

  g_print ("***** a new pad %s was created\n", gst_pad_get_name (pad));
  
  gst_element_set_state (GST_ELEMENT (pipeline), GST_STATE_PAUSED);

  /* link to audio pad */
  if (strncmp (gst_pad_get_name (pad), "audio_", 6) == 0) {

    /* construct internal pipeline elements */
    decode_audio = gst_element_factory_make ("mad", "decode_audio");
    g_return_if_fail (decode_audio != NULL);
    play = gst_element_factory_make ("osssink", "play_audio");
    g_return_if_fail (play != NULL);

    /* create the thread and pack stuff into it */
    audio_thread = gst_thread_new ("audio_thread");
    g_return_if_fail (audio_thread != NULL);

    /* construct queue and link everything in the main pipeline */
    audio_queue = gst_element_factory_make ("queue", "audio_queue");
    g_return_if_fail (audio_queue != NULL);

    gst_bin_add_many (GST_BIN (audio_thread), 
                      audio_queue, decode_audio, play, NULL);

    /* set up pad links */
    gst_element_add_ghost_pad (audio_thread,
                               gst_element_get_pad (audio_queue, "sink"),
                               "sink");
    gst_element_link (audio_queue, decode_audio);
    gst_element_link (decode_audio, play);

    gst_bin_add (GST_BIN (pipeline), audio_thread);

    gst_pad_link (pad, gst_element_get_pad (audio_thread, "sink"));

    /* set up thread state and kick things off */
    g_print ("setting to READY state\n");
    gst_element_set_state (GST_ELEMENT (audio_thread), GST_STATE_READY);

  } 
  else if (strncmp (gst_pad_get_name (pad), "video_", 6) == 0) {

    /* construct internal pipeline elements */
    decode_video = gst_element_factory_make ("mpeg2dec", "decode_video");
    g_return_if_fail (decode_video != NULL);

    color = gst_element_factory_make ("colorspace", "color");
    g_return_if_fail (color != NULL);

   
    show = gst_element_factory_make ("xvideosink", "show");
    g_return_if_fail (show != NULL);

    /* construct queue and link everything in the main pipeline */
    video_queue = gst_element_factory_make ("queue", "video_queue");
    g_return_if_fail (video_queue != NULL);

    /* create the thread and pack stuff into it */
    video_thread = gst_thread_new ("video_thread");
    g_return_if_fail (video_thread != NULL);
    gst_bin_add_many (GST_BIN (video_thread), video_queue, 
                      decode_video, color, show, NULL);

    /* set up pad links */
    gst_element_add_ghost_pad (video_thread,
                               gst_element_get_pad (video_queue, "sink"),
                               "sink");
    gst_element_link (video_queue, decode_video);
    gst_element_link_many (decode_video, color, show, NULL);

    gst_bin_add (GST_BIN (pipeline), video_thread);

    gst_pad_link (pad, gst_element_get_pad (video_thread, "sink"));

    /* set up thread state and kick things off */
    g_print ("setting to READY state\n");
    gst_element_set_state (GST_ELEMENT (video_thread), GST_STATE_READY);
  }
  gst_element_set_state (GST_ELEMENT (pipeline), GST_STATE_PLAYING);
}

int 
main (int argc, char *argv[]) 
{
  GstElement *pipeline, *src, *demux;

  gst_init (&amp;argc, &amp;argv);

  pipeline = gst_pipeline_new ("pipeline");
  g_return_val_if_fail (pipeline != NULL, -1);

  src = gst_element_factory_make ("filesrc", "src");
  g_return_val_if_fail (src != NULL, -1);
  if (argc &lt; 2) 
    g_error ("Please specify a video file to play !");

  g_object_set (G_OBJECT (src), "location", argv[1], NULL);

  demux = gst_element_factory_make ("mpegdemux", "demux");
  g_return_val_if_fail (demux != NULL, -1);

  gst_bin_add_many (GST_BIN (pipeline), src, demux, NULL);

  g_signal_connect (G_OBJECT (demux), "new_pad",
                     G_CALLBACK (new_pad_created), pipeline);

  g_signal_connect (G_OBJECT (src), "eos",
                     G_CALLBACK (eof), NULL);

  gst_element_link (src, demux);

  gst_element_set_state (GST_ELEMENT (pipeline), GST_STATE_PLAYING);

  g_idle_add (idle_func, pipeline);

  gst_main ();

  return 0;
}
/* example-end dynamic.c */
  </programlisting>
  <para>
    We create two elements: a file source and an MPEG demuxer. 
    There's nothing special about this piece of code except for 
    the signal 'new_pad' that we linked to the mpegdemux 
    element using:
  </para>
  <programlisting>
  g_signal_connect (G_OBJECT (demux), "new_pad",
                     G_CALLBACK (new_pad_created), pipeline);
  </programlisting>
  <para> 
    When an elementary stream has been detected in the system stream,
    mpegdemux will create a new pad that will provide the data of the
    elementary stream. A function 'new_pad_created' will be called when
    the pad is created.
  </para>
  <para>
    In the above example, we created new elements based on the name of 
    the newly created pad. We then added them to a new thread. 
    There are other possibilities to check the type of the pad, for
    example by using the MIME type and the properties of the pad.
  </para>
  </sect1>


  <sect1 id="chapter-typedetection">
  <title>Type Detection</title>
  <para> 
    Sometimes the capabilities of a pad are not specificied. The filesrc
    element, for example, does not know what type of file it is reading. Before
    you can attach an element to the pad of the filesrc, you need to determine
    the media type in order to be able to choose a compatible element.
  </para>
  <para> 
    To solve this problem, a plugin can provide the <application>GStreamer</application> 
    core library with a type definition. The type definition
    will contain the following information:
    <itemizedlist>
      <listitem>
        <para>
          The MIME type we are going to define.
        </para>
      </listitem>
      <listitem>
        <para>
          An optional string with a list of possible file extensions this 
	  type usually is associated with. the list entries are separated with
	  a space. eg, ".mp3 .mpa .mpg".
        </para>
      </listitem>
      <listitem>
        <para>
	  An optional typefind function. 
        </para>
      </listitem>
    </itemizedlist>
  </para>
  <para> 
    The typefind functions give a meaning to the MIME types that are used
    in GStreamer.  The typefind function is a function with the following definition:
  </para>
  <programlisting>
typedef GstCaps *(*GstTypeFindFunc) (GstBuffer *buf, gpointer priv);
  </programlisting>
  <para> 
    This typefind function will inspect a GstBuffer with data and will output
    a GstCaps structure describing the type. If the typefind function does not
    understand the buffer contents, it will return NULL.
  </para>
  <para>
    <application>GStreamer</application> has a typefind element in the set
    of core elements
    that can be used to determine the type of a given pad.
  </para>
  <para>
    The next example will show how a typefind element can be inserted into a pipeline
    to detect the media type of a file. It will output the capabilities of the pad into
    an XML representation.
  </para>
  <programlisting>
#include &lt;gst/gst.h&gt;

void    type_found      (GstElement *typefind, GstCaps* caps);

int 
main(int argc, char *argv[]) 
{
  GstElement *bin, *filesrc, *typefind;

  gst_init (&amp;argc, &amp;argv);

  if (argc != 2) {
    g_print ("usage: %s &lt;filename&gt;\n", argv[0]);
    exit (-1);
  }

  /* create a new bin to hold the elements */
  bin = gst_bin_new ("bin");
  g_assert (bin != NULL);

  /* create a disk reader */
  filesrc = gst_element_factory_make ("filesrc", "disk_source");
  g_assert (filesrc != NULL);
  g_object_set (G_OBJECT (filesrc), "location", argv[1], NULL);

  /* create the typefind element */
  typefind = gst_element_factory_make ("typefind", "typefind");
  g_assert (typefind != NULL);

  /* add objects to the main pipeline */
  gst_bin_add_many (GST_BIN (bin), filesrc, typefind, NULL);

  g_signal_connect (G_OBJECT (typefind), "have_type", 
		     G_CALLBACK (type_found), NULL);

  gst_element_link (filesrc, typefind);

  /* start playing */
  gst_element_set_state (GST_ELEMENT (bin), GST_STATE_PLAYING);

  gst_bin_iterate (GST_BIN (bin));

  gst_element_set_state (GST_ELEMENT (bin), GST_STATE_NULL);

  exit (0);
}
  </programlisting>
  <para>
    We create a very simple pipeline with only a filesrc and the typefind
    element in it. The sinkpad of the typefind element has been linked
    to the source pad of the filesrc.
  </para>
  <para>
    We attached a signal 'have_type' to the typefind element which will be called 
    when the type of the media stream as been detected.
  </para>
  <para>
    The typefind function will loop over all the registered types and will
    execute each of the typefind functions. As soon as a function returns
    a GstCaps pointer, the type_found function will be called:
  </para>

  <programlisting>
void
type_found (GstElement *typefind, GstCaps* caps) 
{
  xmlDocPtr doc;
  xmlNodePtr parent;
  
  doc = xmlNewDoc ("1.0");  
  doc-&gt;root = xmlNewDocNode (doc, NULL, "Capabilities", NULL);

  parent = xmlNewChild (doc-&gt;root, NULL, "Caps1", NULL);
  gst_caps_save_thyself (caps, parent);

  xmlDocDump (stdout, doc);
}
  </programlisting>
  <para>
    In the type_found function we can print or inspect the type that has been
    detected using the GstCaps APIs. In this example, we just print out the
    XML representation of the caps structure to stdout.
  </para>
  <para>
    A more useful option would be to use the registry to look up an element
    that can handle this particular caps structure, or we can also use the
    autoplugger to link this caps structure to, for example, a videosink.
  </para>

  </sect1>

  <sect1 id="section-autoplugging-spider">
    <title>Another approach to autoplugging</title>
    <para>
      The autoplug API is interesting, but often impractical. It is static;
      it cannot deal with dynamic pipelines. An element that will
      automatically figure out and decode the type is more useful.
      Enter the spider.
    </para>
    <sect2>
      <title>The spider element</title>
      <para>
        The spider element is a generalized autoplugging element. At this point (April 2002), it's
        the best we've got; it can be inserted anywhere within a pipeline to perform caps
        conversion, if possible. Consider the following gst-launch line:
        <programlisting>
          $ gst-launch filesrc location=my.mp3 ! spider ! osssink
        </programlisting>
        The spider will detect the type of the stream, autoplug it to the osssink's caps, and play
        the pipeline. It's neat.
      </para>
    </sect2>
    <sect2>
      <title>Spider features</title>
      <para>
        <orderedlist>
          <listitem>
            <para>
              Automatically typefinds the incoming stream.
            </para>
          </listitem>
         <listitem>
            <para>
              Has request pads on the source side. This means that it can
              autoplug one source stream into many sink streams. For example,
              an MPEG1 system stream can have audio as well as video; that
              pipeline would be represented in gst-launch syntax as

              <programlisting>
                $ gst-launch filesrc location=my.mpeg1 ! spider ! { queue ! osssink } spider.src_%d!
                             { queue ! xvideosink }
              </programlisting>
            </para>
          </listitem>
        </orderedlist>
      </para>
    </sect2>
  </sect1>

</chapter>