gstreamer/docs/pwg/intro-basics.xml

<!-- ############ chapter ############# -->

<chapter id="chapter-intro-basics" xreflabel="Foundations">
  <title>Foundations</title>
  <para><!-- synchronize with AppDevMan -->
    This chapter of the guide introduces the basic concepts of &GStreamer;.
    Understanding these concepts will help you grok the issues involved in
    extending &GStreamer;. Many of these concepts are explained in greater
    detail in the &GstAppDevMan;; the basic concepts presented here serve mainly
    to refresh your memory.
  </para>

  <!-- ############ sect1 ############# -->

  <sect1 id="section-basics-elements" xreflabel="Elements and Plugins">
    <title>Elements and Plugins</title>
    <para>
      Elements are at the core of &GStreamer;. In the context of plugin
      development, an <emphasis>element</emphasis> is an object derived from the
      <ulink type="http" url="../../gstreamer/html/GstElement.html"><classname>
      GstElement</classname></ulink> class. Elements provide some sort of
      functionality when linked with other elements: For example, a source
      element provides data to a stream, and a filter element acts on the data
      in a stream. Without elements, &GStreamer; is just a bunch of conceptual
      pipe fittings with nothing to link. A large number of elements ship
      with &GStreamer;, but extra elements can also be written.
    </para>
    <para>
      Just writing a new element is not entirely enough, however: You will need
      to encapsulate your element in a <emphasis>plugin</emphasis> to enable
      &GStreamer; to use it. A plugin is essentially a loadable block of code,
      usually called a shared object file or a dynamically linked library. A
      single plugin may contain the implementation of several elements, or just
      a single one. For simplicity, this guide concentrates primarily on plugins
      containing one element.
    </para>
    <para>
      A <emphasis>filter</emphasis> is an important type of element that
      processes a stream of data. Producers and consumers of data are called
      <emphasis>source</emphasis> and <emphasis>sink</emphasis> elements,
      respectively. <emphasis>Bin</emphasis> elements contain other elements.
      One type of bin is responsible for synchronization of the elements that they
      contain so that data flows smoothly. Another type of bin, called
      <emphasis>autoplugger</emphasis> elements, automatically add other
      elements to the bin and links them together so that they act as a
      filter between two arbitrary stream types.
    </para>
    <para>
      The plugin mechanism is used everywhere in &GStreamer;, even if only the
      standard packages are being used. A few very basic functions reside in the
      core library, and all others are implemented in plugins. A plugin registry
      is used to store the details of the plugins in an binary registry file.
      This way, a program using &GStreamer; does not have to load all plugins to
      determine which are needed. Plugins are only loaded when their provided
      elements are requested.
    </para>
    <para>
      See the &GstLibRef; for the current implementation details of <ulink
      type="http"
      url="../../gstreamer/html/GstElement.html"><classname>GstElement</classname></ulink>
      and <ulink type="http"
      url="../../gstreamer/html/GstPlugin.html"><classname>GstPlugin</classname></ulink>.
    </para>
  </sect1>

  <!-- ############ sect1 ############# -->

  <sect1 id="section-basics-pads" xreflabel="Pads">
    <title>Pads</title>
    <para>
      <emphasis>Pads</emphasis> are used to negotiate links and data flow
      between elements in &GStreamer;. A pad can be viewed as a
      <quote>place</quote> or <quote>port</quote> on an element where
      links may be made with other elements, and through which data can
      flow to or from those elements.  Pads have specific data handling
      capabilities: A pad can restrict the type of data that flows
      through it.  Links are only allowed between two pads when the
      allowed data types of the two pads are compatible.
    </para>
    <para>
      An analogy may be helpful here. A pad is similar to a plug or jack on a
      physical device. Consider, for example, a home theater system consisting
      of an amplifier, a DVD player, and a (silent) video projector. Linking
      the DVD player to the amplifier is allowed because both devices have audio
      jacks, and linking the projector to the DVD player is allowed because
      both devices have compatible video jacks. Links between the
      projector and the amplifier may not be made because the projector and
      amplifier have different types of jacks. Pads in &GStreamer; serve the
      same purpose as the jacks in the home theater system.
    </para>
    <para>
      For the most part, all data in &GStreamer; flows one way through a link
      between elements. Data flows out of one element through one or more
      <emphasis>source pads</emphasis>, and elements accept incoming data through
      one or more <emphasis>sink pads</emphasis>. Source and sink elements have
      only source and sink pads, respectively.
    </para>
    <para>
      See the &GstLibRef; for the current implementation details of a <ulink
      type="http"
      url="../../gstreamer/html/GstPad.html"><classname>GstPad</classname></ulink>.
    </para>
  </sect1>

  <!-- ############ sect1 ############# -->

  <sect1 id="section-basics-data" xreflabel="Data, Buffers and Events">
    <title>GstMiniObject, Buffers and Events</title>
    <para>
      All streams of data in &GStreamer; are chopped up into chunks that are
      passed from a source pad on one element to a sink pad on another element.
      <emphasis>GstMiniObject</emphasis> is the structure used to hold these
      chunks of data.
    </para>
    <para>
      GstMiniObject contains the following important types:
      <itemizedlist>
        <listitem>
          <para>
            An exact type indicating what type of data (event, buffer, ...)
            this GstMiniObject is.
          </para>
        </listitem>
        <listitem>
          <para>
            A reference count indicating the number of elements currently
            holding a reference to the miniobject. When the reference count
            falls to zero, the miniobject will be disposed, and its memory will be
            freed in some sense (see below for more details).
          </para>
        </listitem>
      </itemizedlist>
    </para>
    <para>
      For data transport, there are two types of GstMiniObject defined:
      events (control) and buffers (content).
    </para>
    <para>
      Buffers may contain any sort of data that the two linked pads
      know how to handle. Normally, a buffer contains a chunk of some sort of
      audio or video data that flows from one element to another.
    </para>
    <para>
      Buffers also contain metadata describing the buffer's contents. Some of
      the important types of metadata are:
      <itemizedlist>
        <listitem>
          <para>
            Pointers to one or more GstMemory objects. GstMemory objects are
            refcounted objects that encapsulate a region of memory.
          </para>
        </listitem>
        <listitem>
          <para>
            A timestamp indicating the preferred display timestamp of the
            content in the buffer.
          </para>
        </listitem>
      </itemizedlist>
    </para>
    <para>
      Events
      contain information on the state of the stream flowing between the two
      linked pads. Events will only be sent if the element explicitly supports
      them, else the core will (try to) handle the events automatically. Events
      are used to indicate, for example, a media type, the end of a
      media stream or that the cache should be flushed.
    </para>
    <para>
      Events may contain several of the following items:
      <itemizedlist>
        <listitem>
          <para>
            A subtype indicating the type of the contained event.
          </para>
        </listitem>
        <listitem>
          <para>
            The other contents of the event depend on the specific event type.
          </para>
        </listitem>
      </itemizedlist>
    </para>
    <para>
      Events will be discussed extensively in <xref linkend="chapter-advanced-events"/>.
      Until then, the only event that will be used is the <emphasis>EOS</emphasis>
      event, which is used to indicate the end-of-stream (usually end-of-file).
    </para>
    <para>
      See the &GstLibRef; for the current implementation details of a <ulink
      type="http"
      url="../../gstreamer/html/gstreamer-GstMiniObject.html"><classname>GstMiniObject</classname></ulink>, <ulink type="http"
      url="../../gstreamer/html/GstBuffer.html"><classname>GstBuffer</classname></ulink> and <ulink type="http"
      url="../../gstreamer/html/GstEvent.html"><classname>GstEvent</classname></ulink>.
    </para>

    <sect2 id="sect2-buffer-allocation" xreflabel="Buffer Allocation">
      <title>Buffer Allocation</title>
      <para>
        Buffers are able to store chunks of memory of several different
	types.  The most generic type of buffer contains memory allocated
	by malloc().  Such buffers, although convenient, are not always
	very fast, since data often needs to be specifically copied into
	the buffer.
      </para>
      <para>
	Many specialized elements create buffers that point to special
	memory.  For example, the filesrc element usually
	maps a file into the address space of the application (using mmap()),
	and creates buffers that point into that address range.  These
	buffers created by filesrc act exactly like generic buffers, except
	that they are read-only.  The buffer freeing code automatically
	determines the correct method of freeing the underlying memory.
	Downstream elements that receive these kinds of buffers do not
	need to do anything special to handle or unreference it.
      </para>
      <para>
        Another way an element might get specialized buffers is to
        request them from a downstream peer through a GstBufferPool or
        GstAllocator.  Elements can ask a GstBufferPool or GstAllocator
        from the downstream peer element. If downstream is able to provide
        these objects, upstream can use them to allocate buffers.
        See more in <xref linkend="chapter-allocation"/>.
      </para>
      <para>
        Many sink elements have accelerated methods for copying data
	to hardware, or have direct access to hardware.  It is common
        for these elements to be able to create a GstBufferPool or 
        GstAllocator for their upstream peers.  One such example is
	ximagesink.  It creates buffers that contain XImages.  Thus,
	when an upstream peer copies data into the buffer, it is copying
	directly into the XImage, enabling ximagesink to draw the
	image directly to the screen instead of having to copy data
	into an XImage first.
      </para>
      <para>
        Filter elements often have the opportunity to either work on
	a buffer in-place, or work while copying from a source buffer
	to a destination buffer.  It is optimal to implement both
	algorithms, since the &GStreamer; framework can choose the
	fastest algorithm as appropriate.  Naturally, this only makes
	sense for strict filters -- elements that have exactly the
	same format on source and sink pads.
      </para>
    </sect2>
  </sect1>

  <!-- ############ sect1 ############# -->

  <sect1 id="section-basics-types" xreflabel="Types and Properties">
    <title>Media types and Properties</title>
    <para>
      &GStreamer; uses a type system to ensure that the data passed between
      elements is in a recognized format. The type system is also important
      for ensuring that the parameters required to fully specify a format match
      up correctly when linking pads between elements. Each link that is
      made between elements has a specified type and optionally a set of
      properties. See more about caps negotiation in
      <xref linkend="chapter-negotiation"/>.
    </para>

    <!-- ############ sect2 ############# -->

    <sect2 id="sect2-types-basictypes" xreflabel="Basic Types">
      <title>The Basic Types</title>
      <para>
        &GStreamer; already supports many basic media types. Following is a
        table of a few of the basic types used for buffers in
        &GStreamer;. The table contains the name ("media type") and a
        description of the type, the properties associated with the type, and
        the meaning of each property. A full list of supported types is
        included in <xref linkend="section-types-definitions"/>.
      </para>

      <table frame="all" id="table-basictypes" xreflabel="Table of Example Types">
        <title>Table of Example Types</title>
        <tgroup cols="6" align="left" colsep="1" rowsep="1">

        <thead>
          <row>
            <entry>Media Type</entry>
            <entry>Description</entry>
            <entry>Property</entry>
            <entry>Property Type</entry>
            <entry>Property Values</entry>
            <entry>Property Description</entry>
          </row>
        </thead>

        <tbody valign="top">

          <!-- ############ type ############# -->

          <row>
            <entry morerows="1">audio/*</entry>
            <entry morerows="1">
              <emphasis>All audio types</emphasis>
            </entry>
            <entry>rate</entry>
            <entry>integer</entry>
            <entry>greater than 0</entry>
            <entry>
              The sample rate of the data, in samples (per channel) per second.
            </entry>
          </row>
          <row>
            <entry>channels</entry>
            <entry>integer</entry>
            <entry>greater than 0</entry>
            <entry>
              The number of channels of audio data.
            </entry>
          </row>

          <!-- ############ type ############# -->

          <row>
            <entry>audio/x-raw</entry>
            <entry>
              Unstructured and uncompressed raw integer audio data.
            </entry>
            <entry>format</entry>
            <entry>string</entry>
            <entry>
              S8 U8 S16LE S16BE U16LE U16BE S24_32LE S24_32BE U24_32LE U24_32BE S32LE S32BE U32LE U32BE
              S24LE S24BE U24LE U24BE S20LE S20BE U20LE U20BE S18LE S18BE U18LE U18BE F32LE F32BE F64LE F64BE
            </entry>
            <entry>
              The format of the sample data.
            </entry>
          </row>

          <!-- ############ type ############# -->

          <row>
            <entry morerows="3">audio/mpeg</entry>
            <entry morerows="3">
              Audio data compressed using the MPEG audio encoding scheme.
            </entry>
            <entry>mpegversion</entry>
            <entry>integer</entry>
            <entry>1, 2 or 4</entry>
            <entry>
              The MPEG-version used for encoding the data. The value 1 refers
              to MPEG-1, -2 and -2.5 layer 1, 2 or 3. The values 2 and 4 refer
              to the MPEG-AAC audio encoding schemes.
            </entry>
          </row>
          <row>
            <entry>framed</entry>
            <entry>boolean</entry>
            <entry>0 or 1</entry>
            <entry>
              A true value indicates that each buffer contains exactly one
              frame. A false value indicates that frames and buffers do not
              necessarily match up.
            </entry>
          </row>
          <row>
            <entry>layer</entry>
            <entry>integer</entry>
            <entry>1, 2, or 3</entry>
            <entry>
              The compression scheme layer used to compress the data
              <emphasis>(only if mpegversion=1)</emphasis>.
            </entry>
          </row>
          <row>
            <entry>bitrate</entry>
            <entry>integer</entry>
            <entry>greater than 0</entry>
            <entry>
              The bitrate, in bits per second. For VBR (variable bitrate)
              MPEG data, this is the average bitrate.
            </entry>
          </row>

          <!-- ############ type ############# -->

          <row>
            <entry>audio/x-vorbis</entry>
            <entry>Vorbis audio data</entry>
            <entry></entry>
            <entry></entry>
            <entry></entry>
            <entry>
              There are currently no specific properties defined for this type.
            </entry>
          </row>
        </tbody>
      </tgroup>
      </table>
    </sect2>
  </sect1>
</chapter>