mirror of
https://gitlab.freedesktop.org/gstreamer/gstreamer.git
synced 2024-12-20 23:36:38 +00:00
433 lines
18 KiB
XML
433 lines
18 KiB
XML
<!-- ############ chapter ############# -->
|
|
|
|
<chapter id="chapter-intro-basics" xreflabel="Foundations">
|
|
<title>Foundations</title>
|
|
<para><!-- synchronize with AppDevMan -->
|
|
This chapter of the guide introduces the basic concepts of &GStreamer;.
|
|
Understanding these concepts will help you grok the issues involved in
|
|
extending &GStreamer;. Many of these concepts are explained in greater
|
|
detail in the &GstAppDevMan;; the basic concepts presented here serve mainly
|
|
to refresh your memory.
|
|
</para>
|
|
|
|
<!-- ############ sect1 ############# -->
|
|
|
|
<sect1 id="section-basics-elements" xreflabel="Elements and Plugins">
|
|
<title>Elements and Plugins</title>
|
|
<para>
|
|
Elements are at the core of &GStreamer;. In the context of plugin
|
|
development, an <emphasis>element</emphasis> is an object derived from the
|
|
<ulink type="http" url="../../gstreamer/html/GstElement.html"><classname>
|
|
GstElement</classname></ulink> class. Elements provide some sort of
|
|
functionality when linked with other elements: For example, a source
|
|
element provides data to a stream, and a filter element acts on the data
|
|
in a stream. Without elements, &GStreamer; is just a bunch of conceptual
|
|
pipe fittings with nothing to link. A large number of elements ship
|
|
with &GStreamer;, but extra elements can also be written.
|
|
</para>
|
|
<para>
|
|
Just writing a new element is not entirely enough, however: You will need
|
|
to encapsulate your element in a <emphasis>plugin</emphasis> to enable
|
|
&GStreamer; to use it. A plugin is essentially a loadable block of code,
|
|
usually called a shared object file or a dynamically linked library. A
|
|
single plugin may contain the implementation of several elements, or just
|
|
a single one. For simplicity, this guide concentrates primarily on plugins
|
|
containing one element.
|
|
</para>
|
|
<para>
|
|
A <emphasis>filter</emphasis> is an important type of element that
|
|
processes a stream of data. Producers and consumers of data are called
|
|
<emphasis>source</emphasis> and <emphasis>sink</emphasis> elements,
|
|
respectively. <emphasis>Bin</emphasis> elements contain other elements.
|
|
One type of bin is responsible for scheduling the elements that they
|
|
contain so that data flows smoothly. Another type of bin, called
|
|
<emphasis>autoplugger</emphasis> elements, automatically add other
|
|
elements to the bin and links them together so that they act as a
|
|
filter between two arbitrary stream types.
|
|
</para>
|
|
<para>
|
|
The plugin mechanism is used everywhere in &GStreamer;, even if only the
|
|
standard packages are being used. A few very basic functions reside in the
|
|
core library, and all others are implemented in plugins. A plugin registry
|
|
is used to store the details of the plugins in an XML file. This way, a
|
|
program using &GStreamer; does not have to load all plugins to determine
|
|
which are needed. Plugins are only loaded when their provided elements are
|
|
requested.
|
|
</para>
|
|
<para>
|
|
See the &GstLibRef; for the current implementation details of <ulink
|
|
type="http"
|
|
url="../../gstreamer/html/GstElement.html"><classname>GstElement</classname></ulink>
|
|
and <ulink type="http"
|
|
url="../../gstreamer/html/GstPlugin.html"><classname>GstPlugin</classname></ulink>.
|
|
</para>
|
|
</sect1>
|
|
|
|
<!-- ############ sect1 ############# -->
|
|
|
|
<sect1 id="section-basics-pads" xreflabel="Pads">
|
|
<title>Pads</title>
|
|
<para>
|
|
<emphasis>Pads</emphasis> are used to negotiate links and data flow
|
|
between elements in &GStreamer;. A pad can be viewed as a
|
|
<quote>place</quote> or <quote>port</quote> on an element where
|
|
links may be made with other elements, and through which data can
|
|
flow to or from those elements. Pads have specific data handling
|
|
capabilities: A pad can restrict the type of data that flows
|
|
through it. Links are only allowed between two pads when the
|
|
allowed data types of the two pads are compatible.
|
|
</para>
|
|
<para>
|
|
An analogy may be helpful here. A pad is similar to a plug or jack on a
|
|
physical device. Consider, for example, a home theater system consisting
|
|
of an amplifier, a DVD player, and a (silent) video projector. Linking
|
|
the DVD player to the amplifier is allowed because both devices have audio
|
|
jacks, and linking the projector to the DVD player is allowed because
|
|
both devices have compatible video jacks. Links between the
|
|
projector and the amplifier may not be made because the projector and
|
|
amplifier have different types of jacks. Pads in &GStreamer; serve the
|
|
same purpose as the jacks in the home theater system.
|
|
</para>
|
|
<para>
|
|
For the most part, all data in &GStreamer; flows one way through a link
|
|
between elements. Data flows out of one element through one or more
|
|
<emphasis>source pads</emphasis>, and elements accept incoming data through
|
|
one or more <emphasis>sink pads</emphasis>. Source and sink elements have
|
|
only source and sink pads, respectively.
|
|
</para>
|
|
<para>
|
|
See the &GstLibRef; for the current implementation details of a <ulink
|
|
type="http"
|
|
url="../../gstreamer/html/GstPad.html"><classname>GstPad</classname></ulink>.
|
|
</para>
|
|
</sect1>
|
|
|
|
<!-- ############ sect1 ############# -->
|
|
|
|
<sect1 id="section-basics-data" xreflabel="Data, Buffers and Events">
|
|
<title>Data, Buffers and Events</title>
|
|
<para>
|
|
All streams of data in &GStreamer; are chopped up into chunks that are
|
|
passed from a source pad on one element to a sink pad on another element.
|
|
<emphasis>Data</emphasis> are structures used to hold these chunks of
|
|
data.
|
|
</para>
|
|
<para>
|
|
Data contains the following important types:
|
|
<itemizedlist>
|
|
<listitem>
|
|
<para>
|
|
An exact type indicating what type of data (control, content, ...)
|
|
this Data is.
|
|
</para>
|
|
</listitem>
|
|
<listitem>
|
|
<para>
|
|
A reference count indicating the number of elements currently
|
|
holding a reference to the buffer. When the buffer reference count
|
|
falls to zero, the buffer will be unlinked, and its memory will be
|
|
freed in some sense (see below for more details).
|
|
</para>
|
|
</listitem>
|
|
</itemizedlist>
|
|
</para>
|
|
<para>
|
|
There are two types of data defined: events (control) and buffers
|
|
(content).
|
|
</para>
|
|
<para>
|
|
Buffers may contain any sort of data that the two linked pads
|
|
know how to handle. Normally, a buffer contains a chunk of some sort of
|
|
audio or video data that flows from one element to another.
|
|
</para>
|
|
<para>
|
|
Buffers also contain metadata describing the buffer's contents. Some of
|
|
the important types of metadata are:
|
|
<itemizedlist>
|
|
<listitem>
|
|
<para>
|
|
A pointer to the buffer's data.
|
|
</para>
|
|
</listitem>
|
|
<listitem>
|
|
<para>
|
|
An integer indicating the size of the buffer's data.
|
|
</para>
|
|
</listitem>
|
|
<listitem>
|
|
<para>
|
|
A timestamp indicating the preferred display timestamp of the
|
|
content in the buffer.
|
|
</para>
|
|
</listitem>
|
|
</itemizedlist>
|
|
</para>
|
|
<para>
|
|
Events
|
|
contain information on the state of the stream flowing between the two
|
|
linked pads. Events will only be sent if the element explicitly supports
|
|
them, else the core will (try to) handle the events automatically. Events
|
|
are used to indicate, for example, a clock discontinuity, the end of a
|
|
media stream or that the cache should be flushed.
|
|
</para>
|
|
<para>
|
|
Events may contain several of the following items:
|
|
<itemizedlist>
|
|
<listitem>
|
|
<para>
|
|
A subtype indicating the type of the contained event.
|
|
</para>
|
|
</listitem>
|
|
<listitem>
|
|
<para>
|
|
The other contents of the event depend on the specific event type.
|
|
</para>
|
|
</listitem>
|
|
</itemizedlist>
|
|
</para>
|
|
<para>
|
|
Events will be discussed extensively in <xref linkend="chapter-advanced-events"/>.
|
|
Until then, the only event that will be used is the <emphasis>EOS</emphasis>
|
|
event, which is used to indicate the end-of-stream (usually end-of-file).
|
|
</para>
|
|
<para>
|
|
See the &GstLibRef; for the current implementation details of a <ulink
|
|
type="http"
|
|
url="../../gstreamer/html/gstreamer-GstMiniObject.html"><classname>GstMiniObject</classname></ulink>, <ulink type="http"
|
|
url="../../gstreamer/html/gstreamer-GstBuffer.html"><classname>GstBuffer</classname></ulink> and <ulink type="http"
|
|
url="../../gstreamer/html/gstreamer-GstEvent.html"><classname>GstEvent</classname></ulink>.
|
|
</para>
|
|
|
|
<sect2 id="sect2-buffer-allocation" xreflabel="Buffer Allocation">
|
|
<title>Buffer Allocation</title>
|
|
<para>
|
|
Buffers are able to store chunks of memory of several different
|
|
types. The most generic type of buffer contains memory allocated
|
|
by malloc(). Such buffers, although convenient, are not always
|
|
very fast, since data often needs to be specifically copied into
|
|
the buffer.
|
|
</para>
|
|
<para>
|
|
Many specialized elements create buffers that point to special
|
|
memory. For example, the filesrc element usually
|
|
maps a file into the address space of the application (using mmap()),
|
|
and creates buffers that point into that address range. These
|
|
buffers created by filesrc act exactly like generic buffers, except
|
|
that they are read-only. The buffer freeing code automatically
|
|
determines the correct method of freeing the underlying memory.
|
|
Downstream elements that receive these kinds of buffers do not
|
|
need to do anything special to handle or unreference it.
|
|
</para>
|
|
<para>
|
|
Another way an element might get specialized buffers is to
|
|
request them from a downstream peer. These are called
|
|
downstream-allocated buffers. Elements can ask a
|
|
peer connected to a source pad to create an empty buffer of
|
|
a given size. If a downstream element is able to create a
|
|
special buffer of the correct size, it will do so. Otherwise
|
|
&GStreamer; will automatically create a generic buffer instead.
|
|
The element that requested the buffer can then copy data into
|
|
the buffer, and push the buffer to the source pad it was
|
|
allocated from.
|
|
</para>
|
|
<para>
|
|
Many sink elements have accelerated methods for copying data
|
|
to hardware, or have direct access to hardware. It is common
|
|
for these elements to be able to create downstream-allocated
|
|
buffers for their upstream peers. One such example is
|
|
ximagesink. It creates buffers that contain XImages. Thus,
|
|
when an upstream peer copies data into the buffer, it is copying
|
|
directly into the XImage, enabling ximagesink to draw the
|
|
image directly to the screen instead of having to copy data
|
|
into an XImage first.
|
|
</para>
|
|
<para>
|
|
Filter elements often have the opportunity to either work on
|
|
a buffer in-place, or work while copying from a source buffer
|
|
to a destination buffer. It is optimal to implement both
|
|
algorithms, since the &GStreamer; framework can choose the
|
|
fastest algorithm as appropriate. Naturally, this only makes
|
|
sense for strict filters -- elements that have exactly the
|
|
same format on source and sink pads.
|
|
</para>
|
|
</sect2>
|
|
</sect1>
|
|
|
|
<!-- ############ sect1 ############# -->
|
|
|
|
<sect1 id="section-basics-types" xreflabel="Types and Properties">
|
|
<title>Mimetypes and Properties</title>
|
|
<para>
|
|
&GStreamer; uses a type system to ensure that the data passed between
|
|
elements is in a recognized format. The type system is also important
|
|
for ensuring that the parameters required to fully specify a format match
|
|
up correctly when linking pads between elements. Each link that is
|
|
made between elements has a specified type and optionally a set of
|
|
properties.
|
|
</para>
|
|
|
|
<!-- ############ sect2 ############# -->
|
|
|
|
<sect2 id="sect2-types-basictypes" xreflabel="Basic Types">
|
|
<title>The Basic Types</title>
|
|
<para>
|
|
&GStreamer; already supports many basic media types. Following is a
|
|
table of a few of the the basic types used for buffers in
|
|
&GStreamer;. The table contains the name ("mime type") and a
|
|
description of the type, the properties associated with the type, and
|
|
the meaning of each property. A full list of supported types is
|
|
included in <xref linkend="section-types-definitions"/>.
|
|
</para>
|
|
|
|
<table frame="all" id="table-basictypes" xreflabel="Table of Example Types">
|
|
<title>Table of Example Types</title>
|
|
<tgroup cols="6" align="left" colsep="1" rowsep="1">
|
|
|
|
<thead>
|
|
<row>
|
|
<entry>Mime Type</entry>
|
|
<entry>Description</entry>
|
|
<entry>Property</entry>
|
|
<entry>Property Type</entry>
|
|
<entry>Property Values</entry>
|
|
<entry>Property Description</entry>
|
|
</row>
|
|
</thead>
|
|
|
|
<tbody valign="top">
|
|
|
|
<!-- ############ type ############# -->
|
|
|
|
<row>
|
|
<entry morerows="1">audio/*</entry>
|
|
<entry morerows="1">
|
|
<emphasis>All audio types</emphasis>
|
|
</entry>
|
|
<entry>rate</entry>
|
|
<entry>integer</entry>
|
|
<entry>greater than 0</entry>
|
|
<entry>
|
|
The sample rate of the data, in samples (per channel) per second.
|
|
</entry>
|
|
</row>
|
|
<row>
|
|
<entry>channels</entry>
|
|
<entry>integer</entry>
|
|
<entry>greater than 0</entry>
|
|
<entry>
|
|
The number of channels of audio data.
|
|
</entry>
|
|
</row>
|
|
|
|
<!-- ############ type ############# -->
|
|
|
|
<row>
|
|
<entry morerows="3">audio/x-raw-int</entry>
|
|
<entry morerows="3">
|
|
Unstructured and uncompressed raw integer audio data.
|
|
</entry>
|
|
<entry>endianness</entry>
|
|
<entry>integer</entry>
|
|
<entry>G_BIG_ENDIAN (4321) or G_LITTLE_ENDIAN (1234)</entry>
|
|
<entry>
|
|
The order of bytes in a sample. The value G_LITTLE_ENDIAN (1234)
|
|
means <quote>little-endian</quote> (byte-order is <quote>least
|
|
significant byte first</quote>). The value G_BIG_ENDIAN (4321)
|
|
means <quote>big-endian</quote> (byte order is <quote>most
|
|
significant byte first</quote>).
|
|
</entry>
|
|
</row>
|
|
<row>
|
|
<entry>signed</entry>
|
|
<entry>boolean</entry>
|
|
<entry>TRUE or FALSE</entry>
|
|
<entry>
|
|
Whether the values of the integer samples are signed or not.
|
|
Signed samples use one bit to indicate sign (negative or
|
|
positive) of the value. Unsigned samples are always positive.
|
|
</entry>
|
|
</row>
|
|
<row>
|
|
<entry>width</entry>
|
|
<entry>integer</entry>
|
|
<entry>greater than 0</entry>
|
|
<entry>
|
|
Number of bits allocated per sample.
|
|
</entry>
|
|
</row>
|
|
<row>
|
|
<entry>depth</entry>
|
|
<entry>integer</entry>
|
|
<entry>greater than 0</entry>
|
|
<entry>
|
|
The number of bits used per sample. This must be less than or
|
|
equal to the width: If the depth is less than the width, the
|
|
low bits are assumed to be the ones used. For example, a width
|
|
of 32 and a depth of 24 means that each sample is stored in a
|
|
32 bit word, but only the low 24 bits are actually used.
|
|
</entry>
|
|
</row>
|
|
|
|
<!-- ############ type ############# -->
|
|
|
|
<row>
|
|
<entry morerows="3">audio/mpeg</entry>
|
|
<entry morerows="3">
|
|
Audio data compressed using the MPEG audio encoding scheme.
|
|
</entry>
|
|
<entry>mpegversion</entry>
|
|
<entry>integer</entry>
|
|
<entry>1, 2 or 4</entry>
|
|
<entry>
|
|
The MPEG-version used for encoding the data. The value 1 refers
|
|
to MPEG-1, -2 and -2.5 layer 1, 2 or 3. The values 2 and 4 refer
|
|
to the MPEG-AAC audio encoding schemes.
|
|
</entry>
|
|
</row>
|
|
<row>
|
|
<entry>framed</entry>
|
|
<entry>boolean</entry>
|
|
<entry>0 or 1</entry>
|
|
<entry>
|
|
A true value indicates that each buffer contains exactly one
|
|
frame. A false value indicates that frames and buffers do not
|
|
necessarily match up.
|
|
</entry>
|
|
</row>
|
|
<row>
|
|
<entry>layer</entry>
|
|
<entry>integer</entry>
|
|
<entry>1, 2, or 3</entry>
|
|
<entry>
|
|
The compression scheme layer used to compress the data
|
|
<emphasis>(only if mpegversion=1)</emphasis>.
|
|
</entry>
|
|
</row>
|
|
<row>
|
|
<entry>bitrate</entry>
|
|
<entry>integer</entry>
|
|
<entry>greater than 0</entry>
|
|
<entry>
|
|
The bitrate, in bits per second. For VBR (variable bitrate)
|
|
MPEG data, this is the average bitrate.
|
|
</entry>
|
|
</row>
|
|
|
|
<!-- ############ type ############# -->
|
|
|
|
<row>
|
|
<entry>audio/x-vorbis</entry>
|
|
<entry>Vorbis audio data</entry>
|
|
<entry></entry>
|
|
<entry></entry>
|
|
<entry></entry>
|
|
<entry>
|
|
There are currently no specific properties defined for this type.
|
|
</entry>
|
|
</row>
|
|
</tbody>
|
|
</tgroup>
|
|
</table>
|
|
</sect2>
|
|
</sect1>
|
|
</chapter>
|