mirror of
https://gitlab.freedesktop.org/gstreamer/gstreamer.git
synced 2024-11-15 22:01:27 +00:00
501 lines
20 KiB
XML
501 lines
20 KiB
XML
<!-- ############ chapter ############# -->
|
|
|
|
<chapter id="cha-intro-basics" xreflabel="Basic Concepts">
|
|
<title>Basic Concepts</title>
|
|
<para>
|
|
This chapter of the guide introduces the basic concepts of &GStreamer;.
|
|
Understanding these concepts will help you grok the issues involved in
|
|
extending &GStreamer;. Many of these concepts are explained in greater
|
|
detail in the &GstAppDevMan;; the basic concepts presented here serve mainly
|
|
to refresh your memory.
|
|
</para>
|
|
|
|
<!-- ############ sect1 ############# -->
|
|
|
|
<sect1 id="sect1-basics-elements" xreflabel="Elements and Plugins">
|
|
<title>Elements and Plugins</title>
|
|
<para>
|
|
Elements are at the core of &GStreamer;. In the context of plugin
|
|
development, an <emphasis>element</emphasis> is an object derived from the
|
|
<classname>GstElement</classname> class. Elements provide some sort of
|
|
functionality when linked with other elements: For example, a source
|
|
element provides data to a stream, and a filter element acts on the data
|
|
in a stream. Without elements, &GStreamer; is just a bunch of conceptual
|
|
pipe fittings with nothing to link. A large number of elements ship
|
|
with &GStreamer;, but extra elements can also be written.
|
|
</para>
|
|
<para>
|
|
Just writing a new element is not entirely enough, however: You will need
|
|
to encapsulate your element in a <emphasis>plugin</emphasis> to enable
|
|
&GStreamer; to use it. A plugin is essentially a loadable block of code,
|
|
usually called a shared object file or a dynamically linked library. A
|
|
single plugin may contain the implementation of several elements, or just
|
|
a single one. For simplicity, this guide concentrates primarily on plugins
|
|
containing one element.
|
|
</para>
|
|
<para>
|
|
A <emphasis>filter</emphasis> is an important type of element that
|
|
processes a stream of data. Producers and consumers of data are called
|
|
<emphasis>source</emphasis> and <emphasis>sink</emphasis> elements,
|
|
respectively. <emphasis>Bin</emphasis> elements contain other elements.
|
|
One type of bin is responsible for scheduling the elements that they
|
|
contain so that data flows smoothly. Another type of bin, called
|
|
<emphasis>autoplugger</emphasis> elements, automatically add other
|
|
elements to the bin and link them together so that they act as a
|
|
filter between two arbitary stream types.
|
|
</para>
|
|
<para>
|
|
The plugin mechanism is used everywhere in &GStreamer;, even if only the
|
|
standard packages are being used. A few very basic functions reside in the
|
|
core library, and all others are implemented in plugins. A plugin registry
|
|
is used to store the details of the plugins in an XML file. This way, a
|
|
program using &GStreamer; does not have to load all plugins to determine
|
|
which are needed. Plugins are only loaded when their provided elements are
|
|
requested.
|
|
</para>
|
|
<para>
|
|
See the &GstLibRef; for the current implementation details of <ulink
|
|
type="http"
|
|
url="../gstreamer/gstelement.html"><classname>GstElement</classname></ulink>
|
|
and <ulink type="http"
|
|
url="../gstreamer/gstreamer-gstplugin.html"><classname>GstPlugin</classname></ulink>.
|
|
</para>
|
|
</sect1>
|
|
|
|
<!-- ############ sect1 ############# -->
|
|
|
|
<sect1 id="sect1-basics-pads" xreflabel="Pads">
|
|
<title>Pads</title>
|
|
<para>
|
|
<emphasis>Pads</emphasis> are used to negotiate links and data flow
|
|
between elements in &GStreamer;. A pad can be viewed as a
|
|
<quote>place</quote> or <quote>port</quote> on an element where
|
|
links may be made with other elements, and through which data can
|
|
flow to or from those elements. Pads have specific data handling
|
|
capabilities: A pad can restrict the type of data that flows
|
|
through it. Links are only allowed between two pads when the
|
|
allowed data types of the two pads are compatible.
|
|
</para>
|
|
<para>
|
|
An analogy may be helpful here. A pad is similar to a plug or jack on a
|
|
physical device. Consider, for example, a home theater system consisting
|
|
of an amplifier, a DVD player, and a (silent) video projector. Linking
|
|
the DVD player to the amplifier is allowed because both devices have audio
|
|
jacks, and linking the projector to the DVD player is allowed because
|
|
both devices have compatible video jacks. Links between the
|
|
projector and the amplifier may not be made because the projector and
|
|
amplifier have different types of jacks. Pads in &GStreamer; serve the
|
|
same purpose as the jacks in the home theater system.
|
|
</para>
|
|
<para>
|
|
For the most part, all data in &GStreamer; flows one way through a link
|
|
between elements. Data flows out of one element through one or more
|
|
<emphasis>source pads</emphasis>, and elements accept incoming data through
|
|
one or more <emphasis>sink pads</emphasis>. Source and sink elements have
|
|
only source and sink pads, respectively.
|
|
</para>
|
|
<para>
|
|
See the &GstLibRef; for the current implementation details of a <ulink
|
|
type="http"
|
|
url="../gstreamer/gstreamer-gstpad.html"><classname>GstPad</classname></ulink>.
|
|
</para>
|
|
</sect1>
|
|
|
|
<!-- ############ sect1 ############# -->
|
|
|
|
<sect1 id="sect1-basics-buffers" xreflabel="Buffers">
|
|
<title>Buffers</title>
|
|
<para>
|
|
All streams of data in &GStreamer; are chopped up into chunks that are
|
|
passed from a source pad on one element to a sink pad on another element.
|
|
<emphasis>Buffers</emphasis> are structures used to hold these chunks of
|
|
data. Buffers can be of any size, theoretically, and they may contain any
|
|
sort of data that the two linked pads know how to handle. Normally, a
|
|
buffer contains a chunk of some sort of audio or video data that flows
|
|
from one element to another.
|
|
</para>
|
|
<para>
|
|
Buffers also contain metadata describing the buffer's contents. Some of
|
|
the important types of metadata are:
|
|
<itemizedlist>
|
|
<listitem>
|
|
<para>
|
|
A pointer to the buffer's data.
|
|
</para>
|
|
</listitem>
|
|
<listitem>
|
|
<para>
|
|
An integer indicating the size of the buffer's data.
|
|
</para>
|
|
</listitem>
|
|
<listitem>
|
|
<para>
|
|
A <classname>GstData</classname> object describing the type of the
|
|
buffer's data.
|
|
</para>
|
|
</listitem>
|
|
<listitem>
|
|
<para>
|
|
A reference count indicating the number of elements currently
|
|
holding a reference to the buffer. When the buffer reference count
|
|
falls to zero, the buffer will be unlinked, and its memory will be
|
|
freed in some sense (see below for more details).
|
|
</para>
|
|
</listitem>
|
|
</itemizedlist>
|
|
</para>
|
|
<para>
|
|
See the &GstLibRef; for the current implementation details of a <ulink
|
|
type="http"
|
|
url="../gstreamer/gstreamer-gstbuffer.html"><classname>GstBuffer</classname></ulink>.
|
|
</para>
|
|
|
|
<sect2 id="sect2-buffers-bufferpools" xreflabel="Buffer Allocation and
|
|
Buffer Pools">
|
|
<title>Buffer Allocation and Buffer Pools</title>
|
|
<para>
|
|
Buffers can be allocated using various schemes, and they may either be
|
|
passed on by an element or unreferenced, thus freeing the memory used by
|
|
the buffer. Buffer allocation and unlinking are important concepts when
|
|
dealing with real time media processing, since memory allocation is
|
|
relatively slow on most systems.
|
|
</para>
|
|
<para>
|
|
To improve the latency in a media pipeline, many &GStreamer; elements
|
|
use a <emphasis>buffer pool</emphasis> to handle buffer allocation and
|
|
unlinking. A buffer pool is a relatively large chunk of memory that is
|
|
the &GStreamer; process requests early on from the operating system.
|
|
Later, when elements request memory for a new buffer, the buffer pool
|
|
can serve the request quickly by giving out a piece of the allocated
|
|
memory. This saves a call to the operating system and lowers latency.
|
|
[If it seems at this point like &GStreamer; is acting like an operating
|
|
system (doing memory management, etc.), don't worry: &GStreamer;OS isn't
|
|
due out for quite a few years!]
|
|
</para>
|
|
<para>
|
|
Normally in a media pipeline, most filter elements in &GStreamer; deal
|
|
with a buffer in place, meaning that they do not create or destroy
|
|
buffers. Sometimes, however, elements might need to alter the reference
|
|
count of a buffer, either by copying or destroying the buffer, or by
|
|
creating a new buffer. These topics are generally reserved for
|
|
non-filter elements, so they will be addressed at that point.
|
|
</para>
|
|
</sect2>
|
|
</sect1>
|
|
|
|
<!-- ############ sect1 ############# -->
|
|
|
|
<sect1 id="sect1-basics-types" xreflabel="Types and Properties">
|
|
<title>Types and Properties</title>
|
|
<para>
|
|
&GStreamer; uses a type system to ensure that the data passed between
|
|
elements is in a recognized format. The type system is also important for
|
|
ensuring that the parameters required to fully specify a format match up
|
|
correctly when linking pads between elements. Each link that is
|
|
made between elements has a specified type.
|
|
</para>
|
|
|
|
<!-- ############ sect2 ############# -->
|
|
|
|
<sect2 id="sect2-types-basictypes" xreflabel="Basic Types">
|
|
<title>The Basic Types</title>
|
|
<para>
|
|
&GStreamer; already supports many basic media types. Following is a
|
|
table of the basic types used for buffers in &GStreamer;. The table
|
|
contains the name ("mime type") and a description of the type, the
|
|
properties associated with the type, and the meaning of each property.
|
|
</para>
|
|
|
|
<table frame="all" id="table-basictypes" xreflabel="Table of Basic Types">
|
|
<title>Table of Basic Types</title>
|
|
<tgroup cols="6" align="left" colsep="1" rowsep="1">
|
|
|
|
<thead>
|
|
<row>
|
|
<entry>Mime Type</entry>
|
|
<entry>Description</entry>
|
|
<entry>Property</entry>
|
|
<entry>Property Type</entry>
|
|
<entry>Property Values</entry>
|
|
<entry>Property Description</entry>
|
|
</row>
|
|
</thead>
|
|
|
|
<tbody valign="top">
|
|
|
|
<!-- ############ type ############# -->
|
|
|
|
<row>
|
|
<entry morerows="10">audio/raw</entry>
|
|
<entry morerows="10">
|
|
Unstructured and uncompressed raw audio data.
|
|
</entry>
|
|
<entry>rate</entry>
|
|
<entry>integer</entry>
|
|
<entry>greater than 0</entry>
|
|
<entry>
|
|
The sample rate of the data, in samples per second.
|
|
</entry>
|
|
</row>
|
|
<row>
|
|
<entry>channels</entry>
|
|
<entry>integer</entry>
|
|
<entry>greater than 0</entry>
|
|
<entry>
|
|
The number of channels of audio data.
|
|
</entry>
|
|
</row>
|
|
<row>
|
|
<entry>format</entry>
|
|
<entry>string</entry>
|
|
<entry><quote>int</quote> or <quote>float</quote></entry>
|
|
<entry>
|
|
The format in which the audio data is passed.
|
|
</entry>
|
|
</row>
|
|
<row>
|
|
<entry>law</entry>
|
|
<entry>integer</entry>
|
|
<entry>0, 1, or 2</entry>
|
|
<entry>
|
|
(Valid only if the data is in integer format.) The law used to
|
|
describe the data. The value 0 indicates <quote>linear</quote>, 1
|
|
indicates <quote>mu law</quote>, and 2 indicates
|
|
<quote>A law</quote>.
|
|
</entry>
|
|
</row>
|
|
<row>
|
|
<entry>endianness</entry>
|
|
<entry>boolean</entry>
|
|
<entry>0 or 1</entry>
|
|
<entry>
|
|
(Valid only if the data is in integer format.) The order of bytes
|
|
in a sample. The value 0 means <quote>little-endian</quote> (bytes
|
|
are least significant first). The value 1 means
|
|
<quote>big-endian</quote> (most significant byte first).
|
|
</entry>
|
|
</row>
|
|
<row>
|
|
<entry>signed</entry>
|
|
<entry>boolean</entry>
|
|
<entry>0 or 1</entry>
|
|
<entry>
|
|
(Valid only if the data is in integer format.) Whether the samples
|
|
are signed or not.
|
|
</entry>
|
|
</row>
|
|
<row>
|
|
<entry>width</entry>
|
|
<entry>integer</entry>
|
|
<entry>greater than 0</entry>
|
|
<entry>
|
|
(Valid only if the data is in integer format.) The number of bits
|
|
per sample.
|
|
</entry>
|
|
</row>
|
|
<row>
|
|
<entry>depth</entry>
|
|
<entry>integer</entry>
|
|
<entry>greater than 0</entry>
|
|
<entry>
|
|
(Valid only if the data is in integer format.) The number of bits
|
|
used per sample. This must be less than or equal to the width: If
|
|
the depth is less than the width, the low bits are assumed to be
|
|
the ones used. For example, a width of 32 and a depth of 24 means
|
|
that each sample is stored in a 32 bit word, but only the low 24
|
|
bits are actually used.
|
|
</entry>
|
|
</row>
|
|
<row>
|
|
<entry>layout</entry>
|
|
<entry>string</entry>
|
|
<entry><quote>gfloat</quote></entry>
|
|
<entry>
|
|
(Valid only if the data is in float format.) A string representing
|
|
the way in which the floating point data is represented.
|
|
</entry>
|
|
</row>
|
|
<row>
|
|
<entry>intercept</entry>
|
|
<entry>float</entry>
|
|
<entry>any, normally 0</entry>
|
|
<entry>
|
|
(Valid only if the data is in float format.) A floating point
|
|
value representing the value that the signal
|
|
<quote>centers</quote> on.
|
|
</entry>
|
|
</row>
|
|
<row>
|
|
<entry>slope</entry>
|
|
<entry>float</entry>
|
|
<entry>any, normally 1.0</entry>
|
|
<entry>
|
|
(Valid only if the data is in float format.) A floating point
|
|
value representing how far the signal deviates from the intercept.
|
|
A slope of 1.0 and an intercept of 0.0 would mean an audio signal
|
|
with minimum and maximum values of -1.0 and 1.0. A slope of
|
|
0.5 and intercept of 0.5 would represent values in the range 0.0
|
|
to 1.0.
|
|
</entry>
|
|
</row>
|
|
|
|
<!-- ############ type ############# -->
|
|
|
|
<row>
|
|
<entry morerows="4">audio/mp3</entry>
|
|
<entry morerows="4">
|
|
Audio data compressed using the mp3 encoding scheme.
|
|
</entry>
|
|
<entry>framed</entry>
|
|
<entry>boolean</entry>
|
|
<entry>0 or 1</entry>
|
|
<entry>
|
|
A true value indicates that each buffer contains exactly one
|
|
frame. A false value indicates that frames and buffers do not
|
|
necessarily match up.
|
|
</entry>
|
|
</row>
|
|
<row>
|
|
<entry>layer</entry>
|
|
<entry>integer</entry>
|
|
<entry>1, 2, or 3</entry>
|
|
<entry>
|
|
The compression scheme layer used to compress the data.
|
|
</entry>
|
|
</row>
|
|
<row>
|
|
<entry>bitrate</entry>
|
|
<entry>integer</entry>
|
|
<entry>greater than 0</entry>
|
|
<entry>
|
|
The bitrate, in kilobits per second. For VBR (variable bitrate)
|
|
mp3 data, this is the average bitrate.
|
|
</entry>
|
|
</row>
|
|
<row>
|
|
<entry>channels</entry>
|
|
<entry>integer</entry>
|
|
<entry>greater than 0</entry>
|
|
<entry>
|
|
The number of channels of audio data present.
|
|
</entry>
|
|
</row>
|
|
<row>
|
|
<entry>joint-stereo</entry>
|
|
<entry>boolean</entry>
|
|
<entry>0 or 1</entry>
|
|
<entry>
|
|
If true, this implies that stereo data is stored as a combined
|
|
signal and the difference between the signals, rather than as two
|
|
entirely separate signals. If true, the <quote>channels</quote>
|
|
attribute must not be zero.
|
|
</entry>
|
|
</row>
|
|
|
|
<!-- ############ type ############# -->
|
|
|
|
<row>
|
|
<entry morerows="0">audio/x-ogg</entry>
|
|
<entry morerows="0">
|
|
Audio data compressed using the Ogg Vorbis encoding scheme.
|
|
</entry>
|
|
<entry></entry>
|
|
<entry></entry>
|
|
<entry></entry>
|
|
<entry>
|
|
FIXME: There are currently no parameters defined for this type.
|
|
</entry>
|
|
</row>
|
|
|
|
<!-- ############ type ############# -->
|
|
|
|
<row>
|
|
<entry morerows="2">video/raw</entry>
|
|
<entry morerows="2">
|
|
Raw video data.
|
|
</entry>
|
|
<entry>fourcc</entry>
|
|
<entry>FOURCC code</entry>
|
|
<entry></entry>
|
|
<entry>
|
|
A FOURCC code identifying the format in which this data is stored.
|
|
FOURCC (Four Character Code) is a simple system to allow
|
|
unambiguous identification of a video datastream format. See
|
|
<ulink url="http://www.webartz.com/fourcc/"
|
|
type="http">http://www.webartz.com/fourcc/</ulink>
|
|
</entry>
|
|
</row>
|
|
<row>
|
|
<entry>width</entry>
|
|
<entry>integer</entry>
|
|
<entry>greater than 0</entry>
|
|
<entry>
|
|
The number of pixels wide that each video frame is.
|
|
</entry>
|
|
</row>
|
|
<row>
|
|
<entry>height</entry>
|
|
<entry>integer</entry>
|
|
<entry>greater than 0</entry>
|
|
<entry>
|
|
The number of pixels high that each video frame is.
|
|
</entry>
|
|
</row>
|
|
|
|
<!-- ############ type ############# -->
|
|
|
|
<row>
|
|
<entry morerows="0">video/mpeg</entry>
|
|
<entry morerows="0">
|
|
Video data compressed using an MPEG encoding scheme.
|
|
</entry>
|
|
<entry></entry>
|
|
<entry></entry>
|
|
<entry></entry>
|
|
<entry>
|
|
FIXME: There are currently no parameters defined for this type.
|
|
</entry>
|
|
</row>
|
|
|
|
<!-- ############ type ############# -->
|
|
|
|
<row>
|
|
<entry morerows="0">video/x-msvideo</entry>
|
|
<entry morerows="0">
|
|
Video data compressed using the AVI encoding scheme.
|
|
</entry>
|
|
<entry></entry>
|
|
<entry></entry>
|
|
<entry></entry>
|
|
<entry>
|
|
FIXME: There are currently no parameters defined for this type.
|
|
</entry>
|
|
</row>
|
|
|
|
</tbody>
|
|
</tgroup>
|
|
</table>
|
|
</sect2>
|
|
</sect1>
|
|
|
|
<!-- ############ sect1 ############# -->
|
|
|
|
<sect1 id="sect1-basics-events" xreflabel="Events">
|
|
<title>Events</title>
|
|
<para>
|
|
Sometimes elements in a media processing pipeline need to know that
|
|
something has happened. An <emphasis>event</emphasis> is a special type of
|
|
data in &GStreamer; designed to serve this purpose. Events describe some
|
|
sort of activity that has happened somewhere in an element's pipeline, for
|
|
example, the end of the media stream or a clock discontinuity. Just like
|
|
any other data type, an event comes to an element on a sink pad and is
|
|
contained in a normal buffer. Unlike normal stream buffers, though, an
|
|
event buffer contains only an event, not any media stream data.
|
|
</para>
|
|
<para>
|
|
See the &GstLibRef; for the current implementation details of a <ulink
|
|
type="http"
|
|
url="../gstreamer/gstreamer-gstevent.html"><classname>GstEvent</classname></ulink>.
|
|
</para>
|
|
</sect1>
|
|
</chapter>
|