gstreamer/markdown/application-development/advanced/pipeline-manipulation.md

1405 lines
48 KiB
Markdown
Raw Normal View History

2016-06-04 06:55:52 +00:00
---
title: Pipeline manipulation
...
# Pipeline manipulation
This chapter will discuss how you can manipulate your pipeline in
several ways from your application on. Parts of this chapter are very
lowlevel, so be assured that you'll need some programming knowledge and
a good understanding of GStreamer before you start reading this.
Topics that will be discussed here include how you can insert data into
a pipeline from your application, how to read data from a pipeline, how
to manipulate the pipeline's speed, length, starting point and how to
listen to a pipeline's data processing.
## Using probes
2016-06-04 06:55:52 +00:00
Probing is best envisioned as a pad listener. Technically, a probe is
nothing more than a callback that can be attached to a pad. You can
attach a probe using `gst_pad_add_probe ()`. Similarly, one can use the
`gst_pad_remove_probe ()` to remove the callback again. The probe
notifies you of any activity that happens on the pad, like buffers,
events and queries. You can define what kind of notifications you are
interested in when you add the probe.
The probe can notify you of the following activity on pads:
- A buffer is pushed or pulled. You want to specify the
GST\_PAD\_PROBE\_TYPE\_BUFFER when registering the probe. Because
the pad can be scheduled in different ways, it is possible to also
specify in what scheduling mode you are interested with the optional
GST\_PAD\_PROBE\_TYPE\_PUSH and GST\_PAD\_PROBE\_TYPE\_PULL flags.
2016-06-04 06:55:52 +00:00
You can use this probe to inspect, modify or drop the buffer. See
[Data probes](#data-probes).
- A bufferlist is pushed. Use the GST\_PAD\_PROBE\_TYPE\_BUFFER\_LIST
when registering the probe.
- An event travels over a pad. Use the
GST\_PAD\_PROBE\_TYPE\_EVENT\_DOWNSTREAM and
GST\_PAD\_PROBE\_TYPE\_EVENT\_UPSTREAM flags to select downstream
and upstream events. There is also a convenience
GST\_PAD\_PROBE\_TYPE\_EVENT\_BOTH to be notified of events going
both upstream and downstream. By default, flush events do not cause
a notification. You need to explicitly enable
GST\_PAD\_PROBE\_TYPE\_EVENT\_FLUSH to receive callbacks from
flushing events. Events are always only notified in push mode.
2016-06-04 06:55:52 +00:00
You can use this probe to inspect, modify or drop the event.
- A query travels over a pad. Use the
GST\_PAD\_PROBE\_TYPE\_QUERY\_DOWNSTREAM and
GST\_PAD\_PROBE\_TYPE\_QUERY\_UPSTREAM flags to select downstream
and upstream queries. The convenience
GST\_PAD\_PROBE\_TYPE\_QUERY\_BOTH can also be used to select both
directions. Query probes will be notified twice, once when the query
travels upstream/downstream and once when the query result is
returned. You can select in what stage the callback will be called
with the GST\_PAD\_PROBE\_TYPE\_PUSH and
GST\_PAD\_PROBE\_TYPE\_PULL, respectively when the query is
performed and when the query result is returned.
2016-06-04 06:55:52 +00:00
You can use this probe to inspect or modify the query. You can also
answer the query in the probe callback by placing the result value
in the query and by returning GST\_PAD\_PROBE\_DROP from the
callback.
- In addition to notifying you of dataflow, you can also ask the probe
to block the dataflow when the callback returns. This is called a
blocking probe and is activated by specifying the
GST\_PAD\_PROBE\_TYPE\_BLOCK flag. You can use this flag with the
other flags to only block dataflow on selected activity. A pad
becomes unblocked again if you remove the probe or when you return
GST\_PAD\_PROBE\_REMOVE from the callback. You can let only the
currently blocked item pass by returning GST\_PAD\_PROBE\_PASS from
the callback, it will block again on the next item.
2016-06-04 06:55:52 +00:00
Blocking probes are used to temporarily block pads because they are
unlinked or because you are going to unlink them. If the dataflow is
not blocked, the pipeline would go into an error state if data is
pushed on an unlinked pad. We will se how to use blocking probes to
partially preroll a pipeline. See also [Play a region of a media
file](#play-a-region-of-a-media-file).
- Be notified when no activity is happening on a pad. You install this
probe with the GST\_PAD\_PROBE\_TYPE\_IDLE flag. You can specify
GST\_PAD\_PROBE\_TYPE\_PUSH and/or GST\_PAD\_PROBE\_TYPE\_PULL to
only be notified depending on the pad scheduling mode. The IDLE
probe is also a blocking probe in that it will not let any data pass
on the pad for as long as the IDLE probe is installed.
2016-06-04 06:55:52 +00:00
You can use idle probes to dynamically relink a pad. We will see how
to use idle probes to replace an element in the pipeline. See also
[Dynamically changing the
pipeline](#dynamically-changing-the-pipeline).
### Data probes
2016-06-04 06:55:52 +00:00
Data probes allow you to be notified when there is data passing on a
pad. When adding the probe, specify the GST\_PAD\_PROBE\_TYPE\_BUFFER
and/or GST\_PAD\_PROBE\_TYPE\_BUFFER\_LIST.
Data probes run in pipeline streaming thread context, so callbacks
should try to not block and generally not do any weird stuff, since this
could have a negative impact on pipeline performance or, in case of
bugs, cause deadlocks or crashes. More precisely, one should usually not
call any GUI-related functions from within a probe callback, nor try to
change the state of the pipeline. An application may post custom
messages on the pipeline's bus though to communicate with the main
application thread and have it do things like stop the pipeline.
In any case, most common buffer operations that elements can do in
`_chain ()` functions, can be done in probe callbacks as well. The
example below gives a short impression on how to use them.
``` c
2016-06-04 06:55:52 +00:00
#include <gst/gst.h>
static GstPadProbeReturn
cb_have_data (GstPad *pad,
GstPadProbeInfo *info,
gpointer user_data)
{
gint x, y;
GstMapInfo map;
guint16 *ptr, t;
GstBuffer *buffer;
buffer = GST_PAD_PROBE_INFO_BUFFER (info);
buffer = gst_buffer_make_writable (buffer);
/* Making a buffer writable can fail (for example if it
* cannot be copied and is used more than once)
*/
if (buffer == NULL)
return GST_PAD_PROBE_OK;
/* Mapping a buffer can fail (non-writable) */
if (gst_buffer_map (buffer, &map, GST_MAP_WRITE)) {
ptr = (guint16 *) map.data;
/* invert data */
for (y = 0; y < 288; y++) {
for (x = 0; x < 384 / 2; x++) {
t = ptr[384 - 1 - x];
ptr[384 - 1 - x] = ptr[x];
ptr[x] = t;
}
ptr += 384;
}
gst_buffer_unmap (buffer, &map);
}
GST_PAD_PROBE_INFO_DATA (info) = buffer;
return GST_PAD_PROBE_OK;
}
gint
main (gint argc,
gchar *argv[])
{
GMainLoop *loop;
GstElement *pipeline, *src, *sink, *filter, *csp;
GstCaps *filtercaps;
GstPad *pad;
/* init GStreamer */
gst_init (&argc, &argv);
loop = g_main_loop_new (NULL, FALSE);
/* build */
pipeline = gst_pipeline_new ("my-pipeline");
src = gst_element_factory_make ("videotestsrc", "src");
if (src == NULL)
g_error ("Could not create 'videotestsrc' element");
filter = gst_element_factory_make ("capsfilter", "filter");
g_assert (filter != NULL); /* should always exist */
csp = gst_element_factory_make ("videoconvert", "csp");
if (csp == NULL)
g_error ("Could not create 'videoconvert' element");
sink = gst_element_factory_make ("xvimagesink", "sink");
if (sink == NULL) {
sink = gst_element_factory_make ("ximagesink", "sink");
if (sink == NULL)
g_error ("Could not create neither 'xvimagesink' nor 'ximagesink' element");
}
gst_bin_add_many (GST_BIN (pipeline), src, filter, csp, sink, NULL);
gst_element_link_many (src, filter, csp, sink, NULL);
filtercaps = gst_caps_new_simple ("video/x-raw",
"format", G_TYPE_STRING, "RGB16",
"width", G_TYPE_INT, 384,
"height", G_TYPE_INT, 288,
"framerate", GST_TYPE_FRACTION, 25, 1,
NULL);
g_object_set (G_OBJECT (filter), "caps", filtercaps, NULL);
gst_caps_unref (filtercaps);
pad = gst_element_get_static_pad (src, "src");
gst_pad_add_probe (pad, GST_PAD_PROBE_TYPE_BUFFER,
(GstPadProbeCallback) cb_have_data, NULL, NULL);
gst_object_unref (pad);
/* run */
gst_element_set_state (pipeline, GST_STATE_PLAYING);
/* wait until it's up and running or failed */
if (gst_element_get_state (pipeline, NULL, NULL, -1) == GST_STATE_CHANGE_FAILURE) {
g_error ("Failed to go into PLAYING state");
}
g_print ("Running ...\n");
g_main_loop_run (loop);
/* exit */
gst_element_set_state (pipeline, GST_STATE_NULL);
gst_object_unref (pipeline);
return 0;
}
2016-06-04 06:55:52 +00:00
```
Compare that output with the output of “gst-launch-1.0 videotestsrc \!
xvimagesink”, just so you know what you're looking for.
Strictly speaking, a pad probe callback is only allowed to modify the
buffer content if the buffer is writable. Whether this is the case or
not depends a lot on the pipeline and the elements involved. Often
enough, this is the case, but sometimes it is not, and if it is not then
unexpected modification of the data or metadata can introduce bugs that
are very hard to debug and track down. You can check if a buffer is
writable with `gst_buffer_is_writable ()`. Since you can pass back a
different buffer than the one passed in, it is a good idea to make the
buffer writable in the callback function with `gst_buffer_make_writable
()`.
Pad probes are suited best for looking at data as it passes through the
pipeline. If you need to modify data, you should better write your own
GStreamer element. Base classes like GstAudioFilter, GstVideoFilter or
GstBaseTransform make this fairly easy.
If you just want to inspect buffers as they pass through the pipeline,
you don't even need to set up pad probes. You could also just insert an
identity element into the pipeline and connect to its "handoff" signal.
The identity element also provides a few useful debugging tools like the
"dump" property or the "last-message" property (the latter is enabled by
passing the '-v' switch to gst-launch and by setting the silent property
on the identity to FALSE).
### Play a region of a media file
2016-06-04 06:55:52 +00:00
In this example we will show you how to play back a region of a media
file. The goal is to only play the part of a file from 2 seconds to 5
seconds and then EOS.
In a first step we will set a uridecodebin element to the PAUSED state
and make sure that we block all the source pads that are created. When
all the source pads are blocked, we have data on all source pads and we
say that the uridecodebin is prerolled.
In a prerolled pipeline we can ask for the duration of the media and we
can also perform seeks. We are interested in performing a seek operation
on the pipeline to select the range of media that we are interested in.
After we configure the region we are interested in, we can link the sink
element, unblock the source pads and set the pipeline to the playing
state. You will see that exactly the requested region is played by the
sink before it goes to EOS.
What follows is an example application that loosly follows this
algorithm.
``` c
2016-06-04 06:55:52 +00:00
#include <gst/gst.h>
static GMainLoop *loop;
static volatile gint counter;
static GstBus *bus;
static gboolean prerolled = FALSE;
static GstPad *sinkpad;
static void
dec_counter (GstElement * pipeline)
{
if (prerolled)
return;
if (g_atomic_int_dec_and_test (&counter)) {
/* all probes blocked and no-more-pads signaled, post
* message on the bus. */
prerolled = TRUE;
gst_bus_post (bus, gst_message_new_application (
GST_OBJECT_CAST (pipeline),
gst_structure_new_empty ("ExPrerolled")));
}
}
/* called when a source pad of uridecodebin is blocked */
static GstPadProbeReturn
cb_blocked (GstPad *pad,
GstPadProbeInfo *info,
gpointer user_data)
{
GstElement *pipeline = GST_ELEMENT (user_data);
if (prerolled)
return GST_PAD_PROBE_REMOVE;
dec_counter (pipeline);
return GST_PAD_PROBE_OK;
}
/* called when uridecodebin has a new pad */
static void
cb_pad_added (GstElement *element,
GstPad *pad,
gpointer user_data)
{
GstElement *pipeline = GST_ELEMENT (user_data);
if (prerolled)
return;
g_atomic_int_inc (&counter);
gst_pad_add_probe (pad, GST_PAD_PROBE_TYPE_BLOCK_DOWNSTREAM,
(GstPadProbeCallback) cb_blocked, pipeline, NULL);
/* try to link to the video pad */
gst_pad_link (pad, sinkpad);
}
/* called when uridecodebin has created all pads */
static void
cb_no_more_pads (GstElement *element,
gpointer user_data)
{
GstElement *pipeline = GST_ELEMENT (user_data);
if (prerolled)
return;
dec_counter (pipeline);
}
/* called when a new message is posted on the bus */
static void
cb_message (GstBus *bus,
GstMessage *message,
gpointer user_data)
{
GstElement *pipeline = GST_ELEMENT (user_data);
switch (GST_MESSAGE_TYPE (message)) {
case GST_MESSAGE_ERROR:
g_print ("we received an error!\n");
g_main_loop_quit (loop);
break;
case GST_MESSAGE_EOS:
g_print ("we reached EOS\n");
g_main_loop_quit (loop);
break;
case GST_MESSAGE_APPLICATION:
{
if (gst_message_has_name (message, "ExPrerolled")) {
/* it's our message */
g_print ("we are all prerolled, do seek\n");
gst_element_seek (pipeline,
1.0, GST_FORMAT_TIME,
GST_SEEK_FLAG_FLUSH | GST_SEEK_FLAG_ACCURATE,
GST_SEEK_TYPE_SET, 2 * GST_SECOND,
GST_SEEK_TYPE_SET, 5 * GST_SECOND);
gst_element_set_state (pipeline, GST_STATE_PLAYING);
}
break;
}
default:
break;
}
}
gint
main (gint argc,
gchar *argv[])
{
GstElement *pipeline, *src, *csp, *vs, *sink;
/* init GStreamer */
gst_init (&argc, &argv);
loop = g_main_loop_new (NULL, FALSE);
if (argc < 2) {
g_print ("usage: %s <uri>", argv[0]);
return -1;
}
/* build */
pipeline = gst_pipeline_new ("my-pipeline");
bus = gst_pipeline_get_bus (GST_PIPELINE (pipeline));
gst_bus_add_signal_watch (bus);
g_signal_connect (bus, "message", (GCallback) cb_message,
pipeline);
src = gst_element_factory_make ("uridecodebin", "src");
if (src == NULL)
g_error ("Could not create 'uridecodebin' element");
g_object_set (src, "uri", argv[1], NULL);
csp = gst_element_factory_make ("videoconvert", "csp");
if (csp == NULL)
g_error ("Could not create 'videoconvert' element");
vs = gst_element_factory_make ("videoscale", "vs");
if (csp == NULL)
g_error ("Could not create 'videoscale' element");
sink = gst_element_factory_make ("autovideosink", "sink");
if (sink == NULL)
g_error ("Could not create 'autovideosink' element");
gst_bin_add_many (GST_BIN (pipeline), src, csp, vs, sink, NULL);
/* can't link src yet, it has no pads */
gst_element_link_many (csp, vs, sink, NULL);
sinkpad = gst_element_get_static_pad (csp, "sink");
/* for each pad block that is installed, we will increment
* the counter. for each pad block that is signaled, we
* decrement the counter. When the counter is 0 we post
* an app message to tell the app that all pads are
* blocked. Start with 1 that is decremented when no-more-pads
* is signaled to make sure that we only post the message
* after no-more-pads */
g_atomic_int_set (&counter, 1);
g_signal_connect (src, "pad-added",
(GCallback) cb_pad_added, pipeline);
g_signal_connect (src, "no-more-pads",
(GCallback) cb_no_more_pads, pipeline);
gst_element_set_state (pipeline, GST_STATE_PAUSED);
g_main_loop_run (loop);
gst_element_set_state (pipeline, GST_STATE_NULL);
gst_object_unref (sinkpad);
gst_object_unref (bus);
gst_object_unref (pipeline);
g_main_loop_unref (loop);
return 0;
}
```
Note that we use a custom application message to signal the main thread
that the uridecidebin is prerolled. The main thread will then issue a
flushing seek to the requested region. The flush will temporarily
unblock the pad and reblock them when new data arrives again. We detect
this second block to remove the probes. Then we set the pipeline to
PLAYING and it should play from 2 to 5 seconds, then EOS and exit the
application.
## Manually adding or removing data from/to a pipeline
2016-06-04 06:55:52 +00:00
Many people have expressed the wish to use their own sources to inject
data into a pipeline. Some people have also expressed the wish to grab
the output in a pipeline and take care of the actual output inside their
application. While either of these methods are strongly discouraged,
GStreamer offers support for this. *Beware\! You need to know what you
are doing.* Since you don't have any support from a base class you need
to thoroughly understand state changes and synchronization. If it
doesn't work, there are a million ways to shoot yourself in the foot.
It's always better to simply write a plugin and have the base class
manage it. See the Plugin Writer's Guide for more information on this
topic. Also see the next section, which will explain how to embed
plugins statically in your application.
There's two possible elements that you can use for the above-mentioned
purposes. Those are called “appsrc” (an imaginary source) and “appsink”
(an imaginary sink). The same method applies to each of those elements.
Here, we will discuss how to use those elements to insert (using appsrc)
or grab (using appsink) data from a pipeline, and how to set
negotiation.
Both appsrc and appsink provide 2 sets of API. One API uses standard
GObject (action) signals and properties. The same API is also available
as a regular C api. The C api is more performant but requires you to
link to the app library in order to use the elements.
### Inserting data with appsrc
2016-06-04 06:55:52 +00:00
First we look at some examples for appsrc, which lets you insert data
into the pipeline from the application. Appsrc has some configuration
options that define how it will operate. You should decide about the
following configurations:
- Will the appsrc operate in push or pull mode. The stream-type
property can be used to control this. stream-type of “random-access”
will activate pull mode scheduling while the other stream-types
activate push mode.
- The caps of the buffers that appsrc will push out. This needs to be
configured with the caps property. The caps must be set to a fixed
caps and will be used to negotiate a format downstream.
- If the appsrc operates in live mode or not. This can be configured
with the is-live property. When operating in live-mode it is
important to configure the min-latency and max-latency in appsrc.
The min-latency should be set to the amount of time it takes between
capturing a buffer and when it is pushed inside appsrc. In live
mode, you should timestamp the buffers with the pipeline
running-time when the first byte of the buffer was captured before
feeding them to appsrc. You can let appsrc do the timestaping with
the do-timestamp property (but then the min-latency must be set to 0
because it timestamps based on the running-time when the buffer
entered appsrc).
- The format of the SEGMENT event that appsrc will push. The format
has implications for how the running-time of the buffers will be
calculated so you must be sure you understand this. For live sources
you probably want to set the format property to GST\_FORMAT\_TIME.
For non-live source it depends on the media type that you are
handling. If you plan to timestamp the buffers, you should probably
put a GST\_FORMAT\_TIME format, otherwise GST\_FORMAT\_BYTES might
be appropriate.
- If appsrc operates in random-access mode, it is important to
configure the size property of appsrc with the number of bytes in
the stream. This will allow downstream elements to know the size of
the media and alows them to seek to the end of the stream when
needed.
The main way of handling data to appsrc is by using the function
`gst_app_src_push_buffer ()` or by emiting the push-buffer action
signal. This will put the buffer onto a queue from which appsrc will
read from in its streaming thread. It is important to note that data
transport will not happen from the thread that performed the push-buffer
call.
The “max-bytes” property controls how much data can be queued in appsrc
before appsrc considers the queue full. A filled internal queue will
always signal the “enough-data” signal, which signals the application
that it should stop pushing data into appsrc. The “block” property will
cause appsrc to block the push-buffer method until free data becomes
available again.
When the internal queue is running out of data, the “need-data” signal
is emitted, which signals the application that it should start pushing
more data into appsrc.
In addition to the “need-data” and “enough-data” signals, appsrc can
emit the “seek-data” signal when the “stream-mode” property is set to
“seekable” or “random-access”. The signal argument will contain the
new desired position in the stream expressed in the unit set with the
“format” property. After receiving the seek-data signal, the
application should push-buffers from the new position.
When the last byte is pushed into appsrc, you must call
`gst_app_src_end_of_stream ()` to make it send an EOS downstream.
These signals allow the application to operate appsrc in push and pull
mode as will be explained next.
#### Using appsrc in push mode
2016-06-04 06:55:52 +00:00
When appsrc is configured in push mode (stream-type is stream or
seekable), the application repeatedly calls the push-buffer method with
a new buffer. Optionally, the queue size in the appsrc can be controlled
with the enough-data and need-data signals by respectively
stopping/starting the push-buffer calls. The value of the min-percent
property defines how empty the internal appsrc queue needs to be before
the need-data signal will be fired. You can set this to some value \>0
to avoid completely draining the queue.
When the stream-type is set to seekable, don't forget to implement a
seek-data callback.
Use this model when implementing various network protocols or hardware
devices.
#### Using appsrc in pull mode
2016-06-04 06:55:52 +00:00
In the pull model, data is fed to appsrc from the need-data signal
handler. You should push exactly the amount of bytes requested in the
need-data signal. You are only allowed to push less bytes when you are
at the end of the stream.
Use this model for file access or other randomly accessable sources.
#### Appsrc example
2016-06-04 06:55:52 +00:00
This example application will generate black/white (it switches every
second) video to an Xv-window output by using appsrc as a source with
caps to force a format. We use a colorspace conversion element to make
sure that we feed the right format to your X server. We configure a
video stream with a variable framerate (0/1) and we set the timestamps
on the outgoing buffers in such a way that we play 2 frames per second.
Note how we use the pull mode method of pushing new buffers into appsrc
although appsrc is running in push mode.
``` c
2016-06-04 06:55:52 +00:00
#include <gst/gst.h>
static GMainLoop *loop;
static void
cb_need_data (GstElement *appsrc,
guint unused_size,
gpointer user_data)
{
static gboolean white = FALSE;
static GstClockTime timestamp = 0;
GstBuffer *buffer;
guint size;
GstFlowReturn ret;
size = 385 * 288 * 2;
buffer = gst_buffer_new_allocate (NULL, size, NULL);
/* this makes the image black/white */
gst_buffer_memset (buffer, 0, white ? 0xff : 0x0, size);
white = !white;
GST_BUFFER_PTS (buffer) = timestamp;
GST_BUFFER_DURATION (buffer) = gst_util_uint64_scale_int (1, GST_SECOND, 2);
timestamp += GST_BUFFER_DURATION (buffer);
g_signal_emit_by_name (appsrc, "push-buffer", buffer, &ret);
gst_buffer_unref (buffer);
if (ret != GST_FLOW_OK) {
/* something wrong, stop pushing */
g_main_loop_quit (loop);
}
}
gint
main (gint argc,
gchar *argv[])
{
GstElement *pipeline, *appsrc, *conv, *videosink;
/* init GStreamer */
gst_init (&argc, &argv);
loop = g_main_loop_new (NULL, FALSE);
/* setup pipeline */
pipeline = gst_pipeline_new ("pipeline");
appsrc = gst_element_factory_make ("appsrc", "source");
conv = gst_element_factory_make ("videoconvert", "conv");
videosink = gst_element_factory_make ("xvimagesink", "videosink");
/* setup */
g_object_set (G_OBJECT (appsrc), "caps",
gst_caps_new_simple ("video/x-raw",
"format", G_TYPE_STRING, "RGB16",
"width", G_TYPE_INT, 384,
"height", G_TYPE_INT, 288,
"framerate", GST_TYPE_FRACTION, 0, 1,
NULL), NULL);
gst_bin_add_many (GST_BIN (pipeline), appsrc, conv, videosink, NULL);
gst_element_link_many (appsrc, conv, videosink, NULL);
/* setup appsrc */
g_object_set (G_OBJECT (appsrc),
"stream-type", 0,
"format", GST_FORMAT_TIME, NULL);
g_signal_connect (appsrc, "need-data", G_CALLBACK (cb_need_data), NULL);
/* play */
gst_element_set_state (pipeline, GST_STATE_PLAYING);
g_main_loop_run (loop);
/* clean up */
gst_element_set_state (pipeline, GST_STATE_NULL);
gst_object_unref (GST_OBJECT (pipeline));
g_main_loop_unref (loop);
return 0;
}
2016-06-04 06:55:52 +00:00
```
### Grabbing data with appsink
2016-06-04 06:55:52 +00:00
Unlike appsrc, appsink is a little easier to use. It also supports a
pull and push based model of getting data from the pipeline.
The normal way of retrieving samples from appsink is by using the
`gst_app_sink_pull_sample()` and `gst_app_sink_pull_preroll()` methods
or by using the “pull-sample” and “pull-preroll” signals. These methods
block until a sample becomes available in the sink or when the sink is
shut down or reaches EOS.
Appsink will internally use a queue to collect buffers from the
streaming thread. If the application is not pulling samples fast enough,
this queue will consume a lot of memory over time. The “max-buffers”
property can be used to limit the queue size. The “drop” property
controls whether the streaming thread blocks or if older buffers are
dropped when the maximum queue size is reached. Note that blocking the
streaming thread can negatively affect real-time performance and should
be avoided.
If a blocking behaviour is not desirable, setting the “emit-signals”
property to TRUE will make appsink emit the “new-sample” and
“new-preroll” signals when a sample can be pulled without blocking.
The “caps” property on appsink can be used to control the formats that
appsink can receive. This property can contain non-fixed caps, the
format of the pulled samples can be obtained by getting the sample caps.
If one of the pull-preroll or pull-sample methods return NULL, the
appsink is stopped or in the EOS state. You can check for the EOS state
with the “eos” property or with the `gst_app_sink_is_eos()` method.
The eos signal can also be used to be informed when the EOS state is
reached to avoid polling.
Consider configuring the following properties in the appsink:
- The “sync” property if you want to have the sink base class
synchronize the buffer against the pipeline clock before handing you
the sample.
- Enable Quality-of-Service with the “qos” property. If you are
dealing with raw video frames and let the base class sycnhronize on
the clock, it might be a good idea to also let the base class send
QOS events upstream.
- The caps property that contains the accepted caps. Upstream elements
will try to convert the format so that it matches the configured
caps on appsink. You must still check the `GstSample` to get the
actual caps of the buffer.
#### Appsink example
2016-06-04 06:55:52 +00:00
What follows is an example on how to capture a snapshot of a video
stream using appsink.
``` c
2016-06-04 06:55:52 +00:00
#include <gst/gst.h>
#ifdef HAVE_GTK
#include <gtk/gtk.h>
#endif
#include <stdlib.h>
#define CAPS "video/x-raw,format=RGB,width=160,pixel-aspect-ratio=1/1"
int
main (int argc, char *argv[])
{
GstElement *pipeline, *sink;
gint width, height;
GstSample *sample;
gchar *descr;
GError *error = NULL;
gint64 duration, position;
GstStateChangeReturn ret;
gboolean res;
GstMapInfo map;
gst_init (&argc, &argv);
if (argc != 2) {
g_print ("usage: %s <uri>\n Writes snapshot.png in the current directory\n",
argv[0]);
exit (-1);
}
/* create a new pipeline */
descr =
g_strdup_printf ("uridecodebin uri=%s ! videoconvert ! videoscale ! "
" appsink name=sink caps=\"" CAPS "\"", argv[1]);
pipeline = gst_parse_launch (descr, &error);
if (error != NULL) {
g_print ("could not construct pipeline: %s\n", error->message);
g_clear_error (&error);
exit (-1);
}
/* get sink */
sink = gst_bin_get_by_name (GST_BIN (pipeline), "sink");
/* set to PAUSED to make the first frame arrive in the sink */
ret = gst_element_set_state (pipeline, GST_STATE_PAUSED);
switch (ret) {
case GST_STATE_CHANGE_FAILURE:
g_print ("failed to play the file\n");
exit (-1);
case GST_STATE_CHANGE_NO_PREROLL:
/* for live sources, we need to set the pipeline to PLAYING before we can
* receive a buffer. We don't do that yet */
g_print ("live sources not supported yet\n");
exit (-1);
default:
break;
}
/* This can block for up to 5 seconds. If your machine is really overloaded,
* it might time out before the pipeline prerolled and we generate an error. A
* better way is to run a mainloop and catch errors there. */
ret = gst_element_get_state (pipeline, NULL, NULL, 5 * GST_SECOND);
if (ret == GST_STATE_CHANGE_FAILURE) {
g_print ("failed to play the file\n");
exit (-1);
}
/* get the duration */
gst_element_query_duration (pipeline, GST_FORMAT_TIME, &duration);
if (duration != -1)
/* we have a duration, seek to 5% */
position = duration * 5 / 100;
else
/* no duration, seek to 1 second, this could EOS */
position = 1 * GST_SECOND;
/* seek to the a position in the file. Most files have a black first frame so
* by seeking to somewhere else we have a bigger chance of getting something
* more interesting. An optimisation would be to detect black images and then
* seek a little more */
gst_element_seek_simple (pipeline, GST_FORMAT_TIME,
GST_SEEK_FLAG_KEY_UNIT | GST_SEEK_FLAG_FLUSH, position);
/* get the preroll buffer from appsink, this block untils appsink really
* prerolls */
g_signal_emit_by_name (sink, "pull-preroll", &sample, NULL);
/* if we have a buffer now, convert it to a pixbuf. It's possible that we
* don't have a buffer because we went EOS right away or had an error. */
if (sample) {
GstBuffer *buffer;
GstCaps *caps;
GstStructure *s;
/* get the snapshot buffer format now. We set the caps on the appsink so
* that it can only be an rgb buffer. The only thing we have not specified
* on the caps is the height, which is dependant on the pixel-aspect-ratio
* of the source material */
caps = gst_sample_get_caps (sample);
if (!caps) {
g_print ("could not get snapshot format\n");
exit (-1);
}
s = gst_caps_get_structure (caps, 0);
/* we need to get the final caps on the buffer to get the size */
res = gst_structure_get_int (s, "width", &width);
res |= gst_structure_get_int (s, "height", &height);
if (!res) {
g_print ("could not get snapshot dimension\n");
exit (-1);
}
/* create pixmap from buffer and save, gstreamer video buffers have a stride
* that is rounded up to the nearest multiple of 4 */
buffer = gst_sample_get_buffer (sample);
/* Mapping a buffer can fail (non-readable) */
if (gst_buffer_map (buffer, &map, GST_MAP_READ)) {
#ifdef HAVE_GTK
pixbuf = gdk_pixbuf_new_from_data (map.data,
GDK_COLORSPACE_RGB, FALSE, 8, width, height,
GST_ROUND_UP_4 (width * 3), NULL, NULL);
/* save the pixbuf */
gdk_pixbuf_save (pixbuf, "snapshot.png", "png", &error, NULL);
#endif
gst_buffer_unmap (buffer, &map);
}
gst_sample_unref (sample);
} else {
g_print ("could not make snapshot\n");
}
/* cleanup and exit */
gst_element_set_state (pipeline, GST_STATE_NULL);
gst_object_unref (pipeline);
exit (0);
}
```
## Forcing a format
2016-06-04 06:55:52 +00:00
Sometimes you'll want to set a specific format, for example a video size
and format or an audio bitsize and number of channels. You can do this
by forcing a specific `GstCaps` on the pipeline, which is possible by
using *filtered caps*. You can set a filtered caps on a link by using
the “capsfilter” element in between the two elements, and specifying a
`GstCaps` as “caps” property on this element. It will then only allow
types matching that specified capability set for negotiation. See also
[Creating capabilities for filtering][filter-caps].
[filter-caps]: application-development/basics/pads.md#creating-capabilities-for-filtering
2016-06-04 06:55:52 +00:00
### Changing format in a PLAYING pipeline
2016-06-04 06:55:52 +00:00
It is also possible to dynamically change the format in a pipeline while
PLAYING. This can simply be done by changing the caps property on a
capsfilter. The capsfilter will send a RECONFIGURE event upstream that
will make the upstream element attempt to renegotiate a new format and
allocator. This only works if the upstream element is not using fixed
caps on the source pad.
Below is an example of how you can change the caps of a pipeline while
in the PLAYING state:
``` c
2016-06-04 06:55:52 +00:00
#include <stdlib.h>
#include <gst/gst.h>
#define MAX_ROUND 100
int
main (int argc, char **argv)
{
GstElement *pipe, *filter;
GstCaps *caps;
gint width, height;
gint xdir, ydir;
gint round;
GstMessage *message;
gst_init (&argc, &argv);
pipe = gst_parse_launch_full ("videotestsrc ! capsfilter name=filter ! "
"ximagesink", NULL, GST_PARSE_FLAG_NONE, NULL);
g_assert (pipe != NULL);
filter = gst_bin_get_by_name (GST_BIN (pipe), "filter");
g_assert (filter);
width = 320;
height = 240;
xdir = ydir = -10;
for (round = 0; round < MAX_ROUND; round++) {
gchar *capsstr;
g_print ("resize to %dx%d (%d/%d) \r", width, height, round, MAX_ROUND);
/* we prefer our fixed width and height but allow other dimensions to pass
* as well */
capsstr = g_strdup_printf ("video/x-raw, width=(int)%d, height=(int)%d",
width, height);
caps = gst_caps_from_string (capsstr);
g_free (capsstr);
g_object_set (filter, "caps", caps, NULL);
gst_caps_unref (caps);
if (round == 0)
gst_element_set_state (pipe, GST_STATE_PLAYING);
width += xdir;
if (width >= 320)
xdir = -10;
else if (width < 200)
xdir = 10;
height += ydir;
if (height >= 240)
ydir = -10;
else if (height < 150)
ydir = 10;
message =
gst_bus_poll (GST_ELEMENT_BUS (pipe), GST_MESSAGE_ERROR,
50 * GST_MSECOND);
if (message) {
g_print ("got error \n");
gst_message_unref (message);
}
}
g_print ("done \n");
gst_object_unref (filter);
gst_element_set_state (pipe, GST_STATE_NULL);
gst_object_unref (pipe);
return 0;
}
2016-06-04 06:55:52 +00:00
```
Note how we use `gst_bus_poll()` with a small timeout to get messages
and also introduce a short sleep.
It is possible to set multiple caps for the capsfilter separated with a
;. The capsfilter will try to renegotiate to the first possible format
from the list.
## Dynamically changing the pipeline
2016-06-04 06:55:52 +00:00
In this section we talk about some techniques for dynamically modifying
the pipeline. We are talking specifically about changing the pipeline
while it is in the PLAYING state without interrupting the flow.
There are some important things to consider when building dynamic
pipelines:
- When removing elements from the pipeline, make sure that there is no
dataflow on unlinked pads because that will cause a fatal pipeline
error. Always block source pads (in push mode) or sink pads (in pull
mode) before unlinking pads. See also [Changing elements in a
pipeline](#changing-elements-in-a-pipeline).
- When adding elements to a pipeline, make sure to put the element
into the right state, usually the same state as the parent, before
allowing dataflow the element. When an element is newly created, it
is in the NULL state and will return an error when it receives data.
See also [Changing elements in a
pipeline](#changing-elements-in-a-pipeline).
- When adding elements to a pipeline, GStreamer will by default set
the clock and base-time on the element to the current values of the
pipeline. This means that the element will be able to construct the
same pipeline running-time as the other elements in the pipeline.
This means that sinks will synchronize buffers like the other sinks
in the pipeline and that sources produce buffers with a running-time
that matches the other sources.
- When unlinking elements from an upstream chain, always make sure to
flush any queued data in the element by sending an EOS event down
the element sink pad(s) and by waiting that the EOS leaves the
elements (with an event probe).
2016-06-04 06:55:52 +00:00
If you do not do this, you will lose the data which is buffered by
the unlinked element. This can result in a simple frame loss (one or
more video frames, several milliseconds of audio). However if you
remove a muxer (and in some cases an encoder or similar elements)
from the pipeline, you risk getting a corrupted file which could not
be played properly, as some relevant metadata (header, seek/index
tables, internal sync tags) will not be stored or updated properly.
2016-06-04 06:55:52 +00:00
See also [Changing elements in a
pipeline](#changing-elements-in-a-pipeline).
- A live source will produce buffers with a running-time of the
current running-time in the pipeline.
2016-06-04 06:55:52 +00:00
A pipeline without a live source produces buffers with a
running-time starting from 0. Likewise, after a flushing seek, those
pipelines reset the running-time back to 0.
2016-06-04 06:55:52 +00:00
The running-time can be changed with `gst_pad_set_offset ()`. It is
important to know the running-time of the elements in the pipeline
in order to maintain synchronization.
- Adding elements might change the state of the pipeline. Adding a
non-prerolled sink, for example, brings the pipeline back to the
prerolling state. Removing a non-prerolled sink, for example, might
change the pipeline to PAUSED and PLAYING state.
2016-06-04 06:55:52 +00:00
Adding a live source cancels the preroll stage and put the pipeline
to the playing state. Adding a live source or other live elements
might also change the latency of a pipeline.
2016-06-04 06:55:52 +00:00
Adding or removing elements to the pipeline might change the clock
selection of the pipeline. If the newly added element provides a
clock, it might be worth changing the clock in the pipeline to the
new clock. If, on the other hand, the element that provides the
clock for the pipeline is removed, a new clock has to be selected.
- Adding and removing elements might cause upstream or downstream
elements to renegotiate caps and or allocators. You don't really
need to do anything from the application, plugins largely adapt
themself to the new pipeline topology in order to optimize their
formats and allocation strategy.
2016-06-04 06:55:52 +00:00
What is important is that when you add, remove or change elements in
the pipeline, it is possible that the pipeline needs to negotiate a
new format and this can fail. Usually you can fix this by inserting
the right converter elements where needed. See also [Changing
elements in a pipeline](#changing-elements-in-a-pipeline).
GStreamer offers support for doing about any dynamic pipeline
modification but it requires you to know a bit of details before you can
do this without causing pipeline errors. In the following sections we
will demonstrate a couple of typical use-cases.
### Changing elements in a pipeline
2016-06-04 06:55:52 +00:00
In the next example we look at the following chain of elements:
```
2016-06-04 06:55:52 +00:00
- ----. .----------. .---- -
element1 | | element2 | | element3
src -> sink src -> sink
- ----' '----------' '---- -
2016-06-04 06:55:52 +00:00
```
We want to change element2 by element4 while the pipeline is in the
PLAYING state. Let's say that element2 is a visualization and that you
want to switch the visualization in the pipeline.
We can't just unlink element2's sinkpad from element1's source pad
because that would leave element1's source pad unlinked and would cause
a streaming error in the pipeline when data is pushed on the source pad.
The technique is to block the dataflow from element1's source pad before
we change element2 by element4 and then resume dataflow as shown in the
following steps:
- Block element1's source pad with a blocking pad probe. When the pad
is blocked, the probe callback will be called.
- Inside the block callback nothing is flowing between element1 and
element2 and nothing will flow until unblocked.
- Unlink element1 and element2.
- Make sure data is flushed out of element2. Some elements might
internally keep some data, you need to make sure not to lose data by
forcing it out of element2. You can do this by pushing EOS into
element2, like this:
2016-06-04 06:55:52 +00:00
- Put an event probe on element2's source pad.
2016-06-04 06:55:52 +00:00
- Send EOS to element2's sinkpad. This makes sure the all the data
inside element2 is forced out.
2016-06-04 06:55:52 +00:00
- Wait for the EOS event to appear on element2's source pad. When
the EOS is received, drop it and remove the event probe.
- Unlink element2 and element3. You can now also remove element2 from
the pipeline and set the state to NULL.
- Add element4 to the pipeline, if not already added. Link element4
and element3. Link element1 and element4.
- Make sure element4 is in the same state as the rest of the elements
in the pipeline. It should be at least in the PAUSED state before it
can receive buffers and events.
- Unblock element1's source pad probe. This will let new data into
element4 and continue streaming.
The above algorithm works when the source pad is blocked, i.e. when
there is dataflow in the pipeline. If there is no dataflow, there is
also no point in changing the element (just yet) so this algorithm can
be used in the PAUSED state as well.
Let show you how this works with an example. This example changes the
video effect on a simple pipeline every second.
``` c
2016-06-04 06:55:52 +00:00
#include <gst/gst.h>
static gchar *opt_effects = NULL;
#define DEFAULT_EFFECTS "identity,exclusion,navigationtest," \
"agingtv,videoflip,vertigotv,gaussianblur,shagadelictv,edgetv"
static GstPad *blockpad;
static GstElement *conv_before;
static GstElement *conv_after;
static GstElement *cur_effect;
static GstElement *pipeline;
static GQueue effects = G_QUEUE_INIT;
static GstPadProbeReturn
event_probe_cb (GstPad * pad, GstPadProbeInfo * info, gpointer user_data)
{
GMainLoop *loop = user_data;
GstElement *next;
if (GST_EVENT_TYPE (GST_PAD_PROBE_INFO_DATA (info)) != GST_EVENT_EOS)
return GST_PAD_PROBE_PASS;
gst_pad_remove_probe (pad, GST_PAD_PROBE_INFO_ID (info));
/* push current effect back into the queue */
g_queue_push_tail (&effects, gst_object_ref (cur_effect));
/* take next effect from the queue */
next = g_queue_pop_head (&effects);
if (next == NULL) {
GST_DEBUG_OBJECT (pad, "no more effects");
g_main_loop_quit (loop);
return GST_PAD_PROBE_DROP;
}
g_print ("Switching from '%s' to '%s'..\n", GST_OBJECT_NAME (cur_effect),
GST_OBJECT_NAME (next));
gst_element_set_state (cur_effect, GST_STATE_NULL);
/* remove unlinks automatically */
GST_DEBUG_OBJECT (pipeline, "removing %" GST_PTR_FORMAT, cur_effect);
gst_bin_remove (GST_BIN (pipeline), cur_effect);
GST_DEBUG_OBJECT (pipeline, "adding %" GST_PTR_FORMAT, next);
gst_bin_add (GST_BIN (pipeline), next);
GST_DEBUG_OBJECT (pipeline, "linking..");
gst_element_link_many (conv_before, next, conv_after, NULL);
gst_element_set_state (next, GST_STATE_PLAYING);
cur_effect = next;
GST_DEBUG_OBJECT (pipeline, "done");
return GST_PAD_PROBE_DROP;
}
static GstPadProbeReturn
pad_probe_cb (GstPad * pad, GstPadProbeInfo * info, gpointer user_data)
{
GstPad *srcpad, *sinkpad;
GST_DEBUG_OBJECT (pad, "pad is blocked now");
/* remove the probe first */
gst_pad_remove_probe (pad, GST_PAD_PROBE_INFO_ID (info));
/* install new probe for EOS */
srcpad = gst_element_get_static_pad (cur_effect, "src");
gst_pad_add_probe (srcpad, GST_PAD_PROBE_TYPE_BLOCK |
GST_PAD_PROBE_TYPE_EVENT_DOWNSTREAM, event_probe_cb, user_data, NULL);
gst_object_unref (srcpad);
/* push EOS into the element, the probe will be fired when the
* EOS leaves the effect and it has thus drained all of its data */
sinkpad = gst_element_get_static_pad (cur_effect, "sink");
gst_pad_send_event (sinkpad, gst_event_new_eos ());
gst_object_unref (sinkpad);
return GST_PAD_PROBE_OK;
}
static gboolean
timeout_cb (gpointer user_data)
{
gst_pad_add_probe (blockpad, GST_PAD_PROBE_TYPE_BLOCK_DOWNSTREAM,
pad_probe_cb, user_data, NULL);
return TRUE;
}
static gboolean
bus_cb (GstBus * bus, GstMessage * msg, gpointer user_data)
{
GMainLoop *loop = user_data;
switch (GST_MESSAGE_TYPE (msg)) {
case GST_MESSAGE_ERROR:{
GError *err = NULL;
gchar *dbg;
gst_message_parse_error (msg, &err, &dbg);
gst_object_default_error (msg->src, err, dbg);
g_clear_error (&err);
g_free (dbg);
g_main_loop_quit (loop);
break;
}
default:
break;
}
return TRUE;
}
int
main (int argc, char **argv)
{
GOptionEntry options[] = {
{"effects", 'e', 0, G_OPTION_ARG_STRING, &opt_effects,
"Effects to use (comma-separated list of element names)", NULL},
{NULL}
};
GOptionContext *ctx;
GError *err = NULL;
GMainLoop *loop;
GstElement *src, *q1, *q2, *effect, *filter1, *filter2, *sink;
gchar **effect_names, **e;
ctx = g_option_context_new ("");
g_option_context_add_main_entries (ctx, options, NULL);
g_option_context_add_group (ctx, gst_init_get_option_group ());
if (!g_option_context_parse (ctx, &argc, &argv, &err)) {
g_print ("Error initializing: %s\n", err->message);
g_clear_error (&amp;err);
g_option_context_free (ctx);
return 1;
}
g_option_context_free (ctx);
if (opt_effects != NULL)
effect_names = g_strsplit (opt_effects, ",", -1);
else
effect_names = g_strsplit (DEFAULT_EFFECTS, ",", -1);
for (e = effect_names; e != NULL && *e != NULL; ++e) {
GstElement *el;
el = gst_element_factory_make (*e, NULL);
if (el) {
g_print ("Adding effect '%s'\n", *e);
g_queue_push_tail (&effects, el);
}
}
pipeline = gst_pipeline_new ("pipeline");
src = gst_element_factory_make ("videotestsrc", NULL);
g_object_set (src, "is-live", TRUE, NULL);
filter1 = gst_element_factory_make ("capsfilter", NULL);
gst_util_set_object_arg (G_OBJECT (filter1), "caps",
"video/x-raw, width=320, height=240, "
"format={ I420, YV12, YUY2, UYVY, AYUV, Y41B, Y42B, "
"YVYU, Y444, v210, v216, NV12, NV21, UYVP, A420, YUV9, YVU9, IYU1 }");
q1 = gst_element_factory_make ("queue", NULL);
blockpad = gst_element_get_static_pad (q1, "src");
conv_before = gst_element_factory_make ("videoconvert", NULL);
effect = g_queue_pop_head (&effects);
cur_effect = effect;
conv_after = gst_element_factory_make ("videoconvert", NULL);
q2 = gst_element_factory_make ("queue", NULL);
filter2 = gst_element_factory_make ("capsfilter", NULL);
gst_util_set_object_arg (G_OBJECT (filter2), "caps",
"video/x-raw, width=320, height=240, "
"format={ RGBx, BGRx, xRGB, xBGR, RGBA, BGRA, ARGB, ABGR, RGB, BGR }");
sink = gst_element_factory_make ("ximagesink", NULL);
gst_bin_add_many (GST_BIN (pipeline), src, filter1, q1, conv_before, effect,
conv_after, q2, sink, NULL);
gst_element_link_many (src, filter1, q1, conv_before, effect, conv_after,
q2, sink, NULL);
gst_element_set_state (pipeline, GST_STATE_PLAYING);
loop = g_main_loop_new (NULL, FALSE);
gst_bus_add_watch (GST_ELEMENT_BUS (pipeline), bus_cb, loop);
g_timeout_add_seconds (1, timeout_cb, loop);
g_main_loop_run (loop);
gst_element_set_state (pipeline, GST_STATE_NULL);
gst_object_unref (pipeline);
return 0;
}
2016-06-04 06:55:52 +00:00
```
Note how we added videoconvert elements before and after the effect.
This is needed because some elements might operate in different
colorspaces than other elements. By inserting the conversion elements
you ensure that the right format can be negotiated at any time.