gstreamer/markdown/tutorials/playback/hardware-accelerated-video-decoding.md

187 lines
8.5 KiB
Markdown
Raw Normal View History

# Playback tutorial 8: Hardware-accelerated video decoding
2016-05-16 14:30:34 +00:00
### Goal
2016-05-16 14:30:34 +00:00
Hardware-accelerated video decoding has rapidly become a necessity, as
low-power devices grow more common. This tutorial (more of a lecture,
actually) gives some background on hardware acceleration and explains
how does GStreamer benefit from it.
Sneak peek: if properly setup, you do not need to do anything special to
activate hardware acceleration; GStreamer automatically takes advantage
of it.
### Introduction
2016-05-16 14:30:34 +00:00
Video decoding can be an extremely CPU-intensive task, especially for
higher resolutions like 1080p HDTV. Fortunately, modern graphics cards,
equipped with programmable GPUs, are able to take care of this job,
allowing the CPU to concentrate on other duties. Having dedicated
hardware becomes essential for low-power CPUs which are simply incapable
of decoding such media fast enough.
2016-06-16 01:26:59 +00:00
In the current state of things (June 2016) each GPU manufacturer offers
2016-05-16 14:30:34 +00:00
a different method to access their hardware (a different API), and a
strong industry standard has not emerged yet.
2016-06-16 01:26:59 +00:00
As of June 2016, there exist at least 8 different video decoding
2016-05-16 14:30:34 +00:00
acceleration APIs:
2016-06-16 01:26:59 +00:00
- [VAAPI](http://en.wikipedia.org/wiki/Video_Acceleration_API) (*Video
2016-05-16 14:30:34 +00:00
Acceleration API*): Initially designed by
2016-06-16 01:26:59 +00:00
[Intel](http://en.wikipedia.org/wiki/Intel) in 2007, targeted at the X
Window System on Unix-based operating systems, now open-source. It now also
supports Wayland through dmabuf. It is
2016-05-16 14:30:34 +00:00
currently not limited to Intel GPUs as other manufacturers are free to
use this API, for example, [Imagination
Technologies](http://en.wikipedia.org/wiki/Imagination_Technologies) or
[S3 Graphics](http://en.wikipedia.org/wiki/S3_Graphics). Accessible to
2018-08-01 04:20:27 +00:00
GStreamer through the [gstreamer-vaapi](https://gitlab.freedesktop.org/gstreamer/gstreamer-vaapi/) package.
2016-05-16 14:30:34 +00:00
2016-06-16 01:26:59 +00:00
- [VDPAU](http://en.wikipedia.org/wiki/VDPAU) (*Video Decode and
2016-05-16 14:30:34 +00:00
Presentation API for UNIX*): Initially designed by
2016-06-16 01:26:59 +00:00
[NVidia](http://en.wikipedia.org/wiki/NVidia) in 2008, targeted at the X
2016-05-16 14:30:34 +00:00
Window System on Unix-based operating systems, now open-source. Although
it is also an open-source library, no manufacturer other than NVidia is
2016-06-16 01:26:59 +00:00
using it yet. Accessible to GStreamer through
2018-08-01 04:20:27 +00:00
the [vdpau](http://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/tree/master/sys/vdpau) element in plugins-bad.
2016-06-16 01:26:59 +00:00
- [OpenMAX](http://en.wikipedia.org/wiki/OpenMAX) (*Open Media
Acceleration*): Managed by the non-profit technology consortium [Khronos
2016-05-16 14:30:34 +00:00
Group](http://en.wikipedia.org/wiki/Khronos_Group "Khronos Group"),
2016-06-16 01:26:59 +00:00
it is a "royalty-free, cross-platform set of C-language programming
2016-05-16 14:30:34 +00:00
interfaces that provides abstractions for routines especially useful for
2016-06-16 01:26:59 +00:00
audio, video, and still images". Accessible to GStreamer through
the [gst-omx](http://git.freedesktop.org/gstreamer/gst-omx) plugin.
2016-05-16 14:30:34 +00:00
2016-06-16 01:26:59 +00:00
- [OVD](http://developer.amd.com/sdks/AMDAPPSDK/assets/OpenVideo_Decode_API.PDF)
2016-05-16 14:30:34 +00:00
(*Open Video Decode*): Another API from [AMD
2016-06-16 01:26:59 +00:00
Graphics](http://en.wikipedia.org/wiki/AMD_Graphics), designed to be a
2016-05-16 14:30:34 +00:00
platform agnostic method for softrware developers to leverage the
[Universal Video
Decode](http://en.wikipedia.org/wiki/Unified_Video_Decoder) (UVD)
2016-06-16 01:26:59 +00:00
hardware inside AMD Radeon graphics cards. Currently unavailable to
GStreamer .
2016-05-16 14:30:34 +00:00
2016-06-16 01:26:59 +00:00
- [DCE](http://en.wikipedia.org/wiki/Distributed_Codec_Engine)
(*Distributed Codec Engine*): An open source software library ("libdce")
and API specification by [Texas
2016-05-16 14:30:34 +00:00
Instruments](http://en.wikipedia.org/wiki/Texas_Instruments), targeted
2016-06-16 01:26:59 +00:00
at Linux systems and ARM platforms. Accessible to GStreamer through
the [gstreamer-ducati](https://github.com/robclark/gst-ducati) plugin.
- [Android
MediaCodec](https://developer.android.com/reference/android/media/MediaCodec.html): This is Android's API to access the device's
hardware decoder and encoder if available. This is accessible through the
`androidmedia` plugin in gst-plugins-bad. This includes both encoding and
decoding.
- Apple VideoTool Box Framework: Apple's API to access h is available
through the `applemedia` plugin which includes both encoding through
the `vtenc` element and decoding through the `vtdec` element.
- Video4Linux: Recent Linux kernels have a kernel API to expose
hardware codecs in a standard way, this is now supported by the
`v4l2` plugin in `gst-plugins-good`. This can support both decoding
and encoding depending on the platform.
### Inner workings of hardware-accelerated video decoding plugins
2016-05-16 14:30:34 +00:00
These APIs generally offer a number of functionalities, like video
2016-06-16 01:26:59 +00:00
decoding, post-processing, or presentation of the decoded
frames. Correspondingly, plugins generally offer a different GStreamer
element for each of these functions, so pipelines can be built to
accommodate any need.
2016-05-16 14:30:34 +00:00
2016-06-16 01:26:59 +00:00
For example, the `gstreamer-vaapi` plugin offers the `vaapidecode`,
`vaapipostproc` and `vaapisink` elements that allow
2016-05-16 14:30:34 +00:00
hardware-accelerated decoding through VAAPI, upload of raw video frames
to GPU memory, download of GPU frames to system memory and presentation
of GPU frames, respectively.
It is important to distinguish between conventional GStreamer frames,
which reside in system memory, and frames generated by
2016-06-16 01:26:59 +00:00
hardware-accelerated APIs. The latter reside in GPU memory and cannot
be touched by GStreamer. They can usually be downloaded to system
memory and treated as conventional GStreamer frames when they are
mapped, but it is far more efficient to leave them in the GPU and
display them from there.
2016-05-16 14:30:34 +00:00
GStreamer needs to keep track of where these “hardware buffers” are
2016-06-16 01:26:59 +00:00
though, so conventional buffers still travel from element to
element. They look like regular buffers, but mapping their content is
much slower as it has to be retrieved from the special memory used by
hardware accelerated elements. This special memory types are
negotiated using the allocation query mechanism.
2016-05-16 14:30:34 +00:00
This all means that, if a particular hardware acceleration API is
present in the system, and the corresponding GStreamer plugin is also
2016-06-16 01:26:59 +00:00
available, auto-plugging elements like `playbin` are free to use
2016-05-16 14:30:34 +00:00
hardware acceleration to build their pipelines; the application does not
need to do anything special to enable it. Almost:
2016-06-16 01:26:59 +00:00
When `playbin` has to choose among different equally valid elements,
2016-05-16 14:30:34 +00:00
like conventional software decoding (through `vp8dec`, for example) or
hardware accelerated decoding (through `vaapidecode`, for example), it
uses their *rank* to decide. The rank is a property of each element that
2016-06-16 01:26:59 +00:00
indicates its priority; `playbin` will simply select the element that
2016-05-16 14:30:34 +00:00
is able to build a complete pipeline and has the highest rank.
2016-06-16 01:26:59 +00:00
So, whether `playbin` will use hardware acceleration or not will depend
2016-05-16 14:30:34 +00:00
on the relative ranks of all elements capable of dealing with that media
type. Therefore, the easiest way to make sure hardware acceleration is
enabled or disabled is by changing the rank of the associated element,
as shown in this code:
2016-06-06 00:58:09 +00:00
``` c
2016-05-16 14:30:34 +00:00
static void enable_factory (const gchar *name, gboolean enable) {
GstRegistry *registry = NULL;
GstElementFactory *factory = NULL;
2016-05-16 14:30:34 +00:00
registry = gst_registry_get_default ();
if (!registry) return;
2016-05-16 14:30:34 +00:00
factory = gst_element_factory_find (name);
if (!factory) return;
2016-05-16 14:30:34 +00:00
if (enable) {
gst_plugin_feature_set_rank (GST_PLUGIN_FEATURE (factory), GST_RANK_PRIMARY + 1);
}
else {
gst_plugin_feature_set_rank (GST_PLUGIN_FEATURE (factory), GST_RANK_NONE);
}
2016-05-16 14:30:34 +00:00
gst_registry_add_feature (registry, GST_PLUGIN_FEATURE (factory));
return;
}
```
The first parameter passed to this method is the name of the element to
2016-06-16 01:26:59 +00:00
modify, for example, `vaapidecode` or `fluvadec`.
2016-05-16 14:30:34 +00:00
2016-06-16 01:26:59 +00:00
The key method is `gst_plugin_feature_set_rank()`, which will set the
2016-05-16 14:30:34 +00:00
rank of the requested element factory to the desired level. For
convenience, ranks are divided in NONE, MARGINAL, SECONDARY and PRIMARY,
but any number will do. When enabling an element, we set it to
PRIMARY+1, so it has a higher rank than the rest of elements which
commonly have PRIMARY rank. Setting an elements rank to NONE will make
the auto-plugging mechanism to never select it.
2016-06-16 01:26:59 +00:00
> ![warning] The GStreamer developers often rank hardware decoders lower than
> the software ones when they are defective. This should act as a warning.
2016-05-16 14:30:34 +00:00
## Conclusion
2016-05-16 14:30:34 +00:00
This tutorial has shown a bit how GStreamer internally manages hardware
accelerated video decoding. Particularly,
- Applications do not need to do anything special to enable hardware
acceleration if a suitable API and the corresponding GStreamer
plugin are available.
- Hardware acceleration can be enabled or disabled by changing the
rank of the decoding element with `gst_plugin_feature_set_rank()`.
2016-06-16 01:26:59 +00:00
It has been a pleasure having you here, and see you soon!
[warning]: images/icons/emoticons/warning.svg