Fix new internal video format API, based on GstVideoFormat, to not
clobber with system symbols. So replace the gst_video_format_* prefix
with gst_vaapi_video_format_ prefix, even if the format type remains
GstVideoFormat.
Simplify gst_vaapi_create_display() helper as gst_vaapi_display_XXX_new()
performs the necessary validation checks for the underlying VA display
prior to returning to the caller. So, if an error occurred, then NULL is
really returned in that case.
If the video buffer pool config doesn't have new caps, then it's not
necessary to reinstantiate the allocator. That could be a costly
operation as we could do some extra heavy checking in there.
Fix reference counting issue whereby gst_memory_init() does not hold
an extra reference to the GstAllocator. So, there could be situations
where the last instance of GstVaapiVideoAllocator gets released before
a dangling GstVaapiVideoMemory object, thus possibly leading to a crash.
Always perform conversion of sources buffers to NV12 since this is
the way we tested for this capability in ensure_allowed_caps(). This
also saves memory bandwidth for further rendering. However, this may
not preserve quality since the YUV buffers are down-sampled to 4:2:0.
This fixes direct linking of vaapidownload element to xvimagesink with
VA drivers supporting vaGetImage() from the native VA surface format to
a different VA image format. i.e. color conversion during download.
http://bugzilla.gnome.org/show_bug.cgi?id=703937
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
The image is now expressed as a standard GstVideoFormat, which is not
a FOURCC but rather a regular enum value.
This is a regression introduced in commit 09397fa.
Fix gst_vaapi_uploader_get_buffer() to not assign caps since they
were already negotiated beforehand, and they are not used from the
buffer in upstream elements.
Clean-up gst_vaapi_uploader_ensure_caps() to use the new image caps
represented as a GstVideoInfo.
Adapt GstVaapiVideoMemory allocator to support creation of VA surfaces
with an explicit pixel format. This allows for direct rendering to
VA surface memory from a software decoder.
Get rid of GstCaps to create surface/image pool, and use GstVideoInfo
structures instead. Those are smaller, and allows for streamlining
libgstvaapi more.
Fix creation of GLX texture, to not depend on the GstCaps video size that
could be wrong, especially in presence of frame cropping. So, use the size
from the source VA surfaces.
An optimization could be to reduce the texture size to the actual visible
size on screen. i.e. scale down the texture size to match the screen dimensions,
while preserving the VA surface aspect ratio. However, some VA drivers don't
honour that.
Add support for GstVideoCropMeta in GStreamer >= 1.0.x builds and gst-vaapi
specific meta information to hold video cropping details. Make the sink
support video cropping in X11 and GLX modes.
Some video clips may have a clipping region that needs to propogate to
the renderer. These helper functions make it possible to attach that
clipping region, as a GstVaapiRectangle, the the video meta associated
with the buffer.
Signed-off-by: Sreerenj Balachandran <sreerenj.balachandran@intel.com>
signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
Expose all raw video formats in the static caps template since the
vaapisink is supporting raw data. We will get the exact set of formats
supported by the driver dynamically through the _get_caps() routine.
This also fixes an inconsistency wrt. GStreamer 0.10 builds.
https://bugzilla.gnome.org/show_bug.cgi?id=702178
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
Now that VA/GLX capable buffers are generated by default on X11, thus
depending on a VA/GLX display, we stil want to use vaPutSurface() for
rendering since it is faster.
Anyway, OpenGL rendering in vaapisink was only meant for testing and
enabling "fancy" effects to play with. This has no real value. So,
disable OpenGL rendering by default.
If the gstreamer-vaapi plug-in elements are built with GLX support, then
try to allocate a GstVaapiDisplayGLX first before resorting to a VA/X11
display next.
https://bugzilla.gnome.org/show_bug.cgi?id=701742
Allow plain gst_buffer_map() interface to work with gstreamer-vaapi
video buffers, i.e. expose the underlying GstVaapiSurfaceProxy to the
caller. This is the only sensible enough thing to do in this mode as
the underlying surface pixels need to be extracted through an explicit
call to the gst_video_frame_map() function instead.
A possible use-case of this is to implement a "handoff" signal handler
to fakesink or identity element for further processing.
Fix gst_vaapi_video_allocator_new() to silently check for direct-rendering
mode support, and not trigger fatal-criticals if either test surface or
image could not be created. Typical case: pixel format mismatch, e.g. NV12
supported by most hardware vs. I420 supported by most software decoders.
On map, ensure we have GST_MAP_WRITE flags since this is only what we
support for now. Likewise, on unmap, make sure that the VA image is
unmapped for either read or write, while still committing it to the
VA surface if write was requested.
In GStreamer 0.10 builds, gst_vaapi_uploader_get_buffer() was used
but it exhibited a memory leak because the surface generated for the
GstVaapiVideoMeta totally lost its parent video pool. So, it was not
possible to release that surface back to the parent pool when the meta
gets released, and the memory consumption kept growing.
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
Since GST_VAAPI_IS_xxx_VIDEO_POOL() was only testing for NULL and not
the underlying object type, the gst_vaapi_video_meta_new_from_pool()
was hereby totally broken. Fixed this regression by using the newly
provided gst_vaapi_video_pool_get_object_type() function.
Add gst_vaapi_decoder_get_frame_with_timeout() helper function that will
wait for a frame to be decoded, until the specified timeout in microseconds,
prior to returning to the caller.
This is a fix to performance regression from 851cc0, whereby the vaapidecode
loop executed on the srcpad task was called to often, thus starving all CPU
resources.
Rework heuristics to detect when downstream element ran into errors,
and thus failing to release any VA surface in due time for the current
frame to get decoded. In particular, recalibrate the render time base
when the first frame gets submitted downstream, or when there is no
timestamp that could be inferred.
Rework GstVideoDecoder::handle_frame() to decode the current frame,
while possibly waiting for a free surface, and separately submit all
decoded frames from a task. This makes it possible to pop and render
decoded frames as soon as possible.
Fix reference counting bug for passthrough mode, whereby the input buffer
was propagated as is downstream through gst_pad_push() without increasing
its reference count before. The was a problem when gst_pad_push() returns
an error and we further decrease the reference count of the input buffer.
Add support for interlaced streams with GStreamer 1.0 too. Basically,
this enables vaapipostproc, though it is not auto-plugged yet. We also
make sure to reply to CAPS queries, and happily handle CAPS events.
Make gst_vaapi_decoder_get_codec_state() return the original codec state,
i.e. make the GstVaapiDecoder object own the return state so that callers
that want an extra reference to it would just gst_video_codec_state_ref()
it before usage. This aligns the behaviour with what we had before with
gst_vaapi_decoder_get_caps().
This is an ABI incompatible change, library major version was bumped from
previous release (0.5.2).
Mark the following functions are internal, i.e. private to the vaapi plug-in:
- gst_vaapi_video_buffer_pool_get_type()
- gst_vaapi_video_converter_glx_get_type()
- gst_vaapi_video_converter_glx_new()
Implement GstSurfaceMeta API for GStreamer 1.0.x. Even though this is
an unstable/deprecated API, this makes it possible to support Clutter
sink with minimal changes. Tested against clutter-gst 1.9.92.
When render-mode is "overlay", then it is not really useful to peek into
the GstBaseSink::last_buffer, since we have our own video_buffer already
recorded and maintained into GstVaapiSink.
Fix memory leak of GstSample objects in GstVideoOverlayInterface::expose().
This also fixes extra unreferencing of the underlying GstBuffer in the common
path afterwards (for both 0.10 or 1.0).
Fix the name of the plug-in element reported to gst-inspect-1.0. i.e. we
need an explicit definition for GStreamer >= 1.0 because the GST_PLUGIN_DEFINE
incorrectly uses #name for creating the plug-in name, instead of using macro
expansion (and let further expansion of macros) through e.g. G_STRINGIFY().
Fix make dist to allow build for either GStreamer 0.10 or 1.0. i.e. make
sure to include all source files in either case while generating source
tarballs.
Implement GstVideoMeta::{,un}map() to support raw YUV buffer upload when
the last component is unmapped. Downloads are not supported yet. The aim
was to first support SW decoding + HW accelerated rendering (vaapisink).
e.g. for Wayland.
Handle GST_QUERY_CAPS, which is the GStreamer 1.0 mechanism to retrieve
the set of allowed caps, i.e. it works similar to GstPad::get_caps().
This fixes fallback to SW decoding if no HW decoder is available.
Introduce a new configure option --with-gstreamer-api that determines
the desired GStreamer API to use. By default, GStreamer 1.0 is selected.
Also integrate more compatibility glue into gstcompat.h and plugins.
Use new GstVaapiVideoBufferPool to maintain video buffers. Implement
GstBaseSink::propose_allocation() to expose that pool to upstream
elements; and also implement GstVideoDecoder::decide_allocation() to
actually use that pool (from downstream), if any, or create one.
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
Add initial support for GstVaapiVideoMemory backed buffer pool. The memory
object currently holds a reference to GstVaapiVideoMeta.
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
Make it possible to copy GstVaapiVideoMeta objects, unless they contain VA
objects created from GstVaapiVideoPool. This is mostly useful to clone a
GstVaapiVideoMeta object containing a VA surface proxy so that to alter its
rendering flags.
Fix GstVaapiVideoMeta to allow VA objects to be destroyed when they are
reset to NULL. i.e. make gst_vaapi_video_meta_set_{image,surface}() and
gst_vaapi_video_meta_set_surface_proxy() actually clear VA objects when
argument is NULL.
Port vaapidecode and vaapisink plugins to GStreamer API >= 1.0. This
is rather minimalistic so that to test the basic functionality.
Disable vaapiupload, vaapidownload and vaapipostproc plugins. The latter
needs polishing wrt. to GStreamer 1.x functionality and the former are
totally phased out in favor of GstVaapiVideoMemory map/unmap facilities,
which are yet to be implemented.
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
Improve check for raw YUV format modes by avoiding checks against strings
("video/x-raw-yuv") for each new GstBuffer allocation. In the usual case,
GstBaseSink::set_caps() is called first and if VA surface format mode is
used, then GstBaseSink::buffer_alloc() is not called. If the latter is
called before set_caps(), then we just make a full check. This one is
pretty rare though, e.g. it usually happens once for custom pipelines.
Fix gst_vaapi_apply_composition() to not fail if no overlay composition
was found. i.e. return success (TRUE). This was harmless though extra
debug messages are not nice.
This is a regression introduced by commit 95b8659.
Don't return static caps that don't mean anything for the underlying codecs
that are actually supported for decoding. i.e. always allocate a VA display
and retrieve the exact set of HW decoders available. That VA display may be
re-used later on during negotiation through GstVideoContext "prepare-context".
This fixes fallback to SW decoding if no HW decoder is available.
Make gst_vaapi_reply_to_query() first check whether the query argument
is actually a video-context query, i.e. with type GST_QUERY_TYPE_CUSTOM.
Then, make sure vaapisink propagates the query to the parent class if
it is not a video-context query.
Add new gst_vaapi_video_buffer_new() helper function that allocates a video
buffer from a GstVaapiVideoMeta. Also remove obsolete and useless function
gst_vaapi_video_buffer_get_meta().
Move GstVaapiVideoMeta from core libgstvaapi decoding library to the
actual plugin elements. That's only useful there. Also inline reference
counting code from GstVaapiMiniObject.
Make sure libgstvaapi core decoding library doesn't include un-needed
dependencies. So, move out GstVaapiVideoConverterGLX to plugins instead.
Besides, even if the vaapisink element is not used, we are bound to have
a correctly populated GstSurfaceBuffer from vaapidecode.
Also clean-up the file along the way.
If the raw YUV buffer was created from vaapisink, through the buffer_alloc()
hook, then it will have a valid GstVaapiVideoMeta object attached to it.
However, we previously assumed in that case that it was a "native" VA buffer,
thus not calling into GstVaapiUploader::process().
Use gst_element_class_set_static_metadata() from GStreamer 1.0, which
basically is the same as gst_element_class_set_details_simple() in
GStreamer 0.10 context.
Move GstImplementsInterface and GstVideoContext support functions up
so that to keep a clear separation between the plugin element and its
interface hooks.
Use GstVideoInfo and gst_video_info_from_caps() helper wherever possible.
Also use the newly added gst_vaapi_image_format_from_structure() helper
in GstVaapiUploader::ensure_allowed_caps().
gst_vaapi_video_buffer_new_from_buffer() needs to reference the source
buffer video meta since it would be unreference'd from the get_buffer()
helper function. For other cases, we still use (steal) the newly created
video meta.
Fix ensure_image() to only zero-initialize the first line of each plane.
Properly initializing each plane to their full vertical resolution would
require to actually compute it based on the image format.
In particular, for NV12 images, the UV plane has half vertical resolution
vs. the Y plane. So using the full image height to initialize the UV plane
will obviously lead to a buffer overflow. Likewise for other YUV format.
Since ensure_image() is only a helper function to initialize something,
and not necessarily the whole thing, it is fine to initializ the first
line only. Besides, the target surface is not rendered either.
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
Decode-only frames may not have a valid surface proxy. So, simply discard
them gracefully, i.e. don't create meta data information. GstVideoDecoder
base class will properly handle this case and won't try to push any buffer
to downstream elements.
Implement GstVideoDecoder::reset() as a destruction of the VA decoder
and the creation of a new VA decoder.
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
Split GstVideoDecoder::handle_frame() implementation into two functions:
(i) one for decoding the provided GstVideoCodecFrame and (ii) another one
for purging all decoded frames and submit them downstream.
Update plugin elements with the new GstVaapiVideoMeta API.
This also fixes support for subpictures/overlay because GstVideoDecoder
generates a sub-buffer from the GstVaapiVideoBuffer. So, that sub-buffer
is marked as read-only. However, when comes in the textoverlay element
for example, it checks whether the input buffer is writable. Since that
buffer read-only, then a new GstBuffer is created. Since gst_buffer_copy()
does not preserve the parent field, the generated buffer in textoverlay
is not exploitable because we lost all VA specific information.
Now, with GstVaapiVideoMeta information attached to a standard GstBuffer,
all information are preserved through gst_buffer_copy() since the latter
does copy metadata (qdata in this case).
Fix calculation of the time-out value for cases where no VA surface is
available for decoding. In this case, we need to wait until downstream
sink consumed at least one surface. The time-out was miscalculated as
it was always set to <current-time> + one second, which is not suitable
for streams with larger gaps.
Don't call gst_video_decoder_drop_frame() if gst_video_decoder_finish_frame()
was already called before and it returned an error. In that case, we were
releasing the frame again, thus leading to a "double-free" condition.
Maintain decoded surfaces as GstVideoCodecFrame objects instead of
GstVaapiSurfaceProxy objects. The latter will tend to be reduced to
the strict minimum: a context and a surface.
Make sure to push all decoded frames downstream as soon as possible.
This makes sure we don't need to wait for a new frame to be ready to
be decoded before receiving new decoded frames.
This also separates the decode process and the output process. The latter
could be moved to a specific GstTask later on.
Determine whether the buffer represents the top-field only by checking for
the GST_VIDEO_BUFFER_TFF flag instead of relying on the GstVaapiSurfaceProxy
flag. Also trust "interlaced" caps to determine whether the input frame
is interleaved or not.
Intermediate elements may produce a sub-buffer from a valid GstVaapiVideoBuffer
for non raw YUV cases. Make sure vaapipostproc now understands those buffers.
Intermediate elements may produce a sub-buffer from a valid GstVaapiVideoBuffer
for non raw YUV cases. Make sure vaapisink now understands those buffers.
Directly use the GstVideoCodecState associated with the VA decoder
instead of parsing caps again.
Signed-off-by: Sreerenj Balachandran <sreerenj.balachandran@intel.com>
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
Make vaapidecode derive from the standard GstVideoDecoder base element
class. This simplifies the code to the strict minimum for the decoder
element and makes it easier to port to GStreamer 1.x API.
Signed-off-by: Sreerenj Balachandran <sreerenj.balachandran@intel.com>
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
GstVaapiSurfaceProxy does not use any particular functionality from
GObject. Actually, it only needs a basic object type with reference
counting.
This is an API and ABI change.
Try to allocate the GstVaapiUploader helper object prior to listing the
supported image formats. Otherwise, only a single generic caps is output
with no particular pixel format referenced in there.
Use GstVaapiUploader helper that automatically handles direct rendering
mode, thus making the "direct-rendering" property obsolete and hence it
is now removed.
The "direct-rendering" level 2, i.e. exposing VA surface buffers, was never
really well supported and it could actually trigger degraded performance.
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
Make vaapisink expose only the set of supported caps for raw YUV buffers.
Add gst_vaapi_uploader_get_caps() helper function to determine the set
of supported YUV caps as source (for images). This function actually
tries to zero and upload each image to a 64x64 test surface. Of course,
this relies on VA drivers to not claim success if vaPutImage() is not
correctly supported.
Add new GstVaapiUploader helper to upload raw YUV buffers to VA surfaces.
It is up to the caller to negotiate source caps (for images) and output
caps (for surfaces). gst_vaapi_uploader_has_direct_rendering() is available
to help decide between the creation of a GstVaapiVideoBuffer or a regular
GstBuffer on sink pads.
Signed-off-by: Zhao Halley <halley.zhao@intel.com>
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
The use of heap allocated GMutex/GCond is deprecated. Instead place them
inside the structure they are locking.
These changes switch to use g_mutex_init/g_cond_init rather than the heap
allocation functions.
Because we cannot test for a NULL pointer for the GMutex/GCond we must
initialise inside the GObject _init function and clear inside the _finalize
which is guaranteed to only be called once and after the object is no longer
in use.
Don't care of the return value for gst_vaapi_decoder_put_buffer()
during destruction of the element. Don't print out (uninitialised)
error code when allocation of video buffer failed.
