Use standard GstVideoInfo related functions to build the output caps,
thus directly preserving additional fields as needed, instead of
manually copying them over through gst_vaapi_append_surface_caps().
Also ensure that the input caps are fixated first.
Add new helper functions to build video template caps.
- gst_vaapi_video_format_new_template_caps():
create GstCaps with size, frame rate and PAR to full range
- gst_vaapi_video_format_new_template_caps_from_list():
try to create a "simplified" list from the supplied formats
Add new helper functions to build GValues from GstVideoFormat:
- gst_vaapi_value_set_format():
build a GValue from the supplied video format
- gst_vaapi_value_set_format_list():
build a GValue list from the supplied array of video formats
Replace gst_vaapi_display_get_{decode,encode}_caps() APIs with more
more convenient APIs that return an array of GstVaapiProfile instead
of GstCaps: gst_vaapi_display_get_{decode,encode}_profiles().
Replace gst_vaapi_display_get_{image,subpicture}_caps() APIs, that
returned GstCaps, with more convenient APIs that return an array of
GstVideoFormat: gst_vaapi_display_get_{image,subpicture}_formats().
Makes the copies of a buffer reference their own GLTextureUploadMeta
user data and prevent the original buffer accessing already freed
memory if its copies has been released and freed.
https://bugzilla.gnome.org/show_bug.cgi?id=720336
[Propagate the original meta texture to the copy too]
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
Factor out propose_allocation() hooks, creation of video buffer pool
for the sink pad, conversion from raw YUV buffers to VA surface backed
buffers. Update vaapidecode, vaapiencode and vaapipostproc to cope
with the new GstVaapiPluginBase abilities.
Fix display creation code to check that any display obtained from a
neighbour actually has the type we expect. Note: if display type is
set to "any", we can then accept any VA display type.
Move common VA display creation code to GstVaapiPluginBase, with the
default display type remaining "any". Also add a "display-changed"
hook so that subclasses could perform additional tasks when/if the
VA display changed, due to a new display type request for instance.
All plug-ins are updated to cope with the new internal APIs.
Introduce a new GstVaapiPluginBase object that will contain all common
data structures and perform all common tasks. First step is to have a
single place to hold VA displays.
While we are at it, also make sure to store and subsequently release
the appropriate debug category for the subclasses.
The GLTextureUploadMeta implementation assumed that for each upload()
sequence, the supplied texture id is always the same as the one that
was previously cached into the underlying GstVaapiTexture. Cope with
any texture id change the expense to recreate the underlying VA/GLX
resources.
https://bugzilla.gnome.org/show_bug.cgi?id=719643
Requesting the GLTextureUpload meta on buffers in the bufferpool
prevents such metas from being de-allocated when buffers are released
in the sink.
This is particulary useful in terms of performance when using the
GLTextureUploadMeta API since the GstVaapiTexture associated with
the target texture is stored in the meta.
https://bugzilla.gnome.org/show_bug.cgi?id=712558
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
Make GstVideoGLTextureUploadMeta::upload() implementation more robust
when the GstVaapiTexture associated with the supplied texture id could
not be created.
Clean public APIs up so that to better align with the decoder APIs.
Most importantly, gst_vaapi_encoder_get_buffer() is changed to only
return the VA coded buffer proxy. Also provide useful documentation
for the public APIs.
Refactor the GstVaapiCodedBuffer APIs so that to more clearly separate
public and private interfaces. Besides, the map/unmap APIs should not
be exposed as is but appropriate accessors should be provided instead.
* GstVaapiCodedBuffer: VA coded buffer abstraction
- gst_vaapi_coded_buffer_get_size(): get coded buffer size.
- gst_vaapi_coded_buffer_copy_into(): copy coded buffer into GstBuffer
* GstVaapiCodedBufferPool: pool of VA coded buffer objects
- gst_vaapi_coded_buffer_pool_new(): create a pool of coded buffers of
the specified max size, and bound to the supplied encoder
* GstVaapiCodedBufferProxy: pool-allocated VA coded buffer object proxy
- gst_vaapi_coded_buffer_proxy_new_from_pool(): create coded buf from pool
- gst_vaapi_coded_buffer_proxy_get_buffer(): get underlying coded buffer
- gst_vaapi_coded_buffer_proxy_get_buffer_size(): get coded buffer size
Rationale: more optimized transfer functions might be provided in the
future, thus rendering the map/unmap mechanism obsolete or sub-optimal.
https://bugzilla.gnome.org/show_bug.cgi?id=719775
Fix GstElement::set_context() implementation for all plug-in elements
to avoid leaking an extra reference to the VA display, thus preventing
correct cleanup of VA resources in GStreamer 1.2 builds.
Return earlier if the creation of a VA display failed. Likewise, simplify
gst_vaapi_video_context_propagate() now that we are guaranteed to have a
valid VA display.
When GstVideoMeta maps were used, the supporting functions incorrectly
used gst_buffer_get_memory() instead of gst_buffer_peek_memory(), thus
always increasing the associated GstMemory reference count and giving
zero chance to actually release that, and subsequently the VA display.
Simplify GstVaapiVideoMeta to only hold a surface proxy, which is
now allocated from a surface pool. This also means that the local
reference to the VA surface is also gone, as it could be extracted
from the associated surface proxy.
Drop the following functions that are not longer used:
- gst_vaapi_video_buffer_new_with_surface()
- gst_vaapi_video_meta_new_with_surface()
- gst_vaapi_video_meta_set_surface()
- gst_vaapi_video_meta_set_surface_from_pool()
Fix gst_vaapi_video_meta_new_from_pool() to allocate VA surface proxies
from surface pools instead of plain VA surfaces. This is to simplify
allocations now that surface proxies are created from a surface pool.
Optimize gst_vaapiencode_handle_frame() to avoid extra memory allocation,
and in particular the GstVaapiEncObjUserData object. i.e. directly use
the VA surface proxy from the source buffer. This also makes the user
data attached to the GstVideoCodecFrame more consistent between both
the decoder and encoder plug-in elements.
Simplify gst_vaapiencode_push_frame(), while also removing the call
to gst_video_encoder_negotiate() since this is implicit in _finish()
if caps changed. Also fixed memory leaks that occured on error.
Constify pointers wherever possible. Drop unused variables, and use
consistent variable names. Fix gst_vaapiencode_h264_allocate_buffer()
to correctly report errors, especially when in-place conversion from
bytestream to avcC format failed.
Move "rate-control" mode and "bitrate" properties to the GstVaapiEncode
base class. The actual range of supported rate control modes is currently
implemented as a plug-in element hook. This ought to be determined from
the GstVaapiEncoder object instead, i.e. from libgstvaapi.
Align the plug-in debug category to its actual name. i.e. enable debug
logs through vaapiencode_<CODEC> where <CODEC> is mpeg2, h264, etc. Fix
the plug-in element description to make it more consistent with other
VA-API plug-ins.
Add a GST_VAAPIENCODE_CAST() helper to avoid run-time checks against
the GObject type system. We are guaranteed to only deal with the same
plug-in element object.
Allow vaapiencode plug-in elements to encode from raw YUV buffers.
The most efficient way to do so is to let the vaapiencode elements
allocate a buffer pool, and subsequently buffers from it. This means
that upstream elements are expected to honour downstream pools.
If upstream elements insist on providing their own allocated buffers
to the vaapiencode elements, then it possibly would be more efficient
to insert a vaapipostproc element before the vaapiencode element.
This is because vaapipostproc currently has better support than other
elements for "foreign" raw YUV buffers.
Add GstVaapiEncodeMPEG2 element object. The actual plug-in element
is called "vaapiencode_mpeg2".
Valid properties:
- rate-control: rate control mode (default: cqp - constant QP)
- bitrate: desired bitrate in kbps (default: auto-calculated)
- key-period: maximal distance between two key frames (default: 30)
- max-bframes: number of B-frames between I and P (default: 2)
- quantizer: constant quantizer (default: 8)
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
Add GstVaapiEncodeH264 element object. The actual plug-in element
is called "vaapiencode_h264".
Valid properties:
- rate-control: rate control mode (default: none)
- bitrate: desired bitrate in kbps (default: auto-calculated)
- key-period: maximal distance between two key frames (default: 30)
- num-slices: number of slices per frame (default: 1)
- max-bframes: number of B-frames between I and P (default: 0)
- min-qp: minimal quantizer (default: 1)
- init-qp: initial quantizer (default: 26)
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
Fix build when Wayland headers don't live in plain system include dirs
like /usr/include but rather in /usr/include/wayland for instance.
Original patch written by Dominique Leuenberger <dimstar@opensuse.org>
https://bugzilla.gnome.org/show_bug.cgi?id=712282
Destroy VPP output surface pool on exit. Also avoid a possible crash
in double-free situation caused by insufficiently reference counted
array of formats returned during initialization.
Fix advanced deinterlacing modes with VPP to track only up to 2 past
reference buffers. This used to be 3 past reference buffers but this
doesn't fit with the existing decode pipeline that only has 4 extra
scratch surfaces.
Also optimize references tracking to be only enabled when needed, i.e.
when advanced deinterlacing mode is used. This means that we don't
need to track past references for basic bob or weave deinterlacing.
In "mixed" interlaced streams, the buffer contains additional flags that
specify whether the frame contained herein is interlaced or not. This means
that we can alternatively get progressive or interlaced frames. Make sure
to disable deinterlacing at the VPP level when the source buffer is no longer
interlaced.
Fix memory leaks with advanced deinterlacing, i.e. when we keep track
of past buffers. Completely reset the deinterlace state, thus destroying
any buffer currently held, on _start(), _stop() and _destroy().
Port vaapipostproc element to GStreamer 1.2. Support is quite minimal
right now so that to cope with auto-plugging issues/regressions. e.g.
this happens when the correct set of expected caps are being exposed.
This means that, currently, the proposed caps are not fully accurate.
Fix basic deinterlacing flags provided to gst_vaapi_set_deinterlacing()
for the first field. Render flags were supplied instead of the actual
deinterlacing flags (deint_flags).
Fix GstBaseTransform::transform_caps() implementation to always return
the complete set of allowed sink pad caps (unfixated) even if the src
pad caps we are getting are fixated. Rationale: there are just so many
possible combinations, and it was wrong to provide a unique set anyway.
As a side effect, this greatly simplifies the ability to derive src pad
caps from fixated sink pad caps.
Fix deinterlacing flags to make more sense. The TFF (top-field-first)
flag is meant to specify the organization of reference frames used in
advanced deinterlacing modes. Introduce the more explicit flag TOPFIELD
to specify that the top-field of the supplied input surface is to be
used for deinterlacing. Conversely, if not set, this means that the
bottom field of the supplied input surface will be used instead.
There are situations where gst_video_decoder_flush() is called, and
this subsequently produces a gst_video_decoder_reset() that kills the
currently active GstVideoCodecFrame. This means that it no longer
exists by the time we reach GstVideoDecoder::finish() callback, thus
possibly resulting in a crash if we assumed spare data was still
available for decode (current_frame_size > 0).
Try to honour GstVideoDecoder::reset() behaviour from GStreamer 1.0
that means a flush, thus performing the actual operations there like
calling gst_video_decoder_have_frame() if pending data is available.
Review all interactions between the main video decoder stream thread
and the decode task to derive a correct sequence of operations for
decoding. Also avoid extra atomic operations that become implicit under
the GstVideoDecoder stream lock.
Fix hard reset for seek cases by flushing the GstVaapiDecoder queue
and completely purge any decoded output frame that may come out from
it. At this stage, the GstVaapiDecoder shall be in a complete clean
state to start decoding over new buffers.
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
vaapidecode used to wait up to one second past the expected time of
presentation for the last decoded frame. This is not realistic in
practice when it comes to video pause/resume. Changed behaviour to
unconditionnally wait for a free VA surface prior to continuing the
decoding. The decode task will continue pushing the output frames to
the downstream element while also reporting errors at the same time
to the main thread.
https://bugzilla.gnome.org/show_bug.cgi?id=707108
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
The srcpad caps exposed for GStreamer 1.2 were missing any useful info
like framerate, pixel-aspect-ratio, interlace-mode et al. Not to mention
that it relied on possibly un-initialized data. Fix srcpad caps to be
initialized from a sanitized copy of GstVideoDecoder output state caps.
Note: the correct way to expose the srcpad caps triggers an additional
issue in core GStreamer auto-plugging capabilities as the correct caps
to be exposed should be format=ENCODED with memory:VASurface caps feature
at the minimum. In some situations, we could determine the underlying
VA surface format, but this is not always possible. e.g. cases where it
is not allowed to expose the underlying VA surface data, or when the
VA driver implementation cannot actually provide such information.
Currently, the decoder only supports YUV 4:2:0 output. So, expose the
output formats for GStreamer 1.2 in caps to a realistic subset. This
means NV12, I420 or YV12 but also ENCODED if we cannot determine the
underlying VA surface format, or if it is actually not allowed to get
access to the surface contents.
Fix vaapidecode srcpad caps to only expose RGBA video format for the
meta:GstVideoGLTextureUploadMeta feature. That's only what is supported
so far. Besides, drop this meta from the vaapisink sinkpad caps since
we really don't support that for rendering.
https://bugzilla.gnome.org/show_bug.cgi?id=711828
Fix raw YUV data uploaded as in the following pipeline:
$ gst-launch-1.0 filesrc video.yuv ! videoparse ! vaapipostproc ! vaapisink
The main reason why it failed was that the videoparse element simply
allocates GstBuffer with raw data chunk'ed off the sink pad without
any prior knowledge of the actual frame info. i.e. it basically just
calls gst_adapter_take_buffer().
We could avoid the extra copy performed in vaapipostproc if the videoparse
element was aware of the downstream pool and bothers copying line by
line, for each plane. This means that, for a single frame per buffer,
the optimizatin will be to allocate the video buffer downstream, map
it, and copy each line that is coming through until we need to fills
in the successive planes.
Still, optimized raw YUV uploads already worked with the following:
$ gst-launch-1.0 videotestsrc ! vaapipostproc ! vaapisink
https://bugzilla.gnome.org/show_bug.cgi?id=711250
[clean-ups, fixed error cases to unmap and unref outbuf]
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
If the currently selected deinterlacing method is not supported by the
underlying hardware, then try to downgrade the method to a supported one.
At the minimum, basic bob-deinterlacing shall always be supported.
Allow basic bob-deinterlacing to work when VPP is enabled. Currently,
this only covers bob-deinterlacing when the output pixel format is
explicitly set.
Add initial support for basic scaling with size specified through the
"width" and "height" properties. If either user-provided dimension is
zero and "force-aspect-ratio" is set to true (the default), then the
other dimension is scaled to preserve the aspect ratio.
If VPP is available, we always try to implicitly convert the source
buffer to the "native" surface format for the underlying accelerator.
This means that no optimization is performed yet to propagate raw YUV
buffers to the downstream element as is, if VPP is available. i.e. it
will always cause a color conversion.
Even if we only support deinterlacing for now, use flags to specify
which filters are to be applied to each frame we receive in transform().
This is preparatory work for integrating new filters.
Add support for "mixed" interlace-mode, whereby the video frame buffer
shall be deinterlaced only if its flags mention that's actually an
interlaced frame buffer.
Reset the buffer pool allocator only if the config caps changed in a
sensible way: format or resolution change. i.e. don't bother with
other caps like colorimetry et al. as this doesn't affect the way to
allocate VA surfaces or images.
Enable read and write mappings only if direct-rendering is supported.
Otherwise, this means that we may need to download data from the VA
surface first for correctness, even if the VA surface doesn't need to
be read at all. i.e. sometimes, READWRITE mappings are meant for
surfaces that are written to first, and read afterwards for further
processing.
https://bugzilla.gnome.org/show_bug.cgi?id=704078
Fix check for direct-rendering if the creation of VA surfaces with
an explicit pixel format is not support, e.g. VA-API < 0.34.0, and
that we tried to allocate a VA surface based on the corresponding
chroma type. i.e. in that particular case, we have to make sure that
the derived image has actually the expected format.
Fix GstVaapiVideoBufferPool::reset_buffer() to reset the underlying
memory resources, and more particularly the VA surface proxy. Most
importantly, the GstVaapiVideoMeta is retained. Cached surface in
memory are released, thus triggering a new allocation the next time
we need to map the buffer.
Make sure GstVaapiVideoMemory allocates VA surface proxies from a
pool stored in the parent VA memory allocator.
This fixes the following scenario:
- VA video buffer 1 is allocated from a buffer pool
- Another video buffer is created, and inherits info from buffer 1
- Buffer 1 is released, thus pushing it back to the buffer pool
- New buffer alloc request comes it, this yields buffer 1 back
- At this stage, buffers 1 and 2 still share the same underlying VA
surface, but buffer 2 was already submitted downstream for further
processing, thus conflicting with additional processing we were
about to perform on buffer 1.
Maybe the core GstBufferPool implementation should have been fixed
instead to actually make sure that the returned GstBuffer memory we
found from the pool is writable?
Always make sure to allocate a VA surface proxy for GstVaapiUploader
allocated buffers, i.e. make gst_vaapi_uploader_get_buffer() allocate
a proxy surface.
This fixes cases where we want to retain the underlying surface longer,
instead of releasing it back to the surface pool right away.
Add gst_caps_set_interlaced() helper function that would reset the
interlace-mode field to "progressive" for GStreamer >= 1.0, or the
interlaced field to "false" for GStreamer 0.10.
Fix gst_vaapi_video_context_prepare() to also query upstream elements
for a valid GstContext. Improve comments regarding the steps used to
lookup or build that context, thus conforming to the GstContext API
recommendations.
https://bugzilla.gnome.org/show_bug.cgi?id=709112
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
If the allocation meta GST_VIDEO_GL_TEXTURE_UPLOAD_META_API_TYPE is
requested, and more specifically under a GLX configuration, then add
the GstVideoGLTextureUploadMeta to the output buffer.
https://bugzilla.gnome.org/show_bug.cgi?id=703236
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
Move VA video buffer memory from "video/x-surface,type=vaapi" format,
as expressed in caps, to the more standard use of caps features. i.e.
add "memory:VASurface" feature attribute to the associated caps.
https://bugzilla.gnome.org/show_bug.cgi?id=703271
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
Fix gst_vaapidecode_query() to correctly display the query type name,
instead of randomly displaying that we shared the underlying display.
Also add debug info for the GstVaapiSink::query() handler, i.e. the
supplied query type name actually.
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
Add support for the new GstContext API from GStreamer 1.2.x.
- implement the GstElement::set_context() hook ;
- reply to the `context' query from downstream elements.
https://bugzilla.gnome.org/show_bug.cgi?id=703235
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
Add thin compatibility layer for the deprecated GstVideoContext API.
For GStreamer API >= 1.2, this involves the following two functions:
- gst_vaapi_video_context_prepare(): queries if a context is already
set in the pipeline ;
- gst_vaapi_video_context_propagate(): propagates the newly-created
context to the rest of the pipeline.
https://bugzilla.gnome.org/show_bug.cgi?id=703235
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
Port vaapidecode and vaapisink plugins to GStreamer API >= 1.2. This
is rather minimalistic so that to test the basic functionality.
Disable vaapipostproc plugin for now as further polishing is needed.
Also disable GstVideoContext interface support since this API is now
gone in 1.2.x. This is preparatory work for GstContext support.
https://bugzilla.gnome.org/show_bug.cgi?id=703235
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
In GStreamer 0.10 builds, make sure that the GstVaapiUploader helper
is setup in case upstream elements allocate buffers themselves without
honouring our GstVaapiSink::bufer_alloc() hook.
In particular, this fixes support for OGG video streams with WebKit.
https://bugzilla.gnome.org/show_bug.cgi?id=703934
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
Handle raw video buffers that were not created from a VA video buffer
pool. Use the generic GstVideo API to copy buffers in GStreamer 1.0.x
builds instead of the GstVaapiUploader.
https://bugs.freedesktop.org/show_bug.cgi?id=55818
Fix _getcaps() implementation to not report codecs with size information
filled in the returned caps. That's totally useless nowadays. Ideally,
this is a hint to insert a video parser element, thus allowing future
optimizations, but this is not a strict requirement for gstreamer-vaapi,
which is able to parse the elementary bitstreams itself.
https://bugzilla.gnome.org/show_bug.cgi?id=704734
If there is no frame delimiter at the end of the stream, e.g. no
end-of-stream or end-of-sequence marker, and that the current frame
was fully parsed correctly, then assume that last frame is complete
and submit it to the decoder.
https://bugzilla.gnome.org/show_bug.cgi?id=705123
Signed-off-by: Guangxin.Xu <Guangxin.Xu@intel.com>
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
Fix creation of GstVaapiVideoBuffer objects (i) to have that type for real;
and (ii) to correctly extract the GstSurfaceConverter from the video buffer
object meta.
This fixes support for cluttersink with GStreamer 0.10 builds.
Other GStreamer sinks, like xvimagesink, have a force-aspect-ratio property,
which allows you to say that you don't want the sink to respect aspect
ratio. Add the same property to vaapisink.
http://lists.freedesktop.org/archives/libva/2012-September/001298.html
Signed-off-by: Simon Farnsworth <simon.farnsworth at onelan.co.uk>
Fix GstBaseSink::get_caps() implementation for GStreamer 1.0.X builds
by honouring the filter caps argument. More precisely, this fixes the
following pipeline: gst-launch-1.0 videotestsrc ! vaapisink
https://bugzilla.gnome.org/show_bug.cgi?id=705192
Signed-off-by: Guangxin.Xu <Guangxin.Xu@intel.com>
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
Add basic deinterlacing support, i.e. bob-deinterlacing whereby only
the selected field from the input surface is kept for the target surface.
Setting gst_vaapi_filter_set_deinterlacing() method argument to
GST_VAAPI_DEINTERLACE_METHOD_NONE means to disable deinterlacing.
Also move GstVaapiDeinterlaceMethod definition from vaapipostproc plug-in
to libgstvaapi core library.
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
Install a new video converter that supports X11 pixmap targets for X11
backends only, or make the GLX converter creation function chain up to
the X11 converter whenever requested.
After the code got moved to create the gst_vaapi_create_display() helper,
this comparison was not updated to dereference the newly-created
pointer, so the code was comparing the pointer itself to the type, and
therefore failing to retrieve the VA display.
This fixes the following error (and gets gst-vaapi decoding again):
ERROR vaapidecode gstvaapidecode.c:807:gst_vaapidecode_ensure_allowed_caps: failed to retrieve VA display
https://bugzilla.gnome.org/show_bug.cgi?id=704410
Signed-off-by: Emilio López <emilio@elopez.com.ar>
Fix new internal video format API, based on GstVideoFormat, to not
clobber with system symbols. So replace the gst_video_format_* prefix
with gst_vaapi_video_format_ prefix, even if the format type remains
GstVideoFormat.
Simplify gst_vaapi_create_display() helper as gst_vaapi_display_XXX_new()
performs the necessary validation checks for the underlying VA display
prior to returning to the caller. So, if an error occurred, then NULL is
really returned in that case.
If the video buffer pool config doesn't have new caps, then it's not
necessary to reinstantiate the allocator. That could be a costly
operation as we could do some extra heavy checking in there.
Fix reference counting issue whereby gst_memory_init() does not hold
an extra reference to the GstAllocator. So, there could be situations
where the last instance of GstVaapiVideoAllocator gets released before
a dangling GstVaapiVideoMemory object, thus possibly leading to a crash.
Always perform conversion of sources buffers to NV12 since this is
the way we tested for this capability in ensure_allowed_caps(). This
also saves memory bandwidth for further rendering. However, this may
not preserve quality since the YUV buffers are down-sampled to 4:2:0.
This fixes direct linking of vaapidownload element to xvimagesink with
VA drivers supporting vaGetImage() from the native VA surface format to
a different VA image format. i.e. color conversion during download.
http://bugzilla.gnome.org/show_bug.cgi?id=703937
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
The image is now expressed as a standard GstVideoFormat, which is not
a FOURCC but rather a regular enum value.
This is a regression introduced in commit 09397fa.
Fix gst_vaapi_uploader_get_buffer() to not assign caps since they
were already negotiated beforehand, and they are not used from the
buffer in upstream elements.
Clean-up gst_vaapi_uploader_ensure_caps() to use the new image caps
represented as a GstVideoInfo.
Adapt GstVaapiVideoMemory allocator to support creation of VA surfaces
with an explicit pixel format. This allows for direct rendering to
VA surface memory from a software decoder.
Get rid of GstCaps to create surface/image pool, and use GstVideoInfo
structures instead. Those are smaller, and allows for streamlining
libgstvaapi more.
Fix creation of GLX texture, to not depend on the GstCaps video size that
could be wrong, especially in presence of frame cropping. So, use the size
from the source VA surfaces.
An optimization could be to reduce the texture size to the actual visible
size on screen. i.e. scale down the texture size to match the screen dimensions,
while preserving the VA surface aspect ratio. However, some VA drivers don't
honour that.
Add support for GstVideoCropMeta in GStreamer >= 1.0.x builds and gst-vaapi
specific meta information to hold video cropping details. Make the sink
support video cropping in X11 and GLX modes.
Some video clips may have a clipping region that needs to propogate to
the renderer. These helper functions make it possible to attach that
clipping region, as a GstVaapiRectangle, the the video meta associated
with the buffer.
Signed-off-by: Sreerenj Balachandran <sreerenj.balachandran@intel.com>
signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
Expose all raw video formats in the static caps template since the
vaapisink is supporting raw data. We will get the exact set of formats
supported by the driver dynamically through the _get_caps() routine.
This also fixes an inconsistency wrt. GStreamer 0.10 builds.
https://bugzilla.gnome.org/show_bug.cgi?id=702178
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
Now that VA/GLX capable buffers are generated by default on X11, thus
depending on a VA/GLX display, we stil want to use vaPutSurface() for
rendering since it is faster.
Anyway, OpenGL rendering in vaapisink was only meant for testing and
enabling "fancy" effects to play with. This has no real value. So,
disable OpenGL rendering by default.
If the gstreamer-vaapi plug-in elements are built with GLX support, then
try to allocate a GstVaapiDisplayGLX first before resorting to a VA/X11
display next.
https://bugzilla.gnome.org/show_bug.cgi?id=701742
Allow plain gst_buffer_map() interface to work with gstreamer-vaapi
video buffers, i.e. expose the underlying GstVaapiSurfaceProxy to the
caller. This is the only sensible enough thing to do in this mode as
the underlying surface pixels need to be extracted through an explicit
call to the gst_video_frame_map() function instead.
A possible use-case of this is to implement a "handoff" signal handler
to fakesink or identity element for further processing.
Fix gst_vaapi_video_allocator_new() to silently check for direct-rendering
mode support, and not trigger fatal-criticals if either test surface or
image could not be created. Typical case: pixel format mismatch, e.g. NV12
supported by most hardware vs. I420 supported by most software decoders.
On map, ensure we have GST_MAP_WRITE flags since this is only what we
support for now. Likewise, on unmap, make sure that the VA image is
unmapped for either read or write, while still committing it to the
VA surface if write was requested.
In GStreamer 0.10 builds, gst_vaapi_uploader_get_buffer() was used
but it exhibited a memory leak because the surface generated for the
GstVaapiVideoMeta totally lost its parent video pool. So, it was not
possible to release that surface back to the parent pool when the meta
gets released, and the memory consumption kept growing.
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
Since GST_VAAPI_IS_xxx_VIDEO_POOL() was only testing for NULL and not
the underlying object type, the gst_vaapi_video_meta_new_from_pool()
was hereby totally broken. Fixed this regression by using the newly
provided gst_vaapi_video_pool_get_object_type() function.
Add gst_vaapi_decoder_get_frame_with_timeout() helper function that will
wait for a frame to be decoded, until the specified timeout in microseconds,
prior to returning to the caller.
This is a fix to performance regression from 851cc0, whereby the vaapidecode
loop executed on the srcpad task was called to often, thus starving all CPU
resources.
Rework heuristics to detect when downstream element ran into errors,
and thus failing to release any VA surface in due time for the current
frame to get decoded. In particular, recalibrate the render time base
when the first frame gets submitted downstream, or when there is no
timestamp that could be inferred.
Rework GstVideoDecoder::handle_frame() to decode the current frame,
while possibly waiting for a free surface, and separately submit all
decoded frames from a task. This makes it possible to pop and render
decoded frames as soon as possible.
Fix reference counting bug for passthrough mode, whereby the input buffer
was propagated as is downstream through gst_pad_push() without increasing
its reference count before. The was a problem when gst_pad_push() returns
an error and we further decrease the reference count of the input buffer.
Add support for interlaced streams with GStreamer 1.0 too. Basically,
this enables vaapipostproc, though it is not auto-plugged yet. We also
make sure to reply to CAPS queries, and happily handle CAPS events.
Make gst_vaapi_decoder_get_codec_state() return the original codec state,
i.e. make the GstVaapiDecoder object own the return state so that callers
that want an extra reference to it would just gst_video_codec_state_ref()
it before usage. This aligns the behaviour with what we had before with
gst_vaapi_decoder_get_caps().
This is an ABI incompatible change, library major version was bumped from
previous release (0.5.2).
Mark the following functions are internal, i.e. private to the vaapi plug-in:
- gst_vaapi_video_buffer_pool_get_type()
- gst_vaapi_video_converter_glx_get_type()
- gst_vaapi_video_converter_glx_new()
Implement GstSurfaceMeta API for GStreamer 1.0.x. Even though this is
an unstable/deprecated API, this makes it possible to support Clutter
sink with minimal changes. Tested against clutter-gst 1.9.92.
When render-mode is "overlay", then it is not really useful to peek into
the GstBaseSink::last_buffer, since we have our own video_buffer already
recorded and maintained into GstVaapiSink.
Fix memory leak of GstSample objects in GstVideoOverlayInterface::expose().
This also fixes extra unreferencing of the underlying GstBuffer in the common
path afterwards (for both 0.10 or 1.0).
Fix the name of the plug-in element reported to gst-inspect-1.0. i.e. we
need an explicit definition for GStreamer >= 1.0 because the GST_PLUGIN_DEFINE
incorrectly uses #name for creating the plug-in name, instead of using macro
expansion (and let further expansion of macros) through e.g. G_STRINGIFY().
Fix make dist to allow build for either GStreamer 0.10 or 1.0. i.e. make
sure to include all source files in either case while generating source
tarballs.
Implement GstVideoMeta::{,un}map() to support raw YUV buffer upload when
the last component is unmapped. Downloads are not supported yet. The aim
was to first support SW decoding + HW accelerated rendering (vaapisink).
e.g. for Wayland.
Handle GST_QUERY_CAPS, which is the GStreamer 1.0 mechanism to retrieve
the set of allowed caps, i.e. it works similar to GstPad::get_caps().
This fixes fallback to SW decoding if no HW decoder is available.
Introduce a new configure option --with-gstreamer-api that determines
the desired GStreamer API to use. By default, GStreamer 1.0 is selected.
Also integrate more compatibility glue into gstcompat.h and plugins.
Use new GstVaapiVideoBufferPool to maintain video buffers. Implement
GstBaseSink::propose_allocation() to expose that pool to upstream
elements; and also implement GstVideoDecoder::decide_allocation() to
actually use that pool (from downstream), if any, or create one.
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
Add initial support for GstVaapiVideoMemory backed buffer pool. The memory
object currently holds a reference to GstVaapiVideoMeta.
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
Make it possible to copy GstVaapiVideoMeta objects, unless they contain VA
objects created from GstVaapiVideoPool. This is mostly useful to clone a
GstVaapiVideoMeta object containing a VA surface proxy so that to alter its
rendering flags.
Fix GstVaapiVideoMeta to allow VA objects to be destroyed when they are
reset to NULL. i.e. make gst_vaapi_video_meta_set_{image,surface}() and
gst_vaapi_video_meta_set_surface_proxy() actually clear VA objects when
argument is NULL.
Port vaapidecode and vaapisink plugins to GStreamer API >= 1.0. This
is rather minimalistic so that to test the basic functionality.
Disable vaapiupload, vaapidownload and vaapipostproc plugins. The latter
needs polishing wrt. to GStreamer 1.x functionality and the former are
totally phased out in favor of GstVaapiVideoMemory map/unmap facilities,
which are yet to be implemented.
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
Improve check for raw YUV format modes by avoiding checks against strings
("video/x-raw-yuv") for each new GstBuffer allocation. In the usual case,
GstBaseSink::set_caps() is called first and if VA surface format mode is
used, then GstBaseSink::buffer_alloc() is not called. If the latter is
called before set_caps(), then we just make a full check. This one is
pretty rare though, e.g. it usually happens once for custom pipelines.
Fix gst_vaapi_apply_composition() to not fail if no overlay composition
was found. i.e. return success (TRUE). This was harmless though extra
debug messages are not nice.
This is a regression introduced by commit 95b8659.
Don't return static caps that don't mean anything for the underlying codecs
that are actually supported for decoding. i.e. always allocate a VA display
and retrieve the exact set of HW decoders available. That VA display may be
re-used later on during negotiation through GstVideoContext "prepare-context".
This fixes fallback to SW decoding if no HW decoder is available.
Make gst_vaapi_reply_to_query() first check whether the query argument
is actually a video-context query, i.e. with type GST_QUERY_TYPE_CUSTOM.
Then, make sure vaapisink propagates the query to the parent class if
it is not a video-context query.
Add new gst_vaapi_video_buffer_new() helper function that allocates a video
buffer from a GstVaapiVideoMeta. Also remove obsolete and useless function
gst_vaapi_video_buffer_get_meta().
Move GstVaapiVideoMeta from core libgstvaapi decoding library to the
actual plugin elements. That's only useful there. Also inline reference
counting code from GstVaapiMiniObject.
Make sure libgstvaapi core decoding library doesn't include un-needed
dependencies. So, move out GstVaapiVideoConverterGLX to plugins instead.
Besides, even if the vaapisink element is not used, we are bound to have
a correctly populated GstSurfaceBuffer from vaapidecode.
Also clean-up the file along the way.
If the raw YUV buffer was created from vaapisink, through the buffer_alloc()
hook, then it will have a valid GstVaapiVideoMeta object attached to it.
However, we previously assumed in that case that it was a "native" VA buffer,
thus not calling into GstVaapiUploader::process().
Use gst_element_class_set_static_metadata() from GStreamer 1.0, which
basically is the same as gst_element_class_set_details_simple() in
GStreamer 0.10 context.
Move GstImplementsInterface and GstVideoContext support functions up
so that to keep a clear separation between the plugin element and its
interface hooks.
Use GstVideoInfo and gst_video_info_from_caps() helper wherever possible.
Also use the newly added gst_vaapi_image_format_from_structure() helper
in GstVaapiUploader::ensure_allowed_caps().
gst_vaapi_video_buffer_new_from_buffer() needs to reference the source
buffer video meta since it would be unreference'd from the get_buffer()
helper function. For other cases, we still use (steal) the newly created
video meta.
Fix ensure_image() to only zero-initialize the first line of each plane.
Properly initializing each plane to their full vertical resolution would
require to actually compute it based on the image format.
In particular, for NV12 images, the UV plane has half vertical resolution
vs. the Y plane. So using the full image height to initialize the UV plane
will obviously lead to a buffer overflow. Likewise for other YUV format.
Since ensure_image() is only a helper function to initialize something,
and not necessarily the whole thing, it is fine to initializ the first
line only. Besides, the target surface is not rendered either.
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
Decode-only frames may not have a valid surface proxy. So, simply discard
them gracefully, i.e. don't create meta data information. GstVideoDecoder
base class will properly handle this case and won't try to push any buffer
to downstream elements.
Implement GstVideoDecoder::reset() as a destruction of the VA decoder
and the creation of a new VA decoder.
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
Split GstVideoDecoder::handle_frame() implementation into two functions:
(i) one for decoding the provided GstVideoCodecFrame and (ii) another one
for purging all decoded frames and submit them downstream.
Update plugin elements with the new GstVaapiVideoMeta API.
This also fixes support for subpictures/overlay because GstVideoDecoder
generates a sub-buffer from the GstVaapiVideoBuffer. So, that sub-buffer
is marked as read-only. However, when comes in the textoverlay element
for example, it checks whether the input buffer is writable. Since that
buffer read-only, then a new GstBuffer is created. Since gst_buffer_copy()
does not preserve the parent field, the generated buffer in textoverlay
is not exploitable because we lost all VA specific information.
Now, with GstVaapiVideoMeta information attached to a standard GstBuffer,
all information are preserved through gst_buffer_copy() since the latter
does copy metadata (qdata in this case).
Fix calculation of the time-out value for cases where no VA surface is
available for decoding. In this case, we need to wait until downstream
sink consumed at least one surface. The time-out was miscalculated as
it was always set to <current-time> + one second, which is not suitable
for streams with larger gaps.
Don't call gst_video_decoder_drop_frame() if gst_video_decoder_finish_frame()
was already called before and it returned an error. In that case, we were
releasing the frame again, thus leading to a "double-free" condition.
Maintain decoded surfaces as GstVideoCodecFrame objects instead of
GstVaapiSurfaceProxy objects. The latter will tend to be reduced to
the strict minimum: a context and a surface.
Make sure to push all decoded frames downstream as soon as possible.
This makes sure we don't need to wait for a new frame to be ready to
be decoded before receiving new decoded frames.
This also separates the decode process and the output process. The latter
could be moved to a specific GstTask later on.
Determine whether the buffer represents the top-field only by checking for
the GST_VIDEO_BUFFER_TFF flag instead of relying on the GstVaapiSurfaceProxy
flag. Also trust "interlaced" caps to determine whether the input frame
is interleaved or not.
Intermediate elements may produce a sub-buffer from a valid GstVaapiVideoBuffer
for non raw YUV cases. Make sure vaapipostproc now understands those buffers.
Intermediate elements may produce a sub-buffer from a valid GstVaapiVideoBuffer
for non raw YUV cases. Make sure vaapisink now understands those buffers.
Directly use the GstVideoCodecState associated with the VA decoder
instead of parsing caps again.
Signed-off-by: Sreerenj Balachandran <sreerenj.balachandran@intel.com>
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
Make vaapidecode derive from the standard GstVideoDecoder base element
class. This simplifies the code to the strict minimum for the decoder
element and makes it easier to port to GStreamer 1.x API.
Signed-off-by: Sreerenj Balachandran <sreerenj.balachandran@intel.com>
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
GstVaapiSurfaceProxy does not use any particular functionality from
GObject. Actually, it only needs a basic object type with reference
counting.
This is an API and ABI change.
Try to allocate the GstVaapiUploader helper object prior to listing the
supported image formats. Otherwise, only a single generic caps is output
with no particular pixel format referenced in there.
Use GstVaapiUploader helper that automatically handles direct rendering
mode, thus making the "direct-rendering" property obsolete and hence it
is now removed.
The "direct-rendering" level 2, i.e. exposing VA surface buffers, was never
really well supported and it could actually trigger degraded performance.
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
Make vaapisink expose only the set of supported caps for raw YUV buffers.
Add gst_vaapi_uploader_get_caps() helper function to determine the set
of supported YUV caps as source (for images). This function actually
tries to zero and upload each image to a 64x64 test surface. Of course,
this relies on VA drivers to not claim success if vaPutImage() is not
correctly supported.
Add new GstVaapiUploader helper to upload raw YUV buffers to VA surfaces.
It is up to the caller to negotiate source caps (for images) and output
caps (for surfaces). gst_vaapi_uploader_has_direct_rendering() is available
to help decide between the creation of a GstVaapiVideoBuffer or a regular
GstBuffer on sink pads.
Signed-off-by: Zhao Halley <halley.zhao@intel.com>
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
The use of heap allocated GMutex/GCond is deprecated. Instead place them
inside the structure they are locking.
These changes switch to use g_mutex_init/g_cond_init rather than the heap
allocation functions.
Because we cannot test for a NULL pointer for the GMutex/GCond we must
initialise inside the GObject _init function and clear inside the _finalize
which is guaranteed to only be called once and after the object is no longer
in use.
Don't care of the return value for gst_vaapi_decoder_put_buffer()
during destruction of the element. Don't print out (uninitialised)
error code when allocation of video buffer failed.
Reset, i.e. destroy then create, the decoder in _setcaps() handler only
if the underlying codec type actually changed. This makes it possible
to be more tolerant with certain MPEG-2 streams that get parsed to
form caps that are compatible with the previous state but minor changes
to "codec-data".
Add new gst_vaapi_codec_from_caps() helper to determine codec type from
the specified caps. Don't globally expose this function since this is
really trivial and only used in the vaapidecode element.
Previously, vaapidecode would wait up to one second until a free surface
is available, or it aborts decoding. Now, vaapidecode waits until the
last decoded surface was to be presented, plus one second. Besides, end
times are now expressed relative to the monotonic clock.
When playback stops the GstVaapiDecode object is reset into a clean
state. However, surfaces may still be referenced by library users and
unreferencing them after the reset triggers an access to an unset mutex.
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
Forward declaring enums is not allowed by the C standard and aborts
compilation if the header file is included in a C++ project.
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
If either dimension is out-of-bounds, then scale window to fit the
display size, even if the output is to be rotated. Use the standard
gst_video_sink_center_rect() function to center and scale the window
wrt. the outer (display) bounds.
Keep VA surface proxy associated with the surface that is currently
being displayed. This makes sure that surface is not released back
to the pool of surfaces free to use for decoding. This is necessary
with VA driver implementations that support rendering to an overlay
pipe. Otherwise, there could be cases where we are decoding into a
surface that is being displayed, hence some flickering.
This is not useful in practice but for raw performance evaluation when
the sink is invoked with display=drm sync=false. fakesink could also be
used though.
If vaapisink is in the GStreamer pipeline, then we shall allocate a
unique GstVaapiDisplay and propagate it upstream. i.e. subsequent
queries from vaapidecode shall get a valid answer from vaapisink.
Move display types from gstvaapipluginutil.* to gstvaapidisplay.* so that
we could simplify characterization of a GstVaapiDisplay. Also rename "auto"
type to "any", and add a "display-type" attribute.
vaapisink is now built with support for multiple display types, whenever
they are enabled. The new "display" attribute is used to select a particular
renderer.
This flag is obsolete. It was meant to explicitly enable/disable VA/GLX API
support, or fallback to TFP+FBO if this API is not found. Now, we check for
the VA/GLX API by default if --enable-glx is set. If this API is not found,
we now default to use TFP+FBO.
Note: TFP+FBO, i.e. using vaPutSurface() is now also a deprecated usage and
will be removed in the future. If GLX rendering is requested, then the VA/GLX
API shall be used as it covers most usages. e.g. AMD driver can't render to
an X pixmap yet.
GStreamer -base plugins >= 0.10.31 are now required, so the checks for
new APIs like GstXOverlay::set_window_handle() and ::set_render_rectangle()
are no longer necessary.
GStreamer codecparsers-based decoders are the only supported decoders now.
Though, FFmpeg decoders are still available in gstreamer-vaapi 0.3.x series.
Bump GStreamer plugins -base required version to 0.10.31, needed for
gst_x_overlay_got_window_handle().
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
Bump GStreamer required version to 0.10.14, needed for
gst_element_class_set_details_simple().
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
Fix typo whereby plain VADisplay type was used instead of the GstVaapiDisplay
wrapper.
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
Try to gracefully abort when the HW does not support the requested
profile. There is no fallback unless profiles are correctly parsed
and matched through caps beforehand.
Don't forcibly resize foreign X windows. The user is responsible for
their size and vaapisink shall not change this.
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
Fix gst_vaapisink_xoverlay_set_window_handle() when it is called before
caps got negotiated. Besides, when a foreign window is provided by the
user, so should the render rect.
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
Add new "interlaced" attribute to GstVaapiSurfaceProxy. Use this in
vaapipostproc so that to handles cases where bitstream is interlaced
but almost only frame pictures are generated. In this case, we should
not be alternating between top/bottom fields.
Add vaapipostproc element for video postprocessing. So far, only basic
bob deinterlacing is implemented. Interlaced mode is automatically
detected based on sink caps ("interlaced" field).
Allow rendering flags, as a combination of GstVaapiSurfaceRenderFlags,
to be set to the video buffer. In particular, this is mostly useful for
basic deinterlacing.
Rationale: playbin2 links all elements at run-time. Once vaapidecode
is created and a NEWSEGMENT event arrives, downstream element may not
be ready yet. So, delay this event until next element is chained in,
otherwise basesink could output "Received buffer without a new-segment.
Assuming timestamps start from 0".
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
Propagate "interlaced" caps downstream and set "tff" buffer flag
appropriately to output buffers for interlaced pictures.
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
This ensures the display name provided to gst_vaapi_display_*_new()
maps to the system defaults, instead of forcing "" that could be different
from the current DISPLAY name.
Otherwise, the decoder would always create its own X display instead
of probing it from the downstream element, which is not reliable.
e.g. DISPLAY is not :0 or when running on Wayland.
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
With the new video/x-surface abstraction, we can't rely on having a VA
specific sink downstream. Also, there was no particular reason to do that.
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
This new interface allows for upstream and downstream display sharing
that works in both static and dynamic pipelines.
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
