Before pushing a the new frame, the render() method calls sync() to flush the
pending frames. Nonetheless, the last pushed frame never gets rendered, leading
to a memory leak too.
This patch calls sync() in the destroy() to flush the pending frames before
destroying the window.
Also a is_cancelled flag is added. This flag tells to not flush the event
queue again since the method failed previously or were cancelled by the user.
https://bugzilla.gnome.org/show_bug.cgi?id=749078
Otherwise wl_display_dispatch_queue() might prevent the pipeline from
shutting down. This can happen e.g. if the wayland compositor exits while
the pipeline is running.
Changes:
* renamed unlock()/unlock_stop() to unblock()/unblock_cancel() in gstvaapiwindow
* splitted the patch removing wl_display_dispatch_queue()
Signed-off-by: Víctor Manuel Jáquez Leal <victorx.jaquez@intel.com>
https://bugzilla.gnome.org/show_bug.cgi?id=747492https://bugzilla.gnome.org/show_bug.cgi?id=749078
wl_display_dispatch_queue() might prevent the pipeline from shutting
down. This can happen e.g. if the wayland compositor exits while the
pipeline is running.
This patch replaces it with these steps:
- With wl_display_prepare_read() all threads announce their intention
to read.
- wl_display_read_events() is thread save. On threads reads, the other
wait for it to finish.
- With wl_display_dispatch_queue_pending() each thread dispatches its
own events.
wl_display_dispatch_queue_pending() was defined since wayland 1.0.2
Original-patch-by: Michael Olbrich <m.olbrich@pengutronix.de>
* stripped out the unlock() unlock_stop() logic
* stripped out the poll handling
Signed-off-by: Víctor Manuel Jáquez Leal <victorx.jaquez@intel.com>
https://bugzilla.gnome.org/show_bug.cgi?id=749078https://bugzilla.gnome.org/show_bug.cgi?id=747492
Since frame in the private data means the last frame sent, it would
semantically better use last_frame.
Also, this patch makes use of g_atomic_pointer_{compare_and_exchange, set}()
functions.
https://bugzilla.gnome.org/show_bug.cgi?id=749078
Wayland window has a pointer to the last pushed frame and use it to set the
flag for stopping the queue dispatch loop. This may lead to memory leaks,
since we are not keeping track of all the queued frames structures.
This patch removes the last pushed frame pointer and change the binary flag
for an atomic counter, keeping track of number of queued frames and use it for
the queue dispatch loop.
https://bugzilla.gnome.org/show_bug.cgi?id=749078
This patch takes out the wayland's buffer from the the frame structure. The
buffer is queued to wayland and destroyed in the "release" callback. The
frame is freed in the surface's "done" callback.
In this way a buffer may be leaked but not the whole frame structure.
- surface 'done' callback is used to throttle the rendering operation and to
unallocate the frame, but not the buffer.
- buffer 'release' callback is used to destroy wl_buffer.
Original-patch-by: Zhao Halley <halley.zhao@intel.com>
* code rebase
* kept the the event_queue for buffer's proxy
Signed-off-by: Víctor Manuel Jáquez Leal <victorx.jaquez@intel.com>
https://bugzilla.gnome.org/show_bug.cgi?id=749078
This patch fixes several issues found when running the `make distcheck`
target:
- In commit c561b8da, the update of gstcompat.h in Makefile.am was
forgotten.
- In commit c5756a91 add the simple_encoder_source_h in EXTRA_DIST was
forgotten.
- vpx.build.stamp is not generated at all, only vpx.configure.stamp.
- The make target distcleancheck failed because some autogenerated files
were not handled with the DISTCLEANFILES variable.
Note: `make distcheck -jXX` is not currently supported.
GST_VAAPI_ENCODER_STATUS_NO_SURFACE and GST_VAAPI_ENCODER_STATUS_NO_BUFFER
are not errors, so they do not have the ERROR namespace.
This patch fixes this typo in documentation.
On s390x, guintptr and GstVaapiID are not compatible types. The
implementation of gst_vaapi_window_new_internal() and all its callers
seem to assume that its third argument is a GstVaapiID, while the
header gives it guintptr type.
https://bugzilla.gnome.org/show_bug.cgi?id=744559
Since bug #745728 was fixed the oldest supported version of GStreamer is
1.2. That GStreamer release requires glib 2.32, so we can upgrade our
requirement too.
This patch changes the required version of glib in configure.ac and removes
the hacks in glibcompat.h
https://bugzilla.gnome.org/show_bug.cgi?id=748698
This patch only intends to improve readability: in the method
gst_vaapi_window_wayland_sync() the if/do instructions are squashed into a
single while loop.
Also renames the frame_redraw_callback() callback into frame_done_callback(),
which is a bit more aligned to Wayland API.
The Wayland compositor may still use the buffer when the frame done
callback is called.
This patch destroys the frame (which contains the buffer) until the
release callback is called. The draw termination callback only controls
the display queue dispatching.
Signed-off-by: Víctor Manuel Jáquez Leal <vjaquez@igalia.com>
https://bugzilla.gnome.org/show_bug.cgi?id=747492
Based up on the value of uniform_spacing_flag in Picture Parameter Set,
the tile column width and tile row height should be calculated.
Equations: 6-1, 6-2
Tiled video Descriptions: 7.3.2.3, 7.4.3.3
-- Set NoRaslOutputFlag based on EOS and EOB Nal units
-- Fix PicOutputFlag setting for RASL picture
-- Fix prev_poc_lsb/prev_poc_msb calculation
-- Drop the RASL frames if NoRaslOutputFlag is TRUE for the associated IRAP picture
-- Fixed couple of crashes and added cosmetics
There is a race condition where g_drm_device_type can be left set to
DRM_DEVICE_RENDERNODES when it shouldn't.
If thread 1 comes in and falls into the last else statement setting up both
RENDERNODES and LEGACY types. And begins to process the first type (RENDERNODES),
it sets g_drm_device_type = RENDERNODES.
Now when thread 2 comes in and sees g_drm_device_type is RENDERNODES, it queues
up that type to be tried but then encounters the lock and has to wait until the
first thread finishes. Once the lock is acquired it will then proceed to ONLY try
RENDERNODES and fail it. But it doesn't try LEGACY. And from then on, all future
attempts will only try RENDERNODES.
So to avoid this situation I have simply moved the acquisition of the lock higher
up in the attached patch.
https://bugzilla.gnome.org/show_bug.cgi?id=747914
The video pool can be accessed with the display lock held, for example,
when releasing a buffer from inside vaapisink_render, but allocating
a new object can may also take the display lock. Which means a possible
deadlock.
https://bugzilla.gnome.org/show_bug.cgi?id=747944
The support for buffer exports in VA-API was added in version 0.36. These
interfaces are for interop with EGL, OpenCL, etc.
GStreamer-VAAPI uses it for a dmabuf memory allocator. Though, gstreamer-vaapi
has to support VA-API versions ranging from 0.30.4, which doesn't support it.
This patch guards all the buffer exports handling (and dmabuf allocator) if
the detected VA-API version is below 0.36.
https://bugzilla.gnome.org/show_bug.cgi?id=746405
The member size in GstMpeg4Packet is gsize which is unsigned, which cannot be
less than zero. Hence this pre-condition test is a no-op. This patch removes
that code.
https://bugzilla.gnome.org/show_bug.cgi?id=747312
slice_type in slice_param is defined as (char *), but it is compared against a
signed integer. clang complains about this comparison.
This patch casts the variable.
https://bugzilla.gnome.org/show_bug.cgi?id=747312
The symbol GstVaapiCodedBuffer is already defined in
gst-libs/gst/vaapi/gstvaapicodedbuffer.h which is loaded, at the end, by
gstvaapiencoder_objects.h. Clang complains about the symbol re-definition.
This patch removes that redefinition.
https://bugzilla.gnome.org/show_bug.cgi?id=747312
The member value in frame_rate_tab is float, the result of the abs() function
should be float too. But abs() only manages integers.
This patch replaces abs() with fabsf() to handle correctly the possible floats
values.
https://bugzilla.gnome.org/show_bug.cgi?id=747312
The purpose of gstcompat.h is to couple the API differences among
gstreamer-1.0 and gstreamer-0.10. Since gstreamer-0.10 is obsolete, the code
in this compatibility layer shall be removed.
Nevertheless, the gstcompat.h header should be kept, if new incompatibilites
appear in the future, but it shall live in gst/vaapi, not in gst-libs.
This patch removes the crumbs defined gstcompat.h and moves it to gst/vaapi.
In order to avoid layer violations, gstcompat.h includes sysdeps.h and all
the includes in gst/vaapi of sysdeps.h are replaced with gstcompat.h
https://bugzilla.gnome.org/show_bug.cgi?id=745728
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
Signed-off-by: Sreerenj Balachandran <sreerenj.balachandran@intel.com>
This library was intended to add the base classes for video decoders which
where not included in gstreamer-0.10.
Since the support of gstreamer-0.10 is deprecated those classes are not
required, thus the whole library is removed.
https://bugzilla.gnome.org/show_bug.cgi?id=745728https://bugzilla.gnome.org/show_bug.cgi?id=732666
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
This patch only removes the support of gstreamer-0.10 in the autotools
scripts. No other files are touched.
The configuration parameter --gstreamer-api was deleted since now it is always
auto-detected.
The verification of vmethod query in GstBaseSinkClass was removed since it was
added in gstreamer 0.10.35. The same case for GstVideoOverlayComposition and
its format flags.
The precious variable GST_PLUGIN_PATH was removed, while GST_PLUGIN_PATH_1_0
remained.
The automake files were changed accordingly.
Removed, in debian/control, the vaapiupload and vaapidownload descriptions.
https://bugzilla.gnome.org/show_bug.cgi?id=732666https://bugzilla.gnome.org/show_bug.cgi?id=745728
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
Add support for H.264 MVC Multiview High profile encoding with
more than 2 views. All views within the same accesss unit are
provided in increasing order of view order index (VOIdx).
Upto 10 view are supported for now.
A new property "view-ids" has been provided for the plugins to
set the view ids (which is an array of guint values) to be used
for mvc encoding.
https://bugzilla.gnome.org/show_bug.cgi?id=732453
Add support for GstVideoGLTextureOrientation modes. In particular,
add orientation flags to the GstVaapiTexture wrapper and the GLX
implementations. Default mode is that texture memory is laid out
with top lines first, left row first. Flags indicate whether the
X or Y axis need to be inverted.
Add GstVaapiTextureEGL abstraction that can create its own GL texture,
or import a foreign allocated one, while still allowing updates from a
VA surface.
Add helpers to import EGLImage objects into VA surfaces. There are
two operational modes: (i) gst_vaapi_surface_new_from_egl_image(),
which allows for implicit conversion from EGLImage to a VA surface
in native video format, and (ii) gst_vaapi_surface_new_with_egl_image(),
which exactly wraps the source EGLImage, typically in RGBA format
with linear storage.
Note: in case of (i), the EGLImage can be disposed right after the
VA surface creation call, unlike in (ii) where the user shall ensure
that the EGLImage is live until the associated VA surface is no longer
needed.
https://bugzilla.gnome.org/show_bug.cgi?id=743847
Add initial support for EGL to libgstvaapi core library. The target
display server and the desired OpenGL API can be programmatically
selected at run-time.
A comprehensive set of EGL utilities are provided to support those
dynamic selection needs, but also most importantly to ensure that
the GL command stream is executed from within a single thread.
https://bugzilla.gnome.org/show_bug.cgi?id=743846
Added rounding control handling for VC1 simple and Main profile
based on VC1 standard spec: section 8.3.7
https://bugzilla.gnome.org/show_bug.cgi?id=743958
Signed-off-by: Lim Siew Hoon <siew.hoon.lim@intel.com>
Signed-off-by: Sreerenj Balachandran <sreerenj.balachandran@intel.com>
Practically we should be able to support more formats, for eg:
JPEG Encoder can support YUV422, RGBA and all.
But this is causing more issues which need proper fix here and there.
Otherwise the condition could become true before the lock
is taken and the g_cond_signal() could be called
before the g_cond_wait(), so the g_cond_wait() is never
awoken.
https://bugzilla.gnome.org/show_bug.cgi?id=740645
Add support for GEM buffer imports. This is useful for VA/EGL interop
with legacy Mesa implementations, or when it is desired or required to
support outbound textures for instance.
https://bugzilla.gnome.org/show_bug.cgi?id=736718
Add new gst_vaapi_surface_new_with_dma_buf_handle() helper function
to allow for creating VA surfaces from a foreign DRM PRIME fd. The
resulting VA surface owns the supplied buffer handle.
https://bugzilla.gnome.org/show_bug.cgi?id=735362
Add gst_vaapi_surface_new_from_buffer_proxy() helper function to
create a VA surface from an external buffer provided throug the
new GstVaapiBufferProxy object.
Add support for GEM buffer exports. This will only work with VA drivers
based off libdrm, e.g. the Intel HD Graphics VA driver. This is needed
to support interop with EGL and the "Desktop" GL specification. Indeed,
the EXT_image_dma_buf_import extension is not going to be supported in
Desktop GL, due to the lack of support for GL_TEXTURE_EXTERNAL_OES targets
there.
This is useful for implementing VA/EGL interop with legacy Mesa stacks,
in Desktop OpenGL context.
https://bugzilla.gnome.org/show_bug.cgi?id=736717
Use the new VA buffer export APIs to allow for a VA surface to be
exposed as a plain PRIME fd. This is in view to simplifying interop
with EGL or OpenCL for instance.
https://bugzilla.gnome.org/show_bug.cgi?id=735364
The VA buffer export APIs work for a particular lifetime starting from
vaAcquireBufferHandle() and ending with vaReleaseBufferHandle(). As such,
it could be much more convenient to support implicit releases by simply
having a refcount reaching zero.
https://bugzilla.gnome.org/show_bug.cgi?id=736721
Reword surface pool allocation helpers so that to allow for a simple
form, e.g. gst_vaapi_surface_pool_new(format, width, height); and a
somewhat more elaborated/flexible form with optional allocation flags
and precise GstVideoInfo specification.
This is an API/ABI change, and SONAME version needs to be bumped.
Add GstVaapiDisplay::get_{visual_id,colormap}() helpers to help determine
the best suitable window visual id and colormap. This is an indirection in
view to supporting EGL and custom/generic replacements.
Add GstVaapiWindowClass::get_colormap() hook to help determine the
currently active colormap bound to the supplied window, or actually
create it if it does not already exist yet.
Add GstVaapiWindowClass::get_visual_id() function hook to help find
the best suitable visual id for the supplied window. While doing so,
also simplify the process by which an X11 window is created with a
desired Visual, i.e. now use a visual id instead of a Visual object.
Add a new generic helper function gst_vaapi_window_new() to create
a window without having the caller to check for the display type
himself. i.e. internally, there is now a GstVaapiDisplayClass hook
to create windows, and the actual backend implementation fills it in.
Add new generic helper functions gst_vaapi_texture_new_wrapped()
This is a simplification in view to supporting EGL.
Add gst_vaapi_display_has_opengl() helper function to help determining
whether the display can support OpenGL context to be bound to it, i.e.
if the class is of type GST_VAAPI_DISPLAY_TYPE_GLX.
Make gst_vaapi_display_get_display_type() return the actual VA display
type. Conversely, add a gst_vaapi_display_get_class_type() function to
return the type of the GstVaapiDisplay instance. The former is used to
identify the display server onto which the application is running, and
the latter to identify the original object class.
Record the underlying native display instance into the toplevel
GstVaapiDisplay object. This is useful for fast lookups to the
underlying native display, e.g. for creating an EGL display.
Add new generic helper functions gst_vaapi_texture_new_wrapped()
and gst_vaapi_texture_new() to create a texture without having
the caller to uselessly check for the display type himself. i.e.
internally, there is now a GstVaapiDisplayClass hook to create
textures, and the actual backend implementation fills it in.
This is a simplification in view to supporting EGL.
GstVaapiTexture is a generic abstraction that could be moved to the
core libgstvaapi library. While doing this, no extra dependency needs
to be added. This means that a GstVaapitextureClass is now available
for any specific code that needs to be added, e.g. creation of the
underlying GL texture objects, or backend dependent ways to upload
a surface to the texture object.
Generic OpenGL data types (GLuint, GLenum) are also replaced with a
plain guint.
https://bugzilla.gnome.org/show_bug.cgi?id=736715
The VA/GLX interfaces are obsolete. They used to exist for XvBA, and
ease of use, but they had other caveats to deal with. It's now better
to move on to legacy mode, whereby VA/GLX interop is two be provided
through (i) X11 Pixmap, and (ii) other modern means of buffer sharing.
https://bugzilla.gnome.org/show_bug.cgi?id=736711
The gst_vaapi_texture_put_surface() function is missing a crop_rect
argument that would be used during transfer for cropping the source
surface to the desired dimensions.
Note: from a user point-of-view, he should create the GstVaapiTexture
object with the cropped size. That's the default behaviour in software
decoding pipelines that we need to cope with.
This is an API/ABI change, and SONAME version needs to be bumped.
https://bugzilla.gnome.org/show_bug.cgi?id=736712
Add new gst_vaapi_surface_new_full() helper function that allocates
VA surface from a GstVideoInfo template in argument. Additional flags
may include ways to
- allocate linear storage (GST_VAAPI_SURFACE_ALLOC_FLAG_LINEAR_STORAGE) ;
- allocate with fixed strides (GST_VAPI_SURFACE_ALLOC_FLAG_FIXED_STRIDES) ;
- allocate with fixed offsets (GST_VAAPI_SURFACE_ALLOC_FLAG_FIXED_OFFSETS).
Add new gst_vaapi_surface_proxy_new() helper to wrap a surface into
a proxy. The main use case for that is to convey additional information
at the proxy level that would not be suitable to the plain surface.
Re-introduce a GST_VAAPI_ID_INVALID value that represents
a non-zero and invalid id. This is useful to have a value
that is still invalid for cases where zero could actually
be a valid value.
Make it possible to have all libgstvaapi backends (libs) access to a
common GstVaapiMiniObject API and implementation. This is a minor step
towards full exposure when needed, but restrict it to libgstvaapi at
this time.
Really report sample aspect ratio (SAR) as present, and make it match
what we have obtained from the user as pixel-aspect-ratio (PAR). i.e.
really make sure VUI parameter aspect_ratio_info_present_flag is set
to TRUE and that the indication from aspect_ratio_idc is Extended_SAR.
This is a leftover from git commit a12662f.
https://bugzilla.gnome.org/show_bug.cgi?id=740360
Fix gst_vaapi_decoder_mpeg4_parse() to initialize the packet type to
GST_MPEG4_USER_DATA so that a parse error would result in skipping
that packet. Also fix gst_vaapi_decoder_mpeg4_decode_codec_data() to
initialize status to GST_VAAPI_DECODER_STATUS_SUCCESS.
Use the SEI pic_timing() message to track and propagate down the repeat
first field (RFF) flag. This is only initial support as there is one
other condition that could induce the RFF flag, which is not handled
yet.
Fix the decoding process for picture order count type 0 when the previous
picture had a memory_management_control_operation = 5. In particular, fix
the actual variable type for prev_pic_structure to hold the full bits of
the picture structure.
In practice, this used to work though, due to the underlying type used to
express a gboolean.
Use the SEI pic_timing() message to track the pic_struct variable when
present, or infer it from the regular slice header flags field_pic_flag
and bottom_field_flag. This fixes temporal sequence ordering when the
output pictures are to be displayed.
https://bugzilla.gnome.org/show_bug.cgi?id=739291
Add support for DRM Render-Nodes. This is a new feature that appeared
in kernel 3.12 for experimentation purposes, but was later declared
stable enough in kernel 3.15 for getting enabled by default.
This allows headless usages without authentication at all, i.e. usages
through plain ssh connections is possible.
Fix gst_vaapi_surface_proxy_copy() to copy the view-id element, thus
fixing random frames skipped when vaapipostproc element is used in
passthrough mode. In that mode, GstMemory is copied, thus including
the underlying GstVaapiVideoMeta and associated GstVaapiSurfaceProxy.
Ensure the X11 implementation for GstVaapiWindow::get_geometry() is
thread-safe by default, so that upper layer users don't need to handle
that explicitly.
Add gst_vaapi_window_reconfigure() interface to force an update of
the GstVaapiWindow "soft" size, based on the current geometry of the
underlying native window.
This can be useful for instance to synchronize the window size when
the user changed it.
Thanks to Fabrice Bellet for rebasing the patch.
[changed interface to gst_vaapi_window_reconfigure()]
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
Add gst_vaapi_display_get_display_name() helper function to determine
the name associated with the underlying native display. Note that for
raw DRM backends, the display name is actually the device path.
The timestamp generator in gstvaapidecoder_mpeg2.c always interpolated
frame timestamps within a GOP, even when it's been fed input PTS for
every frame.
That leads to incorrect output timestamps in some situations - for example
live playback where input timestamps have been scaled based on arrival time
from the network and don't exactly match the framerate.
https://bugzilla.gnome.org/show_bug.cgi?id=732719
Forbid GstVaapiObject to be created without an associated klass spec.
It is mandatory that the subclass implements an adequate .finalize()
hook, so it shall provide a valid GstVaapiObjectClass.
https://bugzilla.gnome.org/show_bug.cgi?id=722757
[made non-NULL klass argument to gst_vaapi_object_new() a requirement]
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
Call the subclass .init() function in gst_vaapi_object_new(), if
needed. The default behaviour is to zero initialize the subclass
object data, then the .init() function can be used to initialize
fields to non-default values, e.g. VA object ids to VA_INVALID_ID.
Also fix the gst_vaapi_object_new() description, which was merely
copied from GstVaapiMiniObject.
https://bugzilla.gnome.org/show_bug.cgi?id=722757
[changed to always zero initialize the subclass]
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
When a DPB flush is required, e.g. at a natural and of stream or issued
explicitly through an IDR, try to detect any frame left in the DPB that
is interlaced but does not contain two decoded fields. In that case, mark
the picture as having a single field only.
This avoids a hang while decoding tv_cut.mkv.
Simplify the dpb_output() function to exclusively rely on the frame store
buffer to output, since this is now always provided. Besides, also fix
cases where split fields would not be displayed.
This is a regression from f48b1e0.
Cope with latest changes from codecparsers/h264. It is now required
to explicitly clear the GstH264PPS structure as it could contain
additional allocations (slice_group_ids).
Add new GstVaapiSurfaceProxy flag FFB, which means "first frame in
bundle", and really expresses the first view component of a multi
view coded frame. e.g. in H.264 MVC, the surface proxy has flag FFB
set if VOIdx = 0.
Likewise, new API is exposed to retrieve the associated "view-id".
Allow decoders to set the "one-field" attribute when the decoded frame
genuinely has a single field, or if the second field was mis-decoded but
we still want to display the first field.
Make sure to output the decoded picture, and push the associated
GstVideoCodecFrame, only once. The frame fully represents what needs
to be output, included for interlaced streams. Otherwise, the base
GstVideoDecoder class would release the frame twice.
Anyway, the general process is to output decoded frames only when
they are complete. By complete, we mean a full frame was decoded or
both fields of a frame were decoded.
Slightly optimize decoding process by submitting the current VA surface
for decoding earlier to the hardware, and perform the reference picture
marking process and DPB update process afterwards.
This is a minor optimization to let the video decode engine kick in work
earlier, thus improving parallel resources utilization.
Fix decoding of interlaced streams where a first field (e.g. B-slice)
was immediately output and the current decoded field is to be paired
with that former frame, which is no longer in DPB.
https://bugzilla.gnome.org/show_bug.cgi?id=701340
Optimize the process to detect new pictures or start of new access
units by checking if the previous NAL unit was the end of a picture,
or the end of the previous access unit.
Add support for MVC streams with multiple SPS and subset SPS headers
emitted regularly, e.g. at around every I-frame. Track the maximum
number of views in ensure_context() and really reset the DPB size to
the expected value, always. i.e. even if it decreased. dpb_reset()
only cares of ensuring the DPB allocation.
Fix the compaction process when the DPB is cleared for a specific
view, i.e. fix the process of filling in the holes resulting from
removing frame buffers matching the current picture.
It is not necessary to periodically send SPS or subset SPS headers.
This is up to the upper layer (e.g. transport layer) to decide on
if/how to periodically submit those. For now, only generate new SPS
or subset SPS headers when the codec config changed.
Note: the upper layer could readily determine the config headers
(SPS/PPS) through the gst_vaapi_encoder_h264_get_codec_data() function.
https://bugzilla.gnome.org/show_bug.cgi?id=732083
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
Report sample aspect ratio (SAR) as present, and make it match what
we have obtained from the user as pixel-aspect-ratio (PAR). i.e. the
VUI parameter aspect_ratio_info_present_flag now defaults to TRUE.
Set the value of num_anchor_refs_l0, num_anchor_refs_l1, num_non_anchor_refs_l0,
and num_non_anchor_refs_l1 to zero since the inter-view prediction is not yet
supported.
When the seq_parameter_set_data() syntax structure is present in a subset
sequence parameter set and vui_parameters_present_flag is equal to 1, then
timing_info_present_flag shall be equal to 0 (H.7.4.2.1.1).
Submit Prefix NAL headers (nal_unit_type = 14) before every packed
slice header (nal_unit_type = 1 or 5) only for the base view. In non
base views, a Coded Slice Extension NAL header (nal_unit_type = 20)
is required, with an appropriate nal_unit_header_mvc_extension() in
the NAL header bytes.
https://bugzilla.gnome.org/show_bug.cgi?id=732083
Fix search for a picture in the DPB that has a lower POC value than
the current picture. The dpb_find_lowest_poc() function will return
a picture with the lowest POC in DPB and that is marked as "needed
for output", but an additional check against the actual POC value
of the current picture is needed.
This is a regression from 1c46990.
https://bugzilla.gnome.org/show_bug.cgi?id=732130
Fix dpb_clear() to clear previous frame buffers only if they actually
exist to begin with. If the decoder bailed out early, e.g. when it
does not support a specific profile, that array of previous frames
might not be allocated beforehand.
We can avoid scanning for start codes again if the bitstream is fed
in NALU chunks. Currently, we always scan for start codes, and keep
track of remaining bits in a GstAdapter, even if, in practice, we
are likely receiving one GstBuffer per NAL unit. i.e. h264parse with
"nal" alignment.
https://bugzilla.gnome.org/show_bug.cgi?id=723284
[use gst_adapter_available_fast() to determine the top buffer size]
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
The `vaapipostproc' element could never determine if the H.264 stream
was interlaced, and thus always assumed it to be progressive. Fix the
H.264 decoder to report interlace-mode accordingly, thus allowing the
vaapipostproc element to automatically enable deinterlacing.
The packed slice header and packed raw data need to be paired with
the submission of VAEncSliceHeaderParameterBuffer. So handle them
on a per-slice basis insted of a per-picture basis.
[removed useless initializer]
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
Factor out the removal process of unused inter-view only reference
pictures from the DPB, prior to the possible insertion of the current
picture.
Ideally, the compiler could still opt for generating two loops. But
at least, the code is now clearer for maintenance.
Improve process for the removal of pictures from DPB before possible
insertion of the current picture (C.4.4) for H.264 MVC inter-view only
reference components. In particular, handle cases where picture to be
inserted is not the last one of the access unit and if it was already
output and is no longer marked as used for reference, including for
decoding next view components within the same access unit.
While invoking the DPB bumping process in presence of many views,
it could be necessary to output previous pictures that are ready,
in a whole. i.e. emitting all view components from the very first
view order index zero to the very last one in its original access
unit; and not starting from the view order index of the picture
that caused the DPB bumping process to be invoked.
As a reminder, the maximum number of frames in DPB for MultiView
High profile with more than 2 views is not necessarily a multiple
of the number of views.
This fixes decoding of MVCNV-4.264.
Let the utility layer handle dynamic growth of the inter-view pictures
array. By definition, setting a new size to the array will effectively
grow the array, but would also fill in the newly created elements with
empty entries (NULL), thus also increasing the reported length, which
is not correct.
When decoding Multiview High profile streams with a large number of
views, it is not possible to make the VAPictureParameterBufferH264.
ReferenceFrames[] array hold the complete DPB, with all possibly
active pictures to be used for inter-view prediction in the current
access unit.
So reduce the scope of the ReferenceFrames[] array to only include
the set of reference pictures that are going to be used for decoding
the current picture. Basically, this is a union of all RefPicListX[]
array, for all slices constituting the decoded picture.
The inter-view reference components and inter-view only reference
components that are included in the reference picture lists shall
be considered as not being marked as "used for short-term reference"
or "used for long-term reference". This means that reference flags
should all be removed from VAPictureH264.flags.
This fixes decoding of MVCNV-2.264.
If the VA driver exposes ad-hoc H.264 MVC profiles, then we have to
be careful to detect profiles changes and not reset the underlying
VA context erroneously. In MVC situations, we could indeed get a
profile_idc change for every SPS that gets activated, alternatively
(base-view -> non-base view -> base-view, etc.).
An improved fix would be to characterize the exact profile to use
once and for all when SPS NAL units are parsed. This would also
allow for fallbacks to a base-view decoding only mode.
Exclusively use VA drivers that support raw packed headers for encoding.
i.e. simply submit packed headers Subset SPS and Prefix NAL units. This
provides for better compatibility accross the various VA drivers and HW
generations since no particular API is needed beyond what readily exists.
Since we are encoding each view independently from each other, we
need a higher number of pre-allocated surfaces to be used as the
reconstructed frames. For Stereo High profile encoding, this means
to effectively double the number of frames to be stored in the DPB.
Add initial support for Subset SPS, Prefix NAL and Slice Extension NAL
for non-base-view streams encoding, and the usual SPS, PPS and Slice
NALs for base-view encoding.
The H.264 Stereo High profile encoding mode will be turned on when the
"num-views" parameter is set to 2. The source (raw) YUV frames will be
considered as Left/Right view, alternatively.
Each of the two views has its own frames reordering pool and reference
frames list management system. Inter-view references are not supported
yet, so the views are encoded independently from each other.
Signed-off-by: Li Xiaowei <xiaowei.a.li@intel.com>
[limited to Stereo High profile per the definition of MAX_NUM_VIEWS]
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
Create structures to maintain the reference frames list (RefPool) and
frames reordering (ReorderPool) logic.
This is a prerequisite for H.264 MVC support.
Signed-off-by: Li Xiaowei <xiaowei.a.li@intel.com>
Add provisions to write subset SPS headers to the bitstream in view
to supporting the H.264 MVC specification.
This assumes the libva "staging" branch is in use.
Signed-off-by: Li Xiaowei <xiaowei.a.li@intel.com>
Optimize lookups of view ids / view order indices by caching the result
of the calculatiosn right into the GstVaapiParserInfoH264 struct. This
terribly simplifies is_new_access_unit() and find_first_field() functions.
Add safe fallbacks for MVC profiles:
- all MultiView High profile streams with 2 views at most can be decoded
with a Stereo High profile compliant decoder ;
- all Stereo High profile streams with only progressive views can be
decoded with a MultiView High profile compliant decoder ;
- all drivers that support slice-level decoding could normally support
MVC profiles when the DPB holds at most 16 frames.
In order to have a stricter conforming implementation, we need to carefully
detect access unit boundaries. Additional operations could be necessary to
perform at those boundaries.
Detect the first VCL NAL unit of a picture for MVC, based on the
view_id as per H.7.4.1.2.4. Note that we only need to detect new
view components.
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
Always cache the previous NAL unit so that we could check whether
there is a Prefix NAL unit immediately preceding the current slice
or IDR NAL unit. In that case, the NAL unit metadata is copied into
the current NAL unit. Otherwise, some default values are inferred,
tentatively. e.g. view_id shall be set to 0 and inter_view_flag to 1.
[infer default values for slice if previous NAL was not a Prefix]
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
Allow decoding for base views of MVC encoded streams. For now, just skip
the slice extension and prefix NAL units, and skip non-base view frames.
Signed-off-by: Xiaowei Li <xiaowei.a.li@intel.com>
[fixed memory leak, improved check for MVC NAL units]
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
Factor out process by which the decoded picture with the lowest POC
is found, and possibly output. Likewise, the storage and marking of
a reference decoded, or non-reference decoded picture, into the DPB
could also be simplified as they mostly share the same operations.
Make init_picture_ref_lists() more consistent with other functions
related to the reference marking process by supplying the current
picture as argument.
Add gst_vaapi_display_get_vendor_string() helper function to query
the underlying VA driver name. The display object owns the resulting
string, so it shall not be deallocated.
That function is thread-safe. It could be used for debugging purposes,
for instance.
Make sure to initialize one GstVaapiDisplay at a time, even in threaded
environments. This makes sure the display cache is also consistent
during the whole display creation process. In the former implementation,
there were risks that display cache got updated in another thread.
Add support for dynamic growth of the VA surfaces pool. For decoding,
this implies the recreation of the underlying VA context, as per the
requirement from VA-API. Besides, only increases are supported, not
shrinks.
It is a requirement from VA-API specification that the VA context got
from vaCreateContext(), for decoding purposes, binds the supplied set
of VA surfaces. This means that if the set of VA surfaces is to be
changed for the current decode session, then the VA context needs to
be recreated with the new set of VA surfaces.
Complement fix committed as e95a42e.
The H.264 AVC standard has to say: if the field is part of a reference
frame or a complementary reference field pair, and the other field of
the same reference frame or complementary reference field pair is also
marked as "used for long-term reference", the reference frame or
complementary reference field pair is also marked as "used for long-term
reference" and assigned LongTermFrameIdx equal to long_term_frame_idx.
This fixes decoding of MR9_BT_B in strict mode.
https://bugs.freedesktop.org/show_bug.cgi?id=64624https://bugzilla.gnome.org/show_bug.cgi?id=724518
Request the correct chroma format for decoding grayscale streams.
i.e. make lookups of the VA chroma format more generic, thus possibly
supporting more formats in the future.
This means that, if a VA driver doesn't support grayscale formats,
it is now going to fail. We cannot safely assume that maybe grayscale
was implemented on top of some YUV 4:2:0 with the chroma components
all set to 0x80.
Fix reference picture marking process with memory_management_control_op
set to 3 and 6, i.e. assign LongTermFrameIdx to a short-term reference
picture, or the current picture.
This fixes decoding of FRExt_MMCO4_Sony_B.
https://bugs.freedesktop.org/show_bug.cgi?id=64624https://bugzilla.gnome.org/show_bug.cgi?id=724518
[squashed, edited to use GST_VAAPI_PICTURE_IS_COMPLETE() macro]
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
The initialization of reference picture lists (8.2.4.2) applies to all
slices. So, the RefPicList0/1 lists need to be constructed prior to
each slice submission to the HW decoder.
This fixes decoding of video sequences where frames are encoded with
multiple slices of different types, e.g. 4 slices in this order I, P,
I, and P. More precisely, CABAST3_Sony_E and CABASTBR3_Sony_B.
https://bugzilla.gnome.org/show_bug.cgi?id=724518
When NAL units of type 13 (SPS extension) or type 19 (auxiliary slice)
are present in a video, decoders shall perform the (optional) decoding
process specified for these NAL units or shall ignore them (7.4.1).
Implement option 2 (skip) for now, as alpha composition is not
supported yet during the decoding process.
This fixes decoding of the primary coded video in alphaconformanceG.
https://bugzilla.gnome.org/show_bug.cgi?id=703928https://bugzilla.gnome.org/show_bug.cgi?id=728869https://bugzilla.gnome.org/show_bug.cgi?id=724518
[skip NAL units earlier, i.e. at parsing time]
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
When MVC slice NAL units (coded slice extension and prefix NAL) are
present, the number of NAL header bytes is 3, not 1 as usual.
Signed-off-by: Li Xiaowei <xiaowei.a.li@intel.com>
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
At the time the first VCL NAL unit of a primary coded picture is found,
and if that NAL unit was parsed to be an SPS or PPS, then the entries
in the parser may have been overriden. This means that, when the picture
is to be decoded, slice_hdr->pps could point to an invalid (the next)
PPS entry.
So, one way to solve this problem is to not use the parser PPS and
SPS info but rather maintain our own activation chain in the decoder.
https://bugzilla.gnome.org/show_bug.cgi?id=724519https://bugzilla.gnome.org/show_bug.cgi?id=724518
Retain the SEI messages that were parsed from the access unit until we
have completely decoded the current frame. This is done so that we can
peek at that data whenever necessary during decoding. e.g. for exposing
3D stereoscopic information at a later stage.
Fix support for grayscale encoded video clips, and possibly others if
the underlying driver supports the non-YUV 4:2:0 formats. i.e. defer
the decision that a surface with the desired chroma format is not
supported to the actual VA driver implementation.
https://bugzilla.gnome.org/show_bug.cgi?id=728144
Don't force allocation of VA surfaces in YUV 4:2:0 format. Rather, allow
for the upper layer to specify the desired chroma type. If the chroma
type field is not set (or yields zero), then YUV 4:2:0 format is used
by default.
Fix possible bug when a per-segment deblocking filter level value
needs to be set in non-absolute mode, i.e. when the loop filter update
value is negative in delta mode.
Also clamp the resulting filter level value to 0..63 range.
Improve condition to disable the loop filter. The previous heuristic
used to check all filter levels, for all segments. It turns out that
only the base filter_level value defined in the frame header needs
to be checked.
This fixes 00-comprehensive-013.
Fix generation of source tarballs when certain conditionals are not
met. e.g. always include all buildable codecparsers sources in the
distribution tarball, fix plug-in element sources set to include X11
and encoder bits.
The built-in libvpx serves multiple purposes, among which the most
important ones could be: track the most up-to-date, and optimized,
range decoder; allow for future hybrid implementations (non-VLD);
and have a completely independent range decoder implementation.
Apply correct patch from fd.o #722760 to fix several issues: update the
license terms to LGPLv2.1+, fix dependencies to built-in libvpx and fix
make dist.
Add libvpx submodule that tracks the upstream version 1.3.0. This is
needed to build a libgstcodecparsers_vpx.so library with all symbols
placed into the GSTREAMER namespace.
The gst_h264_parse_parse_sei() function now returns an array of SEI
messages, instead of a single SEI message. Reason: it is allowed to
have several SEI messages packed into a single SEI NAL unit, instead
of multiple NAL units.
8fadf40 h264: Fix multiple SEI messages in one SEI RBSP parsing.
644825f h265: remove trailling 0x00 bytes as the spec doesn't allow them
95f9f0f h264: remove trailling 0x00 bytes as the spec doesn't allow them
766007b h265: Initialize pointer correctly that is never assigned but freed in error cases
8ec5816 h265: Fix segfault when parsing HRD parameter
5b1730f h265: Fix segfault when parsing VPS
983b7f7 h265: prevent to overrun chroma_weight_l0_flag
7ba641d h265: Fix debug output
d9f9f9b h264: not all startcodes should have 3-byte 0 prefix
Fix parser and decoder state to sync at the right locations. This is
because we could reset the parser state, while the decoder state was
not copied yet, e.g. when parsing several NAL units from multiple frames
whereas the current frame was not decoded yet.
This is a regression brought in by commit 6fe5496.
Fix gstreamer-vaapi includedir for GStreamer 1.2 setups. i.e. use
the pkgconfig version (1.0) instead of the intended API version (1.2).
libgstvaapi1.0-dev and libgstvaapi1.2-dev packages will now conflict,
as would core GStreamer 1.0 and GStreamer 1.2 dev packages anyway.
Add gst_vaapi_get_config_attribute() helper function that takes a
GstVaapiDisplay and the rest of the arguments with VA types. The aim
is to have thread-safe VA helpers by default.
Make sure to configure the encoder with the set of packed headers we
intend to generate and submit. i.e. make selection of packed headers
to submit more robust.
Cache the first compatible GstVaapiProfile found if the encoder is not
configured yet. Next, factor out the code to check for the supported
rate-control modes by moving out vaGetConfigAttributes() to a separate
function, while also making sure that the attribute type is actually
supported by the encoder.
Also fix the default set of supported rate control modes to not the
"none" variant. It's totally useless to expose it at this point.
Introduce GstVaapiContextUsage so that to explicitly determine the
usage of a VA context. This is useful in view to simplifying the
creation of VA context for VPP too.
Unknown attributes, or attributes that are not supported for the given
profile/entrypoint pair have a return value of VA_ATTRIB_NOT_SUPPORTED.
So, return failure in this case.
Move GstVideoOverlayComposition handling to separate source files.
This helps keeing GstVaapiContext core implementation to the bare
minimal, i.e. simpy helpers to create a VA context and handle pool
of associated VA surfaces.
Improve documentation and debug messages. Clean-up APIs, i.e. strip
them down to the minimal set of interfaces. They are private, so no
need expose getters for instance.
Make sure that libgstvaapi private headers remain internally used to
build libgstvaapi libraries only. All header dependencies were reviewed
and checks for IN_LIBGSTVAAPI definition were added accordingly.
Also rename GST_VAAPI_CORE definition to IN_LIBGSTVAAPI_CORE to keep
consistency.
Set a valid GstVideoCodecFrame.system_frame_number when decoding a
stream in standalone mode. While we are at it, improve the debugging
messages to also include that frame number.
When decoding failed, or that the frame was dropped, the associated
surface proxy is not guaranteed to be present. Thus, the GST_DEBUG()
message needs to check whether the proxy is actually present or not.
https://bugzilla.gnome.org/show_bug.cgi?id=722403
[fixed gst_vaapi_surface_proxy_get_surface_id() to return VA_INVALID_ID]
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
Don't emit NAL HRD parameters for now in the SPS headers because the
SEI buffering_period() and picture_timing() messages are not handled
yet. Some additional changes are necessary to get it right.
https://bugzilla.gnome.org/show_bug.cgi?id=722734
Fix default CPB buffer size to something more reasonable (1500 ms)
and that still fits the level limits. This is a non configurable
property for now. The initial CPB removal delay is also fixed to
750 ms.
https://bugzilla.gnome.org/show_bug.cgi?id=722087
Round down the calculated, or supplied, bitrate (kbps) into a multiple
of the HRD bitrate scale factor. Use a bitrate scale factor of 64 so
that to have less losses in precision. Likewise, don't round up because
that could be a strict constraint imposed by the user.
Fix the level calculation involving bitrate limits. Since we are
targetting NAL HRD conformance, the check against MaxBR from the
Table A-1 limits shall involve cpbBrNalFactor depending on the
active profile.
Submit sequence parameter buffers only once, or when the bitstream
was reconfigured in a way that requires such. Always submit packed
sequence parameter buffers at I-frame period, if the VA driver needs
those.
https://bugzilla.gnome.org/show_bug.cgi?id=722737
Make sure to submit the packed headers only if the underlying VA driver
requires those. Currently, only handle packed sequence and picture
headers.
https://bugzilla.gnome.org/show_bug.cgi?id=722737
The VAEncSequenceParameterBuffer.ip_period value reprents the distance
between the I-frame and the next P-frame. So, this also accounts for
any additional B-frame in the middle of it.
This fixes rate control heuristics for certain VA drivers.
https://bugzilla.gnome.org/show_bug.cgi?id=722735
Fix level characterisation when the bitrate is automatically computed
from the active coding tools. i.e. ensure the bitrate once the profile
is completely characterized but before the level calculation process.
Document and rename a few functions here and there. Drop code that
caps num_bframes variable in reset_properties() since they shall
have been checked beforehand, during properties initialization.
Clean-up GstBitWriter related utility functions and simplify notations.
While we are at it, also make bitstream writing more robust should an
overflow occur. We could later optimize for writing headers capped to
their maximum possible size by using the _unchecked() helper variants.
Drop private header since it was originally used to expose internals
to the plugin element. The proper interface is now the properties API,
thus rendering private headers totally obsolete.
Add support for hue, saturation, brightness and constrat adjustments.
Also fix cap info local copy to match the really expected cap subtype
of interest.
https://bugzilla.gnome.org/show_bug.cgi?id=720376
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
Change the submission order of VA objects so that to make that process
more logical. i.e. submit sequence parameter first, if any; next the
packed headers associated to sequece, picture or slices; and finally
the actual picture and associated slices.
Various clean-ups to improve consistency and readability: rename some
variables, drop unused macro definitions, drop initialization of vars
that are zero-initialized from the base class, drop un-necessary casts,
allocate GPtrArrays with a destroy function.
Fix frame cropping rectangle calculation to handle horizontal resolutions
that don't match a multiple of 16 pixels, but also the vertical resolution
that was incorrectly computed for progressive sequences too.
https://bugzilla.gnome.org/show_bug.cgi?id=722089
For non "Constant-QP" modes, we could provide more reasonable heuristics
for the target bitrate. In general, 48 bits per macroblock with all the
useful coding tools enable looks safe enough. Then, this rate is raised
by +10% to +15% for each coding tool that is disabled.
https://bugzilla.gnome.org/show_bug.cgi?id=719699
Add support for "high-compression" tuning option. First, determine the
largest supported profile by the hardware. Next, check any target limit
set by the user. Then, enable each individual coding tool based on the
resulting profile_idc value to use.
https://bugzilla.gnome.org/show_bug.cgi?id=719696
Allow user to precise the largest profile to use for encoding due
to target decoder constraints. For instance, if CABAC entropy coding
mode is requested by "constrained-baseline" profile only is desired,
then an error is returned during codec configuration.
Also make sure that the suitable profile we derived actually matches
what the HW can cope with.
https://bugzilla.gnome.org/show_bug.cgi?id=719694
Refine the heuristic to determine the maximum size of a coded buffer
to account for the exact number of slices. set_context_info() is the
last step during codec reconfiguration, no additional change is done
afterwards, so re-using the num_slices field here is fine.
https://bugzilla.gnome.org/show_bug.cgi?id=719953
Automatically derive the minimum profile and level to be used for
encoding, based on the activated coding tools. The encoder will
be trying to generate a bitstream that has the best chances to be
decoded on most platforms by default.
Also change the default profile to "constrained-baseline" so that
to ensure maximum compatibility when the stream is decoded.
https://bugzilla.gnome.org/show_bug.cgi?id=719691
Fix lookup for a suitable HW profile, as to be used by the underlying
hardware, based on heuristics that lead to characterize the SW profile,
i.e. the one used by the SW level encoding logic.
Also fix constraint_set0_flag (A.2.1) and constraint_set1_flag (A.2.2)
as they should respectively match the baseline and main profile.
https://bugzilla.gnome.org/show_bug.cgi?id=719827
The libgstvaapi core encoders are meant to support raw bitstreams only.
Henceforth, we are always producing a stream in "byte-stream" format.
However, the "codec-data" buffer which holds SPS and PPS headers is
always available. The "lengthSizeMinusOne" field is always set to 3
so that in-place "byte-stream" format to "avc" format conversion could
be performed.
Various clean-ups to improve consistency and readability: rename some
variables, drop unused macro definitions, drop initialization of vars
that are zero-initialized from the base class, drop un-necessary casts.
Fix lookup for a suitable HW profile, as to be used by the underlying
hardware, based on heuristics that lead to characterize the SW profile,
i.e. the one used by the SW level encoding logic.
Automatically derive the minimum profile and level to be used for
encoding, based on the activated coding tools. Improve lookup for
the best suitable level with the new MPEG-2 helper functions.
Also change the default profile to "simple" so that to ensure maximum
compatibility when the stream is decoded.
https://bugzilla.gnome.org/show_bug.cgi?id=719703
Various clean-ups to improve consistency and readability: drop unused
macro definitions, drop initialization of vars that are zero-initialized
from the base class, drop un-necessary casts.
Add encoder "tune" option to override the default behaviour that is to
favor maximum decoder compatibility at the expense of lower compression
ratios.
Expected tuning options to be developed are:
- "high-compression": improve compression, target best-in-class decoders;
- "low-latency": tune for low-latency decoding;
- "low-power": tune for encoding in low power / resources conditions.
Only expose the exact static set of supported rate-control properties
to the upper layer. For instance, if the GstVaapiEncoderXXX class does
only support CQP rate control, then only add it the the exposed enum
type.
Add helper macros and functions to build a GType for an enum subset.
Add gst_vaapi_encoder_set_keyframe_period() interface to allow the
user control the maximum distance between two keyframes. This new
property can only be set prior to gst_vaapi_encoder_set_codec_state().
A value of zero for "keyframe-period" gets it re-evaluated to the
actual framerate during encoder reconfiguration.
Improve codec reconfiguration to be performed only through a single
function. That is, remove the _set_context_info() hook as subclass
should not alter the parent GstVaapiContextInfo itself. Besides, the
VA context is constructed only at the final stages of reconfigure().
Add interface to communicate the encoder resolution and related info
like framerate, interlaced vs. progressive, etc. This new interface
supersedes gst_vaapi_encoder_set_format() and doesn't use any GstCaps
but rather use GstVideoCodecState.
Note that gst_vaapi_encoder_set_codec_state() is also a synchronization
point for codec config. This means that the encoder is reconfigured
there to match the latest properties.
Add interface to communicate configurable properties to the encoder.
This covers both the common ones (rate-control, bitrate), and the
codec specific properties.
https://bugzilla.gnome.org/show_bug.cgi?id=719529
Add gst_vaapi_encoder_set_bitrate() interface to allow the user control
the bitrate for encoding. Currently, changing this parameter is only
valid before the first frame is encoded. Should the value be modified
afterwards, then GST_VAAPI_ENCODER_STATUS_ERROR_OPERATION_FAILED is
returned.
https://bugzilla.gnome.org/show_bug.cgi?id=719529
Add gst_vaapi_encoder_set_rate_control() interface to request a new
rate control mode for encoding. Changing the rate control mode is
only valid prior to encoding the very first frame. Afterwards, an
error ("operation-failed") is issued.
https://bugzilla.gnome.org/show_bug.cgi?id=719529
Instal <gst/vaapi/gstvaapiutils_h264.h> header but only expose the
H.264 levels in there. The additional helper functions are meant
to be private for now.
Fix formts for various GST_DEBUG et al. invocations. More precisely,
make size_t arguments use the %zu format specifier accordingly; force
XID formats to be a 32-bit unsigned integer; and fix the format used
for gst_vaapi_create_surface_with_format() error cases since we have
been using strings nowadays.
The following helper functions are no longer used, thus are removed:
- gst_vaapi_video_format_from_structure()
- gst_vaapi_video_format_from_caps()
- gst_vaapi_video_format_to_caps()
Replace gst_vaapi_display_get_{decode,encode}_caps() APIs with more
more convenient APIs that return an array of GstVaapiProfile instead
of GstCaps: gst_vaapi_display_get_{decode,encode}_profiles().
Replace gst_vaapi_display_get_{image,subpicture}_caps() APIs, that
returned GstCaps, with more convenient APIs that return an array of
GstVideoFormat: gst_vaapi_display_get_{image,subpicture}_formats().
Fix display creation code to check that any display obtained from a
neighbour actually has the type we expect. Note: if display type is
set to "any", we can then accept any VA display type.
The GLTextureUploadMeta implementation assumed that for each upload()
sequence, the supplied texture id is always the same as the one that
was previously cached into the underlying GstVaapiTexture. Cope with
any texture id change the expense to recreate the underlying VA/GLX
resources.
https://bugzilla.gnome.org/show_bug.cgi?id=719643
Improve robustness when some expected packets where not received yet
or that were not correctly decoded. For example, don't try to decode
a picture if there was no valid frame headers parsed so far.
https://bugs.freedesktop.org/show_bug.cgi?id=57902
Conformance test Base_Ext_Main_profiles/BA3_SVA_C.264 complys with
extended profile specifications. However, the SPS header has the
constraint_set1_flag syntax element set to 1. This means that, if
a Main profile compliant decoder is available, then it should be
able to decode this stream.
This changes makes it possible to fall-back from Extended profile
to Main profile if constraint_set1_flag is set to 1.
https://bugzilla.gnome.org/show_bug.cgi?id=720190
Recognize streams marked as conforming to the "Constrained Baseline
Profile". If VA driver supports that as is, fine. Otherwise, fallback
to baseline, main or high profile.
Constrained Baseline Profile conveys coding tools that are common
to baseline profile and main profile.
https://bugzilla.gnome.org/show_bug.cgi?id=719947
[Added fallbacks to main and high profiles]
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
The GStreamer codecparser layer now parses the scaling lists in zigzag
scan order, as expected, so that to match the original bitstream layout
and specification. However, further convert the scaling lists into
raster scan order to fit the existing practice in most VA drivers.
https://bugzilla.gnome.org/show_bug.cgi?id=706406
- gst_vaapi_utils_h264_get_level():
Returns GstVaapiLevelH264 from H.264 level_idc value
- gst_vaapi_utils_h264_get_level_idc():
Returns H.264 level_idc value from GstVaapiLevelH264
- gst_vaapi_utils_h264_get_level_limits():
Returns level limits as specified in Table A-1 of the H.264 standard
- gst_vaapi_utils_h264_get_level_limits_table():
Returns the Table A-1 specification
* Profiles:
- gst_vaapi_utils_h264_get_profile():
Returns GstVaapiProfile from H.264 profile_idc value
- gst_vaapi_utils_h264_get_profile_idc():
Returns H.264 profile_idc value from GstVaapiProfile
* Chroma formats:
- gst_vaapi_utils_h264_get_chroma_type():
Returns GstVaapiChromaType from H.264 chroma_format_idc value
- gst_vaapi_utils_h264_get_chroma_format_idc():
Returns H.264 chroma_format_idc value from GstVaapiChromaType
The previous fix was only valid to express the maximum size of the
macroblock layer, i.e. without any headers. Now, also account for
the slice headers and top picture header, but also any other header
we might stuff into the VA coded buffer, e.g. sequence headers.
Fix coded buffer size for each codec. A generic issue was that the
number of macroblocks was incorrectly computed. The second issue was
specific to MPEG-2 were the max number of bits per macroblock, and
as defined by the standard, was incorrectly mapped to the (lower)
H.264 requirement. i.e. 4608 bits vs. 3200 bits limit.
Change get_context_info() into a set_context_info() function that
initializes common defaults into the base class, thus allowing the
subclasses to specialize the context info further on.
The set_context_info() hook is also the location where additional
context specific data could be initialized. At this point, we are
guaranteed to have valid video resolution size and framerate. i.e.
gst_vaapi_encoder_set_format() was called beforehand.
Clean public APIs up so that to better align with the decoder APIs.
Most importantly, gst_vaapi_encoder_get_buffer() is changed to only
return the VA coded buffer proxy. Also provide useful documentation
for the public APIs.
Kill GstVaapiEncoderSyncPic objects that are internally and temporarily
allocated. Rather, associate a GstVaapiEncPicture to a coded buffer
through GstVaapiCodedBufferProxy user-data facility.
Besides, use a GAsyncQueue to maintain a thread-safe queue object of
coded buffers.
Partial fix for the following report:
https://bugzilla.gnome.org/show_bug.cgi?id=719530
Fix the GstVaapiEncoderClass parent class type. Make sure to validate
subclass hooks as early as possible, i.e. in gst_vaapi_encoder_init(),
thus avoiding useless run-time checks. Also simplify the subclass
initialization process to be less error prone.
Refactor the GstVaapiCodedBuffer APIs so that to more clearly separate
public and private interfaces. Besides, the map/unmap APIs should not
be exposed as is but appropriate accessors should be provided instead.
* GstVaapiCodedBuffer: VA coded buffer abstraction
- gst_vaapi_coded_buffer_get_size(): get coded buffer size.
- gst_vaapi_coded_buffer_copy_into(): copy coded buffer into GstBuffer
* GstVaapiCodedBufferPool: pool of VA coded buffer objects
- gst_vaapi_coded_buffer_pool_new(): create a pool of coded buffers of
the specified max size, and bound to the supplied encoder
* GstVaapiCodedBufferProxy: pool-allocated VA coded buffer object proxy
- gst_vaapi_coded_buffer_proxy_new_from_pool(): create coded buf from pool
- gst_vaapi_coded_buffer_proxy_get_buffer(): get underlying coded buffer
- gst_vaapi_coded_buffer_proxy_get_buffer_size(): get coded buffer size
Rationale: more optimized transfer functions might be provided in the
future, thus rendering the map/unmap mechanism obsolete or sub-optimal.
https://bugzilla.gnome.org/show_bug.cgi?id=719775
Add gst_vaapi_surface_proxy_copy() function that creates a new surface
proxy with the same information from the parent proxy, except that the
user-defined destroy notify function is not copied over.
The underlying VA surface is pushed back to the video pool only when
the last reference to the parent surface proxy is released.
Optimize gst_vaapiencode_handle_frame() to avoid extra memory allocation,
and in particular the GstVaapiEncObjUserData object. i.e. directly use
the VA surface proxy from the source buffer. This also makes the user
data attached to the GstVideoCodecFrame more consistent between both
the decoder and encoder plug-in elements.
Add gst_buffer_new_allocate() and gst_buffer_fill() implementations.
Fix gst_buffer_new_wrapped_full() implementation to handle the destroy
notify function.
GStreamer 0.10.36 is the latest and ultimate version to be released
from the GStreamer 0.10 branch. i.e. no further releases are to be
made. So, we can safely enable the built-in videoutils replacement
now that they are in sync with the 0.10 branch.
Add initial API for video encoding: only basic interfaces and small
encoder objects are implemented so far.
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
GstBitWriter provides a bit writer that can write any number of bits
to a pre-allocated memory buffer. Helper functions are also provided
to write any number of bits from 8, 16, 32 and 64 bit variables.
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
Extend GstVaapiContextInfo structure to hold the desired rate control
mode for encoding purposes. For decoding purposes, this field is not
used and it is initialized to GST_VAAPI_RATECONTROL_NONE.
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
Add GstVaapiRateControl types and GType values in view to supporting
rate controls for encoding. This is meant to be used for instance in
GstVaapiContext.
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
This library is using symbols that don't exist in GStreamer 0.10 so
it needs to link to built-in implementation (libgstvaapi-videoutils).
https://bugzilla.gnome.org/show_bug.cgi?id=712282
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
Destroy VPP output surface pool on exit. Also avoid a possible crash
in double-free situation caused by insufficiently reference counted
array of formats returned during initialization.
The <gst/vaapi/gstvaapicontext.h> header was removed from the public
set of APIs. So, don't make public headers (gstvaapidecoder.h) depend
on private files.
Add gst_vaapi_fitler_set_deinterlacing_references() API to submit the
list of surfaces used for forward or backward reference in advanced
deinterlacing mode, e.g. Motion-Adaptive, Motion-Compensated.
The list of surfaces used as deinterlacing references shall be live
until the next call to gst_vaapi_filter_process().
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
Fix deinterlacing flags to make more sense. The TFF (top-field-first)
flag is meant to specify the organization of reference frames used in
advanced deinterlacing modes. Introduce the more explicit flag TOPFIELD
to specify that the top-field of the supplied input surface is to be
used for deinterlacing. Conversely, if not set, this means that the
bottom field of the supplied input surface will be used instead.
Add a couple of helper functions:
- gst_vaapi_filter_has_operation(): checks whether the VA driver
advertises support for the supplied operation ;
- gst_vaapi_filter_use_operation(): checks whether the supplied
operation was already enabled to its non-default value.
The user-defined destroy notify function is meant to be called only when
the surface proxy was fully released, i.e. once it actually released the
VA surface back to the underlying pool.
Fix GstVaapiPicture, GstVaapiSlice and GstVaapiSurfaceProxy initialization
sequences to have the expected default values set beforehand in case of an
error raising up further during creation. i.e. make it possible to cleanly
destroy those partially initialized objects.
https://bugzilla.gnome.org/show_bug.cgi?id=707108
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
Fix ensure_operations() to release the VPP operations array if non
NULL, prior to returning to the caller. The former function was also
renamed to a more meaningful get_operations() since the caller owns
the returned array that needs to be released.
Fix first-time operation lookup through find_operation() if the set
of supported operations was not initially determined through the
gst_vaapi_filter_get_operations() helper function.
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
Fix intiialization of GstVaapiFilterOpData for colorbalance related
operations. In particular, fill in the va_subtype field accordingly.
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
VASurfaceAttrib and VAProcFilterParameterBufferType are symbols
that need to be guarded for libva 0.34 and 0.33, respectively.
https://bugzilla.gnome.org/show_bug.cgi?id=709102
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
Fix calculation of MCU count for image sizes that are not a multiple
of 8 pixels in either dimension, but also for non-common sampling
factors like 4:2:2 in non-interleaved mode.
Add support for images with multiple scans per frame. The Huffman table
can be updated before SOS, and thus possibly requiring multiple uploads
of Huffman tables to the VA driver. So, the latter must be able to cope
with multiple VA buffers of type 'huffman-table' and with the correct
sequential order.
Improve robustness when some expected packets where not received yet
or that were not correctly decoded. For example, don't try to decode
a picture if there was no valid frame headers.
Improve debugging and error messages. Rename a few variables to fit the
existing naming conventions. Change some fatal asserts to non-fatal
error codes.
Split the input buffer data into decoder units that represent a JPEG
segment. Handle scan decoder unit specifically so that it can include
both the scan header (SOS) but also any other ECS or RSTi segment.
That way, we parse the input buffer stream only once at the gst-vaapi
level instead of (i) in gst_vaapi_decoder_jpeg_parse() to split the
stream into frames SOI .. EOI and (ii) in decode_buffer() to further
determine segment boundaries and decode them.
In practice, this is a +15 to +25% performance improvement.
Fix decode_buffer() function to gracefully skip comment (COM) segments.
This fixes decoding of streams generated by certain cameras, e.g. like
the Logitech Pro C920.
https://bugzilla.gnome.org/show_bug.cgi?id=708208
Signed-off-by: Junfeng Xu <jun.feng.xu@intel.com>
Look for the exact image bounds characterised by the <SOI> and <EOI>
markers. Use the gst_jpeg_parse() codec parser utility function to
optimize the lookup for the next marker segment.
https://bugzilla.gnome.org/show_bug.cgi?id=707447
Fix calculation of the offset to the next marker segment since the
correction of the codecparser part to match the API specification.
i.e. the GstJpegMarkerSegment.size field represents the size in bytes
of the segment minus any marker prefix.
https://bugzilla.gnome.org/show_bug.cgi?id=707447
Signed-off-by: Junfeng Xu <jun.feng.xu@intel.com>
Fix detection of VA/JPEG decoding API with non-standard libva packages.
More precisely, some packages were shipping with a <va/va.h> header that
did not include <va/va_dec_jpeg.h>.
https://bugzilla.gnome.org/show_bug.cgi?id=706055
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
As a last resort, if video processing capabilities (VPP) are not available,
or they did not produce anything conclusive enough, then try to fallback to
the original rendering code path whereby the whole VA surface is rendered
as is, no matter of video cropping or deinterlacing requests.
Note: under those conditions, the visual outcome won't be correct but at
least, something gets displayed instead of bailing out.
Try to use VA/VPP processing capabilities to handle video cropping and
additional rendering flags that may not be directly supported by the
underlying hardware when exposing a suitable Wayland buffer for the
supplied VA surface. e.g. deinterlacing, different color primaries than
BT.601, etc.
Update the frame redraw infrastructure with a new FrameState stucture
holds all the necessary information used to display the next pending
surface.
While we are at it, delay the sync operation down to when it is actually
needed. That way, we keep performing additional tasks meanwhile.
Disable video cropping in MPEG-2 codec because it is partially implemented
and actually because nobody implements it that way, and the standard spec
does not specify the display process either anyway.
Most notably, there are two possible use cases for sequence_display_extension()
horizontal_display_size & vertical_display_size: (i) guesstimating the
pixel-aspect-ratio, or (ii) implement some kind of span & scan process
in conjunction with picture_display_extension() information.
https://bugzilla.gnome.org/show_bug.cgi?id=704848
This fixes the following issue:
CC libgstvaapi_0.10_la-gstvaapidecoder_mpeg4.lo
gstvaapidecoder_mpeg4.c:113: error: redefinition of typedef
'GstVaapiDecoderMpeg4Class'
gstvaapidecoder_mpeg4.c:44: note: previous declaration of
'GstVaapiDecoderMpeg4Class' was here
make[5]: *** [libgstvaapi_0.10_la-gstvaapidecoder_mpeg4.lo] Error 1
make[5]: Leaving directory
`/builddir/build/BUILD/gstreamer-vaapi-0.5.5.1/gst-libs/gst/vaapi'
https://bugzilla.gnome.org/show_bug.cgi?id=705148
Add basic deinterlacing support, i.e. bob-deinterlacing whereby only
the selected field from the input surface is kept for the target surface.
Setting gst_vaapi_filter_set_deinterlacing() method argument to
GST_VAAPI_DEINTERLACE_METHOD_NONE means to disable deinterlacing.
Also move GstVaapiDeinterlaceMethod definition from vaapipostproc plug-in
to libgstvaapi core library.
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
Add ProcAmp (color balance) adjustments for hue, saturation, brightness
and contrast. The respective range for each filter shall be the same as
for the VA display attributes.
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
Sharpening is configured with a float value. The supported range is
-1.0 .. 1.0 with 0.0 being the default, and that means no sharpening
operation at all.
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
Noise reduction is configured with a float value. The supported range
is 0.0 .. 1.0 with 0.0 being the default, and that means no denoise
operation at all.
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
Add helper functions to ensure an operation VA buffer is allocated to
the right size; that filter caps get parsed and assigned to the right
operation too; and that float parameters are correctly scaled to fit
the reported range from the VA driver.
Add gst_vaapi_video_format_from_string() helper function to convert from
a video format string representation to a suitable GstVideoFormat. This
is just an alias to gst_video_format_from_string() for GStreamer 1.0.x
builds, and a proper iteration over all GstVideoFormat string representations
otherwise for earlier GStreamer 0.10.x builds.
Add helper functions to describe GstVaapiPoint and GstVaapiRectangle
structures as a standard GType. This could be useful to have them
described as a GValue later on.
Don't expose GstVaapiContext APIs and make them totally private to
libgstvaapi core library. That API would also tend to disappear in
a future revision. Likewise, don't expose GstVaapiDisplayCache API
but keep symbols visible so that the various render backends could
share a common display cache implementation in libgstvaapi.
Try to clean-up the documentation from any stale entry too.
Don't expose functions that reference a GstVaapiImageRaw, those are
meant to be internal only for implementing subpictures sync. Also add
a few private definitions to avoid functions calls for retrieving
image size and format information.
Use hardware accelerated XRenderComposite() function, from the RENDER
extension, to blit a pixmap to screen. Besides, this can also support
cropping and scaling.
Implement the new render-to-pixmap API. The only supported pixmap format
that will work is xRGB, with native byte ordering. Others might work but
they were not tested.
Add API to transfer VA urfaces to native pixmaps. Also add an API to
render a native pixmap, for completeness. In general, rendering to
pixmap would only be useful to certain VA drivers and use cases on
X11 display servers. e.g. GLX_EXT_texture_from_pixmap (TFP) handled
in an upper layer.
Mark dummy pictures as output already so that we don't try to submit
them to the upper layer since this is purely internal / temporary
picture for helping the decoder.
Once the picture was output, it is no longer necessary to keep an extra
reference to the underlying GstVideoCodecFrame. So, we can release it
earlier, and maybe subsequently release the associate surface proxy
earlier.
Fix new internal video format API, based on GstVideoFormat, to not
clobber with system symbols. So replace the gst_video_format_* prefix
with gst_vaapi_video_format_ prefix, even if the format type remains
GstVideoFormat.
Fix memory leak when processing interlaced pictures and that occurs
because the first field, represented as a GstVideoCodecFrame, never
gets released. i.e. when the picture is completed, this is generally
the case when the second field is successfully decoded, we need to
propagate the GstVideoCodecFrame of the first field to the original
GstVideoDecoder so that it could reclaim memory.
Otherwise, we keep accumulating the first fields into GstVideoDecoder
private frames list until the end-of-stream is reached. The frames
are eventually released there, but too late, i.e. too much memory
may have been consumed.
https://bugzilla.gnome.org/show_bug.cgi?id=701257
The queue of free objects to used was deallocated with g_queue_free_full().
However, this convenience function shall only be used if the original queue
was allocated with g_queue_new(). This caused memory corruption, eventually
leading to a crash.
The correct solution is to pair the g_queue_init() with the corresponding
g_queue_clear(), while iterating over all free objects to deallocate them.
Fix creation of surface pool objects to honour explicit pixel format
specification. If this operation is not supported, then fallback to
the older interface with chroma format.
If a VA surface was allocated with the chroma-format interface, try to
determine the underlying pixel format on gst_vaapi_surface_get_format(),
or return GST_VIDEO_FORMAT_ENCODED if this is not a supported operation.
Make it possible to create VA surfaces with a specific pixel format.
This is a new capability brought in by VA-API >= 0.34.0. If that
capability is not built-in (e.g. using VA-API < 0.34.0), then
gst_vaapi_surface_new_with_format() will return NULL.
Add gst_video_format_get_chroma_type() helper function to determine
the GstVaapiChromaType from a standard GStreamer video format. It is
possible to reconstruct that from GstVideoFormatInfo but it is much
simpler (and faster?) to use the local GstVideoFormatMap table.
Add new chroma formats available with VA-API >= 0.34.0. In particular,
this includes "RGB" chroma formats, and more YUV subsampled formats.
Also add a new from_GstVaapiChromaType() helper function to convert
libgstvaapi chroma type to VA chroma format.
Make gst_vaapi_image_pool_new() succeed, and thus returning a valid
image pool object, only if the underlying VA display does support the
requested VA image format.
Get rid of GstCaps to create surface/image pool, and use GstVideoInfo
structures instead. Those are smaller, and allows for streamlining
libgstvaapi more.
Add new video format mappings to VA image formats:
- YUV: packed YUV (YUY2, UYVY), grayscale (Y800) ;
- RGB: 32-bit RGB without alpha channel (XRGB, XBGR, RGBX, BGRX).
Fix debug message string with image format expressed with GstVideoFormat
instead of the obsolete format that turned out to be a fourcc.
This is a regression from git commit e61c5fc.
In particular, use gst_video_info_from_caps() helper function in VA image
for implementating gst_vaapi_image_get_buffer() [vaapidownload] and
gst_vaapi_image_update_from_buffer() [subpictures] in GStreamer 0.10 builds.
Drop GstVaapiImageFormat helpers since everything was moved to the new
GstVideoFormat based API. Don't bother with backwards compatibility and
just bump the library major version afterwards.
If the stream has a sequence_display_extenion, then attach the
display_horizontal/display_vertical dimension as the cropping
rectangle width/height to the GstVaapiPicture.
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
If the Advanced profile has display_extension fields, then set the display
width/height dimension as cropping rectangle to the GstVaapiPicture.
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
If the encoded stream has the frame_cropping_flag set, then associate
the cropping rectangle to GstVaapiPicture.
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
Make it possible associate an empty cropping rectangle to the surface
proxy, thus resetting any cropping rectangle that was previously set.
This allows for returning plain NULL when no cropping rectangle was
initially set up to the surface proxy, or if it was reset to defaults.
Add helper macros to retrieve the VA surface information like size
(width, height) or chroma type. This is a micro-optimization to avoid
useless function calls and NULL pointer re-checks in internal routines.
Add gst_vaapi_picture_set_crop_rect() helper function to copy the video
cropping information from raw bitstreams to each picture being decoded.
Also add helper function to surface proxy to propagate that information
outside of libgstvaapi. e.g. plug-in elements or standalone applications.
Signed-off-by: Sreerenj Balachandran <sreerenj.balachandran@intel.com>
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
The MPEG-2 standard specifies (6.3.7) that all quantisation matrices
shall be reset to their default values when a Sequence_Header() is
decoded.
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
Break the circular references between GstVaapiContext and its children
GstVaapiSurfaces. Since the VA surfaces held an extra reference to the
context, which holds a reference to its VA surfaces, then none of those
were released.
How does this impact support for subpictures?
The only situation when the parent context needs to disappear is when
it is replaced with another one because of a resolution change in the
video stream for instance, or a normal destroy. In this case, it does
not really matter to apply subpictures to the peer surfaces since they
are either gone, or those that are left in the pipe can probably bear
a reinstantiation of the subpictures for it.
So, parent_context is set to NULL when the parent context is destroyed,
other VA surfaces can still get subpictures attached to them, individually
not as a whole. i.e. subpictures for surface S1 will be created from
active composition buffers and associated to S1, subpictures for S2 will
be created from the next active composition buffers, etc. We don't try
to cache the subpictures in those cases (pending surfaces until EOS
is reached, or pending surfaces until new surfaces matching new VA context
get to be used instead).
Make sure to create the GLX context once the window object has completed
its creation. Since gl_resize() relies on the newly created window size,
then we cannot simply overload the GstVaapiWindowClass::create() hook.
So, we just call into gst_vaapi_window_glx_ensure_context() once the
window object is created in the gst_vaapi_window_glx_new*() functions.
Make it possible to add extra an extra filter to most of display cache
lookup functions so that the GstVaapiDisplay instance can really match
a compatible and existing display by type, instead of relying on extra
string tags (e.g. "X11:" prefix, etc.).
Drop obsolete GST_VAAPI_IS_xxx() helper macros since we are no longer
deriving from GObject and so those were only checking for whether the
argument was NULL or not. This is now irrelevant, and even confusing
to some extent, because we no longer have type checking.
Note: this incurs more type checking (review) but the libgstvaapi is
rather small, so this is manageable.
Make GstVaapiID a gsize instead of guessing an underlying integer large
enough to hold all bits of a pointer. Also drop GST_VAAPI_ID_NONE since
this is plain zero and that it is no longer passed as varargs.
Port GstVaapiDecoder and GstVaapiDecoder{MPEG2,MPEG4,JPEG,H264,VC1} to
GstVaapiMiniObject. Add gst_vaapi_decoder_set_codec_state_changed_func()
helper function to let the user add a callback to a function triggered
whenever the codec state (e.g. caps) changes.
Port GstVaapiVideoPool, GstVaapiSurfacePool and GstVaapiImagePool to
GstVaapiMiniObject. Drop gst_vaapi_video_pool_get_caps() since it was
no longer used for a long time. Make object allocators static, i.e.
local to the shared library.
Drop support for user-defined data since this capability was not used
so far and GstVaapiMiniObject represents the smallest reference counted
object type. Add missing GST_VAAPI_MINI_OBJECT_CLASS() helper macro.
Besides, since GstVaapiMiniObject is a libgstvaapi internal object, it
is also possible to further simplify the layout of the object. i.e. merge
GstVaapiMiniObjectBase into GstVaapiMiniObject.
Propagate the picture size from the bitstream to the GstVaapiDecoder,
and subsequent user who installed a signal on notify::caps. This fixes
decoding of TS streams when the demuxer failed to extract the required
information.
Add gst_vaapi_decoder_get_frame_with_timeout() helper function that will
wait for a frame to be decoded, until the specified timeout in microseconds,
prior to returning to the caller.
This is a fix to performance regression from 851cc0, whereby the vaapidecode
loop executed on the srcpad task was called to often, thus starving all CPU
resources.
Rework GstVideoDecoder::handle_frame() to decode the current frame,
while possibly waiting for a free surface, and separately submit all
decoded frames from a task. This makes it possible to pop and render
decoded frames as soon as possible.
Add support for interlaced streams with GStreamer 1.0 too. Basically,
this enables vaapipostproc, though it is not auto-plugged yet. We also
make sure to reply to CAPS queries, and happily handle CAPS events.
Fix support for interlaced contents with GStreamer 0.10. In particular,
propagate GstVaapiSurfaceProxy frame flags to GstVideoCodecFrame flags
correctly.
This is a regression from commit 87e5717.
Rename GstVaapiDecoderFrame to GstVaapiParserFrame because this data
structure was only useful to parsing and a proper GstvaapiDecoderFrame
instance will be created instead.
Fix regression from 0.4-branch whereby GstVaapiSurfaceProxy no longer
held any information about the expected presentation timestamp, frame
duration or additional flags like interlaced or top-field-first.
Use new GstVaapiSurfaceProxy internal helper functions to propagate the
necessary GstVideoCodecFrame flags to vaapidecode (GStreamer 0.10).
Also make GstVaapiDecoder push_frame() operate similarly to drop_frame().
i.e. increase the GstVideoCodecFrame reference count in push_frame rather
than gst_vaapi_picture_output().
Add more attributes for raw decoding modes, i.e. directly through the
libgstvaapi helper library. In particular, add presentation timestamp,
duration and a couple of flags (interlaced, TFF, RFF, one-field).
Ensure libgstvaapi-glx*.so builds against libdl since dlsym() is used
to resolve glXGetProcAddress() from GLX libraries. This fix builds on
Fedora 17.
https://bugzilla.gnome.org/show_bug.cgi?id=698046
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
Fix previous commit whereby gst_vaapi_decoder_get_codec_state() was
supposed to make GstVaapiDecoder own the return GstVideoCodecState
object. Only comment was updated, not the actual code.
Make gst_vaapi_decoder_get_codec_state() return the original codec state,
i.e. make the GstVaapiDecoder object own the return state so that callers
that want an extra reference to it would just gst_video_codec_state_ref()
it before usage. This aligns the behaviour with what we had before with
gst_vaapi_decoder_get_caps().
This is an ABI incompatible change, library major version was bumped from
previous release (0.5.2).
Fix the name of the plug-in element reported to gst-inspect-1.0. i.e. we
need an explicit definition for GStreamer >= 1.0 because the GST_PLUGIN_DEFINE
incorrectly uses #name for creating the plug-in name, instead of using macro
expansion (and let further expansion of macros) through e.g. G_STRINGIFY().
Fix make dist to allow build for either GStreamer 0.10 or 1.0. i.e. make
sure to include all source files in either case while generating source
tarballs.
Introduce a new configure option --with-gstreamer-api that determines
the desired GStreamer API to use. By default, GStreamer 1.0 is selected.
Also integrate more compatibility glue into gstcompat.h and plugins.
This integrates support for GStreamer API >= 1.0 only in the libgstvaapi
core decoding library. The changes are kept rather minimal here so that
the library retains as little dependency as possible on core GStreamer
functionality.
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
Drop the following functions that are now obsolete:
- gst_vaapi_context_get_surface()
- gst_vaapi_context_put_surface()
- gst_vaapi_context_find_surface_by_id()
- gst_vaapi_surface_proxy_new()
- gst_vaapi_surface_proxy_get_context()
- gst_vaapi_surface_proxy_set_context()
- gst_vaapi_surface_proxy_set_surface()
This is an API change.
Now that the surface pool is reference counted in the surface proxy wrapper,
we can safely ignore surface size checks in gst_vaapi_decoder_ensure_context().
Besides, this check is already performed in gst_vaapi_context_reset_full().
Introduce gst_vaapi_surface_proxy_new_from_pool() to allocate a new surface
proxy from the context surface pool. This change also makes sure to retain
the parent surface pool in the proxy.
Besides, it was also totally useless to attach/detach parent context to
VA surface each time we acquire/release it. Since the whole context owns
all associated VA surfaces, we can mark this as such only once and for all.
Move GstVaapiVideoMeta from core libgstvaapi decoding library to the
actual plugin elements. That's only useful there. Also inline reference
counting code from GstVaapiMiniObject.
Make sure libgstvaapi core decoding library doesn't include un-needed
dependencies. So, move out GstVaapiVideoConverterGLX to plugins instead.
Besides, even if the vaapisink element is not used, we are bound to have
a correctly populated GstSurfaceBuffer from vaapidecode.
Also clean-up the file along the way.
In decode_codec_data(), force initialization of format to zero so that
we can catch up cases where codec-data has neither "format" nor "wmvversion"
fields, thus making it possible to gracefully fail in this case.
Add a new GstVaapiDecoder::decode_codec_data() hook to actually decode
codec-data in the decoder sub-class. Provide a common shared helper
function to do the actual work and delegating further to the sub-class.
Drop GstVaapiDecoderUnit buffer field (GstBuffer) since it's totally
useless nowadays as creating sub-buffers doesn't bring any value. It
actually means more memory allocations. We can't do without that in
JPEG and MPEG-4:2 decoders.
Use gst_element_class_set_static_metadata() from GStreamer 1.0, which
basically is the same as gst_element_class_set_details_simple() in
GStreamer 0.10 context.
Use newer gst_video_overlay_rectangle_get_pixels_unscaled_raw() helper
function with GStreamer 0.10 compatible semantics, or that tries to
approach the current meaning. Basically, this is also just about moving
the helper to gstcompat.h.
Generated source files were missing a dependency on the complete set of
generated header files. e.g. gstvideodecoder.c requires gstvideoutils.h
to build and almost every codec parser source depends on parserutils.h.
https://bugs.freedesktop.org/show_bug.cgi?id=59575
Signed-off-by: Xiang, Haihao <haihao.xiang@intel.com>
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
Force luma_log2_weight_denom and chroma_log2_weight_denom to zero if
there is no pred_weight_table() that was parsed.
This is a workaround for the VA intel-driver on Ivy Bridge.
g_async_queue_timeout_pop() appeared in glib 2.31.18. Implement it as
g_async_queue_timed_pop() with a GTimeVal as the final time to wait for
new data to arrive in the queue.
There shall be only one place to call decode_current_picture(), and this
is in the end_frame() hook. The EOS unit is processed after end_frame()
so this means we cannot have a valid picture to decode/output at this
point.
Improve robustness when some expected packets where not received yet
or that were not correctly decoded. For example, don't try to decode
a picture if there was no valid sequence or picture headers.
Fix gst_vaapi_decoder_get_surface() to only return frames with a valid
surface proxy, i.e. with a valid VA surface. This means that any frame
marked as decode-only is simply skipped.
If the decoder was not able to decode a frame because insufficient
information was available, e.g. missing sequence or picture header,
then allow the frame to be gracefully dropped without generating
any error.
It is also possible that a frame is not meant to be displayed but
only used as a reference, so dropping that frame is also a valid
operation since GstVideoDecoder base class has extra references to
that GstVideoCodecFrame that needs to be released.
The Wayland API is not fully thread-safe and client applications shall
perform locking themselves on key functions. Besides, make sure to
release the lock if the _render() function fails.
Introduce gst_vaapi_window_wayland_sync() helper function to wait for
the completion of the redraw request. Use it in _render() function to
actually block until the previous draw request is completed.
The redraw callback needs to be attached to the surface prior to the
commit. Otherwise, the callback notifies the next surface repaint,
which is not the desired behaviour. i.e. we want to be notified for
the surface we have just filled.
Another isse was the redraw_pending was reset before the actual completion
of the frame redraw callback function, thus causing concurrency issues.
e.g. the callback could have been called again, but with a NULL buffer.
When the Wayland display is shared, we still have to create our own local
shell and compositor objects, since they are not propagated from the cache.
Likewise, we also need to determine the display size or vaapisink would
fail to account for the display aspect ratio, and will try to create a 0x0
window.
Fix build with newer VC-1 codecparser where dqsbedge was renamed to
dqbedge, and now represents either DQSBEDGE or DQDBEDGE depending on
the actual value of DQPROFILE.
Fix size of encapsulated BDUs since GstVC1BDU.size actually represents
the size of the BDU data, starting from offset, i.e. after any start
code is parsed.
This fixes a buffer overflow during the unescaping process.
The AVI demuxer (avidemux) does not set a proper "format" attribute
to the generated caps. So, try to recover the video codec format from
the "wmvversion" property instead.
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
Don't create temporary GstBuffers for all decoder units, even if they
are lightweight "sub-buffers", since it is not really necessary to keep
the buffer data around.
Implement GstVaapiDecoder.start_frame() and end_frame() semantics so
that to create new VA context earlier and submit VA pictures to the
HW for decoding as soon as possible. i.e. don't wait for the next
frame to start decoding the previous one.
Use GstVaapiDpb interface instead of maintaining our own prev and next
picture pointers. While doing so, try to derive a sensible POC value.
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
Avoid usage of goto. Simplify decode_step() process to first accumulate all
pending buffers into the GstAdapter, and then parse and decode units from
that input adapter. Stop the process once a frame is fully decoded or an
error occurred.
Make sure we always have a free surface left to use for decoding the
current frame. This means that decode_step() has to return once a frame
gets decoded. If the current adapter contains more buffers with valid
frames, they will get parsed and decoded on subsequent iterations.
Keep only one DPB interface and rename gst_vaapi_dpb2_get_references()
to gst_vaapi_dpb_get_neighbours() so that to retrieve pictures in DPB
around the specified picture POC.
Move GstVaapiDpbMpeg2 API to a more generic version that could also be
useful to other decoders that require 2 reference pictures, e.g. VC-1.
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
Fix support for global-alpha subpictures. The previous changes brought
the ability to check for GstVideoOverlayRectangle changes by comparing
the underlying pixel buffer pointers. If sequence number and pixel data
did not change, then this is an indication that only the global-alpha
value changed. Now, try to update the underlying VA subpicture global-alpha
value.
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
Don't re-upload VA subpicture if only the render rectangle changed.
Rather deassociate the subpicture and re-associate it with the new
render rectangle.
A GstVideoOverlayRectangle is created whenever the underlying pixels data
change. However, when global-alpha is supported, it is possible to re-use
the same GstVideoOverlayRectangle but with a change to the global-alpha
value. This process causes a change of sequence number, so we can no longer
check for that.
Still, if sequence numbers did not change, then there was no change in
global-alpha either. So, we need a way to compare the underlying GstBuffer
pointers. There is no API to retrieve the original pixels buffer from
a GstVideoOverlayRectangle. So, we use the following heuristics:
1. Use gst_video_overlay_rectangle_get_pixels_unscaled_argb() with the same
format flags from which the GstVideoOverlayRectangle was created. This
will work if there was no prior consumer of the GstVideoOverlayRectangle
with alternate (non-"native") format flags.
2. In overlay_rectangle_has_changed_pixels(), we have to use the same
gst_video_overlay_rectangle_get_pixels_unscaled_argb() function but
with flags that match the subpicture. This is needed to cope with
platforms that don't support global-alpha in HW, so the gst-video
layer takes care of that and fixes this up with a possibly new
GstBuffer, and hence pixels data (or) in-place by caching the current
global-alpha value applied. So we have to determine the rectangle
was previously used, based on what previous flags were used to
retrieve the ARGB pixels buffer.
We previously assumed that an overlay composition changed if the number
of overlay rectangles in there actually changed, or that the rectangle
was updated, and thus its seqnum was also updated.
Now, we can cope with cases where the GstVideoOverlayComposition grew
by one or a few more overlay rectangles, and the initial overlay rectangles
are kept as is.
Create the GPtrArray once in the _init() function and destroy it only
in the _finalize() function. Then use overlay_clear() to remove all
subpicture associations for intermediate updates, don't recreate the
GPtrArray.
Make GstVaapiOverlayRectangle a reference counted object. Also make
sure that overlay_rectangle_new() actually creates and associates the
VA subpicture.
Handle global-alpha from GstVideoOverlayComposition API. Likewise,
the same code path could also work for premultiplied-alpha but this
was not tested.
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
Add the necessary helpers in GstVaapiDisplay to determine whether subpictures
with global alpha are supported or not. Also add accessors in GstVaapiSubpicture
to address this feature.
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
Add premultiplied-alpha and global-alpha feature flags, along with converters
between VA-API and gstreamer-vaapi definitions. Another round of helpers is
also necessary for GstVideoOverlayComposition API.
Use GPOINTER_TO_SIZE() instead of GPOINTER_TO_UINT() while manipulating
pointers. The latter is meant to be 32-bit only, not uintptr_t like size.
Only a gsize can hold all bits of a pointer.
Thanks to Ouping Zhang for spotting this error.
Heuristic: if the second start-code is available, check whether that
one marks the start of a new frame because e.g. this is a sequence
or picture header. This doesn't save much, since we already cache the
results.
Accelerate scan for start codes by skipping up to 3 bytes per iteration.
A start code prefix is defined by the following bytes: 00 00 01. Thus,
for any group of 3 bytes (xx yy zz), we have the following possible cases:
1. If zz != 1, this cannot be a start code, then skip 3 bytes;
2. If yy != 0, this cannot be a start code, then skip 2 bytes;
3. If xx != 0 or zz != 1, this cannot be a start code, then skip 1 byte;
4. xx == 00, yy == 00, zz == 1, we have match!
This algorithm requires to peek bytes from the adapter. This increases the
amount of bytes copied to a temporary buffer, but this process is much faster
than scanning for all the bytes and using shift/masks. So, overall, this is
a win.
Move parsing back to decoding step, but keep functions separate for now.
This is needed for future optimizations that may introduce some meta data
for parsed info attached to codec frames.
Optimize pre-allocation of decoder units, thus avoiding un-necessary
memory reallocations. The heuristic used is that we could have around
one slice unit per macroblock line.
Use a GArray to hold decoder units in a frame, instead of a single-linked
list. This makes 'append' calls faster, but not that much. At least, this
makes things clearer.
Allocate decoder unit earlier in the main parse() function and don't
delegate this task to derived classes. The ultimate purpose is to get
rid of dynamic allocation of decoder units.
The SPS, PPS and slice headers are not fully zero-initialized in the
codecparsers/ library. Rather, the standard upstream behaviour is to
initialize only certain syntax elements with some inferred values if
they are not present in the bitstream.
At the gstreamer-vaapi decoder level, we need to further initialize
certain syntax elements with some sensible default values so that to
not complicate VA drivers that just pass those verbatim to the HW,
and also avoid an memset() of the whole decoder unit.
Signed-off-by: Sreerenj Balachandran <sreerenj.balachandran@intel.com>
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
Make GstVaapiVideoBuffer a simple wrapper for video meta. This buffer is
no longer necessary but for compatibility with GStreamer 0.10 APIs or users
expecting a GstSurfaceBuffer like Clutter.
Use PTS value computed by the decoder, which could also be derived from
the GstVideoCodecFrame PTS. This makes it possible to fix up the PTS if
the original one was miscomputed or only represented a DTS instead.
Create a new VA context if the encoded surface size changes because we
need to keep the underlying surface pool until the last one was released.
Otherwise, either of the following cases could have happened: (i) release
a VA surface to an inexistent pool, or (ii) release VA surface to an
existing surface pool, but with different size.
Avoid creating a GstBuffer for slice data. Rather, directly use the codec
frame input buffer data. This is possible because the codec frame is valid
until end_frame() where we submit the VA buffers for decoding. Anyway, the
slice data buffer is copied into the VA buffer when it is created.
Implement GstVaapiDecoder.start_frame() and end_frame() semantics so
that to create new VA context earlier and submit VA pictures to the
HW for decoding as soon as possible. i.e. don't wait for the next
frame to start decoding the previous one.
Parse slice() header and first macroblock position earlier in _parse()
function instead of waiting for the _decode() stage. This doesn't change
anything but readability.
Introduce new GstVaapiDecoderUnitMpeg2 object, which holds the standard
GstMpegVideoPacket and additional parsed header info. Besides, we now
parse as early as in the _parse() function so that to avoid un-necessary
creation of sub-buffers in _decode() for video packets that are not slices.
Theory of operations: all units marked as "slice" are moved to the "units"
list. Since this list only contains slice data units, the prev_slice pointer
was removed. Besides, we now maintain two extra lists of units to be decoded
before or after slice data units.
In particular, all units in the "pre_units" list will be decoded before
GstVaapiDecoder::start_frame() is called and units in the "post_units"
list will be decoded after GstVaapiDecoder::end_frame() is called.
Make GstVaapiSurfaceProxy only a thin wrapper around a VA context and a
VA surface. i.e. drop any other attribute like timestamp, duration,
interlaced or top-field-first.
Maintain decoded surfaces as GstVideoCodecFrame objects instead of
GstVaapiSurfaceProxy objects. The latter will tend to be reduced to
the strict minimum: a context and a surface.
Add new gst_vaapi_decoder_get_frame() function meant to be used with
gst_vaapi_decoder_decode(). The purpose is to return the next decoded
frame as a GstVideoCodecFrame and the associated GstVaapiSurfaceProxy
as the user-data object.
Decoder units were zero-initialized, including the SPS/PPS/slice headers.
The latter don't require zero-initialization since the codecparsers/ lib
will do so for key variables already. This is not a great value per se but
at least it makes it possible to check whether the default initialization
decisions made in the codecparsers/ lib were right or not.
This can be reverted if this exposes too many issues.
Drop explicit initialization of most fields that are implicitly set to
zero. Drop helper macros for casting to GstVaapiPictureH264 or
GstVaapiFrameStore. Also remove some useless checks for NULL pointers.
Implement GstVaapiDecoder.start_frame() and end_frame() semantics so
that to create new VA context earlier and submit VA pictures to the
HW for decoding as soon as possible. i.e. don't wait for the next
frame to start decoding the previous one.
Introduce new GstVaapiDecoderUnitH264 object, which holds the standard
NAL unit header (GstH264NalUnit) and additional parsed header info.
Besides, we now parse headers as early as in the _parse() function so
that to avoid un-necessary creation of sub-buffers in _decode() for
NAL units that are not slices.
This is a performance win by ~+1.1% only.
Use standard GstVideoCodecState throughout GstVaapiDecoder and expose
it with a new gst_vaapi_decoder_get_codec_state() function. This makes
it possible to drop picture size (width, height) information, framerate
(fps_n, fps_d) information, pixel aspect ratio (par_n, par_d) information,
and interlace mode (is_interlaced field).
This is a new API with backwards compatibility maintained. In particular,
gst_vaapi_decoder_get_caps() is still available.
Signed-off-by: Sreerenj Balachandran <sreerenj.balachandran@intel.com>
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
Align gst_vaapi_decoder_get_surface() semantics with the rest of the
API. That is, return a GstVaapiDecoderStatus and the decoded surface
as a handle to GstVaapiSurfaceProxy in parameter.
This is an API/ABI change.
Introduce new decoding process whereby a GstVideoCodecFrame is created
first. Next, input stream buffers are accumulated into a GstAdapter,
that is then passed to the _parse() function. The GstVaapiDecoder object
accumulates all parsed units and when a complete frame or field is
detected, that GstVideoCodecFrame is passed to the _decode() function.
Ultimately, the caller receives a GstVaapiSurfaceProxy if decoding
process was successful.
The start_frame() hook is called prior to traversing all decode-units
for decoding. The unit argument represents the first slice in the frame.
Some codecs (e.g. H.264) need to wait for the first slice in order to
determine the actual VA context parameters.
Split decoding process into two steps: (i) parse incoming bitstreams
into simple decoder-units until the frame or field is complete; and
(ii) decode the whole frame or field at once.
This is an ABI change.
Introduce a new GstVaapiDecoderFrame that is just a list of decoder units
(GstVaapiDecoderUnit objects) that constitute a frame. This object is just
an extension to GstVideoCodecFrame for VA decoder purposes. It is available
as the user-data member element.
This is a libgstvaapi internal object.
Introduce GstVaapiDecoderUnit which represents a fragment of the source
stream to be decoded. For instance, a decode-unit will be a NAL unit for
H.264 streams, an EBDU for VC-1 streams, and a video packet for MPEG-2
streams.
This is a libgstvaapi internal object.
GstVaapiSurfaceProxy does not use any particular functionality from
GObject. Actually, it only needs a basic object type with reference
counting.
This is an API and ABI change.
Introduce a new reference counted object that is very lightweight and
also provides flags and user-data functionalities. Initialization and
finalization times are reduced by up to a factor 5x vs GstMiniObject
from GStreamer 0.10 stack.
This is a libgstvaapi internal object.
Fix decode_slice() to ensure a VA context exists prior to creating a
new GstVaapiSliceH264, which invokes vaCreateBuffer() with some VA
context ID. i.e. the latter was not initialized, thus causing failures
on Cedar Trail for example.
Use glib >= 2.32 semantics for GMutex and GRecMutex wrt. initialization
and termination. Basically, the new mutex objects can be used as static
mutex objects from the deprecated APIs, e.g. GStaticMutex and GStaticRecMutex.
This fixes symbol clashes between the gst-vaapi built-in codecparsers/
library and the system-provided one, mainly used by videoparses/. Now,
only symbols with the gst_vaapi_* prefix will be exported, if they are
not marked as "hidden" to libgstvaapi.
Fix gst_vaapi_image_map() to return TRUE and the GstVaapiImageRaw
structure correctly filled in if the image was already mapped.
Likewise, make gst_vaapi_image_unmap() return TRUE if the image
was already unmapped.
Fix reference leak of surface and image in GstVaapiVideoBuffer wrapper,
thus resulting on actual memory leak of GstVaapiImage when using them
for downloads/uploads from VA surfaces and more specifically surfaces
when the pipeline is shutdown. i.e. vaTerminate() was never called
because the resources were not unreferenced, and thus not deallocated
in the end.
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
The picture size signalled by sps->{width,height} is the actual size with
cropping applied, not the original size derived from pic_width_in_mbs_minus1
and pic_height_in_map_units_minus1. VA driver expects that original size,
uncropped.
There is another issue pending: frame cropping information needs to be
taken care of.
Fix commit 18245b4 so that to link and build parserutils.[ch] first.
This is needed since that's the common dependency for actual codec
parsers (gstvc1parser.c for instance).
... this is useful to make sure pixel-aspect-ratio and framerate
information are correctly parsed since we have no means to detect
that at configure time.
Invoke gst_mpeg_video_finalise_mpeg2_sequence_header() to get the
correct PAR values. While doing so, require a newer version of the
bitstream parser library.
Note: it may be necessary to also parse the Sequence_Display_Extension()
header.
Signed-off-by: Sreerenj Balachandran <sreerenj.balachandran@intel.com>
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
This patch updates to relect the 1.0 version of the protocol. The main
changes are the switch to wl_registry for global object notifications
and the way that the event queue and file descriptor is processed.
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
git am got confused somehow, though the end result doesn't change at
all since we require both SPS and PPS to be parsed prior to decoding
the first slice.
Only start decoding slices when at least one SPS and PPS got activated.
This fixes cases when a source represents a substream of another stream
and no SPS and PPS was inserted before the first slice of the generated
substream.
... for interlaced streams. The short_ref[] and long_ref[] arrays may
contain up to 32 fields but VA ReferenceFrames[] array expects up to
16 reference frames, thus including both fields.
Fix decoding of interlaced streams when adaptive_ref_pic_marking_mode_flag
is equal to 1, i.e. when memory management control operations are used. In
particular, when field_pic_flag is set to 0, the new reference flags shall
be applied to both fields.
Decoded frames are only output when they are complete, i.e. when both
fields are decoded. This also means that the "interlaced" caps is not
propagated to vaapipostproc or vaapisink elements. Another limitation
is that interlaced bitstreams with MMCO are unlikely to work.
Split remove_reference_at() into a function that actually removes the
specified entry from the short-term or long-term reference picture array,
and a function that sets reference flags to the desired value, possibly
zero. The latters marks the picture as "unused for reference".
Fix gst_vaapi_picture_new_field() to preserve the original picture type.
e.g. gst_vaapi_picture_new_field() with a GstVaapiPictureH264 argument
shall generate a GstVaapiPictureH264 object.
Introduce new `structure' field to the H.264 specific picture structure
so that to simplify the reference picture marking process. That local
picture structure is derived from the original picture structure, as
defined by the syntax elements field_pic_flag and bottom_field_flag.
Move DPB flush up if the current picture to decode is an IDR. Besides,
don't bother to check for IDR pictures in dpb_add() function since an
explicit DPB flush was already performed in this case.
... to build the short_ref[] and long_ref[] lists from the DPB, instead
of maintaining them separately. This avoids refs/unrefs while making it
possible to generate the list based on the actual picture structure.
This also ensures that the list of generated ReferenceFrames[] actually
matches what reference frames are available in the DPB. i.e. short_ref[]
and long_ref[] entries are implied from the DPB, so there is no risk of
having "dangling" references.
Use the POC member available in the GstVaapiPicture base class and
get rid of the dependency on the local VAPictureH264 TopFieldOrderCnt
and BottomFieldOrderCnt. Rather, use a simple field_poc[] array
initialized to INT_MAX, so that to simplify picture POC calculation
for non frame pictures.
Further get rid of GstVaapiPictureH264-local VAPictureH264.flags for
reference bits, thus simplifying the reference picture marking process
to only track a single set of reference flags. Also introduce a new
long_term_frame_idx member.
Add vaapi_fill_picture() helper function to convert GstVaapiPictureH264
to VAPictureH264 structure. This is preparatory work to get rid of the
local VAPictureH264 member in GstVaapiPictureH264.
Delay ensure_context() until we actually need a VA context for allocating
new VA surfaces, and then GstVaapiPictures, but also when a real activation
of a new picture parameter set occurs, thus also implying an activation
of the related sequence parameter set.
The most important thing was to drop the global pps and sps pointers since
they may not have matched the currently activated picture parameter or
sequence parameter sets at the specified decode point.
Anoter positive side-effect is that this cleans up all occurrences of
decode_current_picture() to only keep those useful in decode_picture(),
before a new picture is allocated, or in decode_sequence_end() when
an end-of-stream or end-of-sequence condition occurred.
... aka fix regression from efaab79. In particular, ScalingList8x8[]
array was partially copied to the VAIQMatrixBufferH264. While we are
at it, also improve bounds checking and avoid copying 8x8 scaling
lists if transform_8x8_mode_flag is set to 0.
Don't copy scaling lists twice to an intermediate state. Rather, directly
use the scaling lists from GstH264PPS since they would match those provided
by SPS header, if necessary. i.e. if PPS-specific scaling lists are not
available in the bitstream.
Remove exit_picture() and exit_picture_poc() since PicOrderCnt(CurrPic)
is now updated accordingly to the standard. Besides, MMCO = 5 specific
operations are moved up to exec_ref_pic_marking_adaptive_mmco_5().
Fix adaptive memory control decoded reference picture marking process
implementation for operations 2 to 6, thus also fixing support for
long-term reference pictures.
This change only splits each individual MMCO handler into several functions
dedicated for each operation. This is needed to perform further work later
on.
Allow build with strict DPB ordering mode whereby evicted entries
are replaced by the next entries, in order instead of optimizing
it away with the last entry in the DPB.
This is only useful for debugging purpose, against a reference SW
decoder for example.
GstH264SliceHdr.n_emulation_prevention_bytes is bound to exist now that
a newer version of codecparsers/ are used if the system provided one is
now recent enough to have those required extensions.
Use newer sources from the codecparsers/ submodule for
- GstH264SliceHdr.n_emulation_prevention_bytes: EPBs;
- GstH264VUIParams.{par_n,par_d}: pixel-aspect-ratio.
Propagate pixel-aspect-ratio determined by the GStreamer codecparser
from the sequence headers.
Signed-off-by: Simon Farnsworth <simon.farnsworth@onelan.co.uk>
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
Split decode_buffer() into the core infrastructure that determines
the NAL units contained in the adapter and the actual function that
decodes the NAL unit.
Split decode_buffer() into the core infrastructure that determines
the packets contained in the adapter and the actual function that
decodes the packet data.
Fix memory leakage of empty packets, i.e. packets that only contain
the start code prefix. In particular, free empty user-data packets.
Besides, the codec parser will already fail gracefully if the packet
to parse does not have the minimum required size. So, we can also
completely drop the block of code that used to handle packets of size 4
(including the start code).
Fix return value when the second scan for start code fails. This means
there is not enough data to determine the full extents of the current
packet and the function shall return GST_VAAPI_DECODER_STATUS_ERROR_NO_DATA
in this case, instead of GST_VAAPI_DECODER_STATUS_SUCCESS.
Improve the semantics for gst_vaapi_decoder_put_buffer() when an empty
buffer is passed on. An empty buffer is a buffer with a NULL data pointer
or with a size equals to zero. In this case, that buffer is simply
skipped and the function returns TRUE. A NULL buffer argument still
marks the end-of-stream.
Rather than always making the surface fullscreen instead implement the
set_fullscreen vfunc on GstVaapiWindow and then set the shell surface
fullscreen on not depending on that.
Reviewed-by: Joe Konno <joe.konno@intel.com>
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
Mesa recently updated the <GL/glext.h> header version to Khronos version 85.
This caused the PFNGLMULTITEXCOORD2FPROC definition to be moved out of the
GL_VERSION_1_3_DEPRECATED block. However, since <GL/gl.h> also defines
GL_VERSION_1_3 to 1, the definitions in <GL/glext.h> are then not enabled,
thus leaving PFNGLMULTITEXCOORD2FPROC undefined as well.
Provide a PFNGLMULTITEXCOORD2FPROC replacement as an interim solution for
newer versions of the <GL/glext.h> header.
Maintaining the sub-buffer is rather suboptimal especially since we
were also maintaining a GstAdapter. Now, we only use the GstAdapter
thus requiring minor extra parsing when receiving avcC buffers.
This allows the compositor to optimize redraws and cull away changes
obscured by the video surface.
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
Reset, i.e. destroy then create, the decoder in _setcaps() handler only
if the underlying codec type actually changed. This makes it possible
to be more tolerant with certain MPEG-2 streams that get parsed to
form caps that are compatible with the previous state but minor changes
to "codec-data".
Make it possible to specify the maximum number of references to use within
a single VA context. This helps reducing GPU memory allocations to the useful
number of references to be used.
Forward declaring enums is not allowed by the C standard and aborts
compilation if the header file is included in a C++ project.
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
Some VA drivers (e.g. EMGD) can have completely random values for initial
display attributes. So, try to improve the discovery process to check the
initial display attribute values actually fall within valid bounds. If not,
try to reset those to some sensible values like the default value reported
through vaQueryDisplayAttributes().
Use g_object_class_install_properties() to install GstVaapiDisplay properties.
It is useful to maintain properties as GParamSpec so that to be able to raise
"notify" signals by id instead of by name in the future.
A rendering mode can be "overlay" or "texture"'ed blit.
The former mode implies that a VA surface used for rendering can't be
re-used right away for decoding, so the sink shall make provisions to
retain the associated surface proxy until the next surface is to be
displayed.
The latter mode implies that the VA surface is implicitly copied to an
intermediate backing store, or back buffer of a frame buffer, so the
associated surface proxy can be disposed right away.
The VA display attributes are mapped to properties so that to maintain the
GStreamer terminology. Properties are to be identified by name, but internal
functions are available to lookup the property by the actual VA display
attribute type.
decode_current_picture() was converted to return a gboolean instead
of a GstVaapiDecoderStatus, so we were not getting out of the decode
loop as expected, or could cause an error instead.
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
Integrate the start code prefix in the slice data buffer that is submitted
to the hardware. VA-API specifies that slice_data_offset is the offset to
the first byte of slice data. And, for MPEG-2, slice() data begins with
the slice_start_code. Some VA driver implementations (EMGD) expect this.
Use g_object_notify_by_pspec() instead of g_object_notify() so that to
avoid a property name lookup. i.e. this makes notifications faster to
the `vaapidecode' element.
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
Two elements in the luminance quantization table were wrong. So,
gst_jpeg_get_default_quantization_tables() now reconstructs tables
in zig-zag order from the standard ones (Tables K.1 and K.2).
... instead of having them pre-calculated. This saves around 1.5 KB
of data in the DSO but requires gst_jpeg_get_default_huffman_tables()
to do more work. Though, the client application may have to call that
function at most once, only.
Move display types from gstvaapipluginutil.* to gstvaapidisplay.* so that
we could simplify characterization of a GstVaapiDisplay. Also rename "auto"
type to "any", and add a "display-type" attribute.
This improves display name comparisons by always allocating a valid display
name. This also helps to disambiguate lookups by name in the global display
cache, should a new backend be implemented.
The vdeo buffer creation routines shall actually be internal to gstreamer-vaapi
plugin elements. So deprecate any explicit creation routines that are not the
new *_typed_new*() variants.
Introduce new typed constructors internal to gstreamer-vaapi plugin elements.
This avoids duplication of code, and makes it possible to further implement
generic video buffer creation routines that automatically map to base or GLX
variants.
If GLX window was created from a foreign Display, then that same Display shall
be used for subsequent glXMakeCurrent(). This means that gl_create_context()
will now use the same Display that the parent, if available.
This fixes cluttersink with the Intel GenX VA driver.
This flag is obsolete. It was meant to explicitly enable/disable VA/GLX API
support, or fallback to TFP+FBO if this API is not found. Now, we check for
the VA/GLX API by default if --enable-glx is set. If this API is not found,
we now default to use TFP+FBO.
Note: TFP+FBO, i.e. using vaPutSurface() is now also a deprecated usage and
will be removed in the future. If GLX rendering is requested, then the VA/GLX
API shall be used as it covers most usages. e.g. AMD driver can't render to
an X pixmap yet.
GStreamer -base plugins >= 0.10.31 are now required, so the checks for
new APIs like GstXOverlay::set_window_handle() and ::set_render_rectangle()
are no longer necessary.
GStreamer codecparsers-based decoders are the only supported decoders now.
Though, FFmpeg decoders are still available in gstreamer-vaapi 0.3.x series.
This is a preferred thread-safe version. Also add an inline version of
g_clear_object() if compiling with glib < 2.28.
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
Add valid flag to GstJpegQuantTable and GstJpegHuffmanTable so that
to determine whether a table actually changed since the last user
synchronization point. That way, this makes it possible for some
hardware accelerated decoding solution to upload only those tables
that changed.
Add new GstJpegHuffmanTables helper structure to hold all possible
AC/DC Huffman tables available to all components.
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
gst_jpeg_parse() now gathers all scans available in the supplied
buffer. A scan comprises of the scan header and any entropy-coded
segments or restart marker following it. The size and offset to
the associated data (ECS + RST segments) are append to a new
GstJpegScanOffsetSize structure.
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
Improve generation of presentation timestamps to be less sensitive
to input stream errors. In practise, GOP is also a synchronization
point for PTS calculation.
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
TRD and TRB fields are not large enough to hold the difference of PTS
expressed with nanosecond resolution. So, compute them from the original
VOP info.
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
Allow MPEG-2 High profile streams only if the HW supports that profile
or no High profile specific bits are used, and thus Main profile could
be used instead. i.e. chroma_format is 4:2:0, intra_dc_precision is not
set to 11 and no sequence_scalable_extension() was parsed.
In P-pictures, prediction shall be made from the two most recently
decoded reference fields. However, when the first I-frame is a field,
the next field of the current picture could be a P-picture but only a
single field was decoded so far. In this case, create a dummy picture
with POC = -1 that will be used as reference.
Some VA drivers would error out if P-pictures don't have a forward
reference picture. This is true in general but not in this very specific
initial case.
Allow fallback from simple to main profile when the HW decoder does
not support the former profile and that no sequence_header_extension()
is available to point out this.
decode_picture() could return an error when an MPEG-4 profile is not
supported for example. In this case, the underlying VA context is not
allocated and no other proper action can be taken. Likewise on exit
from decode_slice().
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
Introduce a POC field in GstVaapiPicture so that to store simpler sequential
numbers. A signed 32-bit integer should be enough for 1 year of continuous
video streaming at 60 Hz.
Use this new POC value to maintain the DPB, instead of 64-bit timestamps.
This also aligns with H.264 that will be migrated to GstVaapiDpb infrastructure.
Always prefer PTS from the demuxer layer for GOP times. If this is invalid,
i.e. demuxer could not determine the PTS or the generated PTS is lower than
max PTS from past pictures, then try to fix it up based on the duration of
a frame.
For picture PTS, simply use the GOP PTS formerly computed then use TSN to
reconstruct a current time. Also now handle wrapped TSN correctly.
Some streams, badly constructed, could have signaled an interlaced
frame while the sequence was meant to be progressive. Warn and force
frame to be progressive in this case.
Add first-field (FF) flag to GstVaapiPicture, thus not requiring is_first_field
member in each decoder. Rather, when a GstVaapiPicture is created, it is considered
as the first field. Any subsequent allocated field will become the second field.
Add gst_vaapi_picture_new_field() function that clones a picture, while
preserving the parent picture surface. i.e. the surface proxy reference
count is increased and other fields copied as is. Besides, the picture
is reset into a "non-output" mode.
Add top-field-first (TFF) and interlaced flags to GstVaapiPicture so they
could be propagated to the surface proxy when it is pushed for rendering.
Besides, top and bottom fields are now expressed with picture structure flags
from GstVaapiSurfaceRenderFlags.
If GstVaapiPicture has flag SKIPPED set, this means gst_vaapi_picture_output()
will not push the underlying surface for rendering. Besides, VC-1 skipped P-frame
has nothing to do with rendering. This only means that the currently decoded
picture is just a copy of its reference picture.
Add new "interlaced" attribute to GstVaapiSurfaceProxy. Use this in
vaapipostproc so that to handles cases where bitstream is interlaced
but almost only frame pictures are generated. In this case, we should
not be alternating between top/bottom fields.
Allow rendering flags, as a combination of GstVaapiSurfaceRenderFlags,
to be set to the video buffer. In particular, this is mostly useful for
basic deinterlacing.
Some streams have incorrect GOP timestamps, or nothing set at all.
i.e. GOP time is 00:00:00 for all GOPs. Try to recover in this case
from demuxer timestamps, which are monotonic.
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
Skip all pictures prior to the first sequence_header(). Besides,
skip all picture_data() if there was no prior picture_header().
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
VA-API expects slice_vertical_position as the initial position from the
bitstream. i.e. the direct slice() information. VA drivers will be fixed
accordingly.
Unlike what VA-API documentation defines, the slice_data_bit_offset
represents the offset to the first macroblock in the slice data, minus
any emulation prevention bytes in the slice_header().
This fix copes with binary-only VA drivers that won't be fixed any
time soon. Besides, this aligns with the current FFmpeg behaviour
that was based on those proprietary drivers implementing the API
incorrectly.
Original values from sequence_header() are 12-bit and the remaining
2 most significant bits are coming from sequence_extension().
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
6.3.15 says that "some slices may have the same slice_vertical_position,
since slices may start and finish anywhere". So, we can't submit the current
picture to the HW right away since subsequent slices would be missing.
vaRenderPicture() implicitly disposes VA buffers. Some VA drivers would
push the VA buffer object into a list of free buffers to be re-used. However,
reference pictures (and data) that was kept would explicitly release the VA
buffer object later on, thus possibly destroying a valid (re-used) object.
Besides, some other VA drivers don't support correctly the vaRenderPicture()
semantics for VA buffers disposal and would leak memory if there is no explicit
vaDestroyBuffer(). The temporary workaround is to explcitily destroy VA buffers
right after vaRenderPicture(). All VA drivers need to be aligned.
This ensures the VA context is clear when the encoded resolution
changes. i.e. make sure older picture is decoded with the older
VA context before it changes.
On sequence end, if the last decoded picture is not output for rendering,
then the proxy surface is not created. In this case, the original surface
must be released explicitly to the context.
VA drivers may have a faster means to transfer user buffers to GPU
buffers than using memcpy(). In particular, on Intel Gen graphics, we
can use pwrite(). This provides for faster upload of bitstream and can
help higher bitrates.
vaapi_create_buffer() helper function was also updated to allow for
un-mapped buffers and pre-initialized data for buffers.
Only the explicit pred_weight_table(), possibly with the inferred default
values, shall be required. e.g. don't fill in the table if weighted_pred_flag
is not set for P/SP slices.
Keep a valid reference to the proxy in GstVaapiPicture so that frames
marked as "used for reference" could be kept during the lifetime of the
picture. i.e. don't release them too soon as they could be re-used right
away.
Drop obsolete gst_vaapi_decoder_push_surface() that was no longer used.
Change gst_vaapi_decoder_push_surface_proxy() semantics to assume PTS
is already set correctly and reference count increased, if necessary.
The new API simplifies a lot reference counting and makes it more
flexible for future additions/changes. The GstVaapiCodecInfo is
also gone. Rather, new helper macros are provided to allocate
picture, slice and quantization matrix parameter buffers.
