According to the SPEC:
The frame id numbers (represented in display_frame_id, current_frame_id,
and RefFrameId[ i ]) are not needed by the decoding process, but allow
decoders to spot when frames have been missed and take an appropriate action.
So we should just print out warning and should not return error in parser when
mismatching. The decoder itself is already robust to handle the reference missing.
Fixes#3622
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/7047>
A large refactoring commit for adding features and improve performance
* Reuse internal converter and overlay compositor:
Converter can be reused as long as input and display formats are not
changed. Also overlay compositor reconstruction is required only if
display format is changed
* Don't wait for full GPU flush on resize or close:
D3D12 swapchain requires GPU idle in order to resize backbuffer.
Thus CPU side waiting is required for swapchain related commands
to be finished. However, don't need to wait for full GPU flushing.
* Support multiple sink on a single external window
Keep installed subclass window procedure even if there's no associated
our internal HWND. This will make window procedure hooking less racy.
Then parent HWND's message will be transferred to our internal HWNDs
if needed.
* Adding support for window handle update
Application can change target HWND even when videosink is playing or
paused state. So, users can call gst_video_overlay_set_window_handle()
against d3d12videosink anytime. The videosink will be able to update
internal state and setup resource upon requested.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/7013>
Observed Intel GPU driver crash when multiple decoders are
configured in a process. It might be because of frequent
command queue alloc/free or too many in-flight decoding commands.
In order to make command queue persistent and limit the number of
in-flight command lists, holds global decoding command queue.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/7019>
Not all GPUs can support arbitrary offset of
D3D12_PLACED_SUBRESOURCE_FOOTPRINT when copying GPU memory between
texture and buffer. Instead of calculating size/offset per plane,
calculate the entire size and offsets at once.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/6967>
drm_fourcc.h should be picked up via the pkgconfig include, not the
system includedir directly.
Also consolidate the libdrm usage in va and msdk.
All this allows it to be picked up consistently (via the subproject,
for example).
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/6932>
From the spec (chapter 34, v1.3.283):
````
UNORM: the components are unsigned normalized values in the range [0, 1]
SRGB: the R, G and B components are unsigned normalized value that represent
values using sRGB nonlinear encoding, while the A component (if one
exists) is a regular unsigned normalized value
```
The difference is the storage encoding, the first one is aimed for image
transfers, while the second is for shaders, mostly in the swapchain stage in the
pipeline, and it's done automatically if needed [1].
As far as I have checked, other frameworks (FFmpeg, GTK+), when import or export
images from/to Vulkan, use exclusively UNORM formats, while SRGB formats are
ignored.
My conclusion is that Vulkan formats are related on how bits are stored in
memory rather their transfer functions (colorimetry).
This patch does two interrelated changes:
1. It swaps certain color format maps to try first, in both
gst_vulkan_format_from_video_info() and gst_vulkan_format_from_video_info_2(),
the UNORM formats, when comparing its usage, and later check for SRGB.
2. It removes the code that check for colorimetry in
gst_vulkan_format_from_video_info_2(), since it not storage related.
1. https://community.khronos.org/t/noob-difference-between-unorm-and-srgb/106132/7
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/6797>
Doing it in gst_play_new() means that bindings that directly call
g_object_new() with the GType wouldn't end up initializing both.
This affects at least the Python and GJS bindings.
gst_init() is nonetheless only called from gst_play_new() once because
calling it from class_init would likely lead to problems as that's
called from somewhere in the middle of GObject.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/6801>
Use of VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT instead of
specific VK_PIPELINE_STAGE_2_VIDEO_DECODE_BIT_KHR
Fix for VUID-vkCmdPipelineBarrier2-srcStageMask-03849
pDependencyInfo->pImageMemoryBarriers[0].srcStageMask
(VK_PIPELINE_STAGE_2_VIDEO_DECODE_BIT_KHR) is not compatible with
the queue family properties
(VK_QUEUE_GRAPHICS_BIT|VK_QUEUE_COMPUTE_BIT|VK_QUEUE_TRANSFER_BIT|
VK_QUEUE_SPARSE_BINDING_BIT|VK_QUEUE_PROTECTED_BIT) of this
command buffer. The Vulkan spec states: The srcStageMask member
of any element of the pMemoryBarriers, pBufferMemoryBarriers, or
pImageMemoryBarriers members of pDependencyInfo must only
include pipeline stages valid for the queue family that was
used to create the command pool that commandBuffer was allocated
from (
https://www.khronos.org/registry/vulkan/specs/1.3-extensions/
html/vkspec.html#VUID-vkCmdPipelineBarrier2-srcStageMask-03849)
The frame barrier should use a compatible srcStageMask for all
the queues.
Remove reset_pipeline_stage_mask as it is redundant
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/6780>
Ideally, GPU waiting should be scheduled just before executing command list.
But handling the case outside of converter is a bit complicated.
Under an assumption that constructed command list will be executed
immediately, schedules GPU-side waiting inside of conversion method
to simplify the flow.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/6749>
Since we don't want to expose video decoding API outside of GStreamer, the
header is removed from installation and both source files are renamed as
-private.
The header must remain in gst-libs because is referred by GstVulkanQueue,
which's the decoder factory.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/6723>
First it derived mapping was disabled for P010 formats, but also there's an
issue with interlaced frames.
It would be possible to disable derived mapping only for interlaced (H.264
decoder and vadeinterlace) but it would spread the hacks along the code. It's
simpler and contained to disable derived completely for Mesa <23.3
Fixes: #3450
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/6729>
Instead of duplicating the GStreamer format to DRM fourcc mapping, this patch
uses the GstVideo library helpers. This duplicates the big O of looking for,
since the two lists are traversed, but it's less error prone.
Partially reverts commit 547f3e8622.
Fixes: #3354
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/6731>
If waylandsink received buffer rate is high which causes frame
drop, the cached staged buffer will be replaced when next buffer
needs to be rendered and be freed after redraw. But there is
chance to get memory leak if ended without redraw. So need to
free staged buffer when do gst_wl_window_finalize().
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/6670>
In addition to device removed status monitoring in gst_d3d11_result()
method, if ID3D11Device4 interface is available,
an event handle will be used for device removed status update.
And "device-removed" signal is removed since applications can monitor
the device removed status via gobject notify
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/6699>
Adding new property in order to notify users of device removed status.
Once device removed status is detected, application should release
all ID3D12Device objects corresponding to the adapter, including
GstD3D12Device object. Otherwise D3D12CreateDevice() call for the
adapter will fail.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/6699>
It seems that when D3D11CreateDevice collides in time
with other D3D11 calls, in particular the proccess of
creating a shader, it can corrupt the memory in the driver.
D3D11 spec doesn't seem to require any thread safety from
D3D11CreateDevice. Following MSDN, it is supposed to be called
in the beginning of the proccess, while GStreamer calls it with each
new pipeline.
Such crashes in the driver were frequently reproducing on the
Intel UHD 630 machine.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/6686>
Fix
gstanalyticsmeta.c:134: Warning: GstAnalytics: "@instance"
parameter unexpected at this location
warning (caused by the extraneous empty line in the doc chunk)
and align function arguments with documentation and header file
(handle -> instance).
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/6585>
`on_error()` can be called with a NULL details structure, so in that situation
the `gst_structure_copy()` would raise a critical warning. Create an empty
structure instead of attempting to copy a NULL one.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/6385>
When this error gets caught the GstD3D11Device object raises the new
"device-removed" signal. This allows to handle the error from outside:
stop the playback, re-create the player, replace the catched GstContext by
the new one.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/6193>
Do not chain up to parent's GstBufferPool::start() which will do
preallocation. We don't want it to be preallocated
since there are various cases where negotiated downstream buffer pool is
not used at all (e.g., zero-copy decoding, IPC elements).
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/6326>
Gets being released memory back to queue even if allocator is flushing
in order to count the number of outstanding memory objects.
Also, clear queue if there's no outstanding memory object and
allocator is flushing
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/6240>
Syncrhonizing buffer commits to the streaming thread can lead to
dropped frames when frame callbacks are not processed before the
next frame is ready for rendering. Depending on the drift between
the wayland compositor and buffer source timings, this can lead to
periods of significant frame drop, especially when the media frame
rate is close to the display frame rate.
Cache buffers in the streaming thread and peform commits on the
display thread to eliminate the buffer commit racing.
The implementation is the same for both waylandsink and gtkwaylandsink,
so move it to the common wayland library under gst-lib.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/6133>
Add synchonized versions of wl_display_sync() and wl_callback_destroy()
that will ensure that to callbacks can be managed in a thread safe way
on the display queue even when they are dispatched from a separate
thread.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/6133>
This inherits from the same rule as gst_buffer_add_meta
```
gst-mpegtspesmetadatameta.h:98: Warning: GstMpegts:
gst_buffer_add_mpegts_pes_metadata_meta: return value: Invalid non-constant
return of bare structure or union; register as boxed type or (skip)
```
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/6146>
MaxDpbSize specified in A.4.2 tells upper bound of decoded picture
buffer size but does not tell actual required size.
Use max_dec_pic_buffering value as a dpb size. Some backends
such as DXVA and NVDEC might require pre-allocated DPB buffer
and unnecessary large DPB size will result in waste of GPU memory.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/6101>
- gst_analytics_cls_mtd_get_length() return a gsize, this type dicated index
type for gst_analytics_cls_mtd_get_quark() and
gst_analytics_cls_mtd_get_level().
- Minor cleanup/improvement on index validation.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/6018>
A client may map dmabufs without the intention to either read or write
to the memory. One example is clients wanting to use the
`gst_video_frame_map()` helper function.
Thus, in order to make buffers from `GstVaDmabufAllocator` conveniently
usable, ignore the modifier check if the client specified neither
`GST_MAP_READ` nor `GST_MAP_WRITE`.
Also skip the `va_sync_surface()` call in that case, as it's likely only
needed for CPU reads/writes.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/5965>
clang does not like the array index assignment without the `=` sign in
it. This is a gnu extension I believe, and adding the sign is proper.
This fixes the following two warnings:
```
../subprojects/gst-plugins-bad/gst-libs/gst/vulkan/gstvkvideo-private.c:32:40:
warning: use of GNU 'missing =' extension in designator [-Wgnu-designator]
[GST_VK_VIDEO_EXTENSION_DECODE_H264] {
^
=
../subprojects/gst-plugins-bad/gst-libs/gst/vulkan/gstvkvideo-private.c:36:40:
warning: use of GNU 'missing =' extension in designator [-Wgnu-designator]
[GST_VK_VIDEO_EXTENSION_DECODE_H265] {
^
=
```
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/5996>
This protocol does what it says on the box, avoiding the need for a 1x1
wl_shm buffer.
A wayland-projects wrap has been added for users who do not have v1.26
available.
This commit was partly authored by Robert Mader.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/2662>
The pool currently defaults to performing a layout transition to
VK_IMAGE_LAYOUT_TRANSFER_DST_OPTIMAL, with some special exceptions for
video usages. This may not be a legal transition depending on the usage.
Provide an API to explicitly control the initial image layout.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/5881>
Since the AV1 specification is not explicitly mentioning about
the array size bounds, array sizes in scalability structure
should be defined as possible maximum sizes that can have.
Also, this commit removes GST_AV1_MAX_SPATIAL_LAYERS define from
public header which is API break but the define is misleading
and this patch is introducing ABI break already
ZDI-CAN-22300
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/5823>
- AU boundary detection reviewed to follow more closely H.264 spec. and more
specifically clauses 7.4.1.2.3 and 7.4.1.2.4.
- The gist of the changes is a look-a-head in then next AU required identify the
last vcl-nal of current AU and firt vcl-nal of next AU (according to
7.4.1.2.4) followed by the identification of the first nal of next AU
(according to 7.4.1.2.3).
- A backlog of all nals of current AU and next AU up to the point where current
AU can identified completed is kept.
- In NAL alignement mode vcl-nal are sent immediatly but the history is kept to
allow AU boundary detection. Non-vcl-nal can be delayed up to the reception of
the next vcl-nal to allow a correct AUD insertion.
- Based on this improved AU boudary detection we can avoid erronous AUD
insertion, like the one highlighted by test
test_parse_sliced_with_prefix_and_sei_nal_au.
- Add support for MVC AU boundary detection. (H.7.4.1.2.4)
- Explicitly report SVC not supported. We don't have the SVC NAL parsing
required to identify boundary. (missing dependency_id and quality_id fields
from SVC, see G.7.4.1.2.4)
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/5741>
VA drivers allocate surfaces given their properties, so there's no need to
provide a buffer size to the VA pool.
Though, the buffer size is provided by the driver, or the canonical size
is used for single planed surfaces.
This patch removes the need to provide a size for the function
gst_va_pool_new_with_config() and adds a helper method to retrieve the surface
size, gst_va_pool_get_buffer_size(). Also change the callers accordingly.
Changes for custom VA pool creation will be addressed in the following commits.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/5805>
The original idea was to select the type of mapping (either using derive images
or downloading the image) in runtime, under the assumption that both methods
shared the same memory layout (offsets and strides), because a single
GstVideoMeta is assigned by the buffer pool at allocation time. Nonetheless, in
recent hardware this assumption is invalid, raising memory access errors.
This patch removes completely the mapping type selection at runtime, using the
method selected when the allocator is configured, synced with the bufferpool
allocation.
This problem was fixed originally for iHD driver only. But now it makes sense to
remove all of it.
Original-patch-by: Mengkejiergeli Ba <mengkejiergeli.ba@intel.com>
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/5760>
When exporting a DMABuf from a VASurface the user might tell that the surface
was allocated with certain fourcc, but the returned VADRMPRIMESurfaceDescriptor
migth tell a different fourcc, as in the case or radeonsi driver, for duplicated
fourcc, such as YUY2 and YUYV.
Originally it was supposed to be a failed exportation. This patch relax this
validation by allowing different fourcc.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/5760>
Clip tile rows and cols to 64 as describe in AV1 specification
to avoid writing outside array range but preserve sb_cols
and sb_rows value which are used to futher computation.
Fixes ZDI-CAN-22226 / CVE-2023-44429
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/5702>
Direct3D feature level 10 supported GPUs were released
more than 15 years ago, around the time when Windows
Vista / 7 were released. Also our d3d11 plugin/library
does not support feature level 9.x very well already.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/5709>
- GstAnalyticRelationMeta is a base class for analytics
meta. It's able to store analytics results (GstAnalyticRelatableMtd)
and describe the relation between each analysis results.
- GstAnalysisRelationMeta also contain an algorithm able to explore
analysis results relation using a bfs.
- Relation(edge) between analysis results (vertice) are stored in an adjacency-matrix
that allow to quickly identify if two analysis results are related and by
which relation they related. It also work for indirect relation
and can provide the path of analysis results by which two
analysis results are related.
- One allocation per buffer to store analysis results. Here we rely on
the application to guess how much space will be required to store all
analysis results. This is something that could be improved
significantly but it's a starting point.
- Define common analysis results, classification, object-detection,
tracking that are subclass of GstAnalyticRelatableMtd. The also
provide exemple of how to extend GstAnalyticRelatableMtd to have them
benefit for the mechanim to express relation with other analysis
results.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/4962>
