Some devices (e.g., Surface Book 2, Surface Pro X) will expose
both MediaStreamType_VideoPreview and MediaStreamType_VideoRecord types
for a logical device. And for some reason, MediaStreamType_VideoPreview
seems to be selected between them while initiailzing device.
But I cannot find any documentation for the decision rule.
To be safe, we will select common formats between them.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1478>
Microsoft defines two I420 formats, one is I420, and the other is
IYUV (but both are same, just names are different).
Since both will be converted to GST_VIDEO_FORMAT_I420,
we should check both I420 and IYUV subtypes during
GstVideoFormat to Microsoft's format conversion.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1478>
In case of UWP, documentation from MS is saying that
ActivateAudioInterfaceAsync() method should be called from UI thread.
And the resulting callback might not happen until user interaction
has been made.
So we cannot wait the activation result on constructed() method.
and therefore we should return gst_wasapi2_client_new()
immediately without waiting the result if wasapi2 elements are
running on UWP application.
In addition to async operation fix, this commit includes COM object
reference counting issue around ActivateAudioInterfaceAsync() call.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1466>
Since the commit c29c71ae9d,
device activation method will be called from an internal thread.
A problem is that, CoreApplication::GetCurrentView()
method will return nullptr if it was called from non-UI thread,
and as a result, currently implemented method for accessing ICoreDispatcher
will not work in any case. There seems to be no robust way for
accessing ICoreDispatcher other then setting it by user.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1466>
If the run_async() method is expected to be called from streaming
thread and not from application thread, use INFINITE as timeout value
so that d3d11window can wait UI dispatcher thread in any case.
There is no way to get a robust timeout value from library side.
So the fixed timeout value might not be optimal and therefore
we should avoid it as much as possible.
Rule whether a timeout value can be INFINITE or not is,
* If the waiting can be cancelled by GstBaseSink:unlock(), use INFINITE.
GstD3D11Window:on_resize() is one case for example.
* Otherwise, use timeout value
Some details are, GstBaseSink:start() and GstBaseSink:stop() will be called
when NULL to READY or READY to NULL state change, so there will be no
chance for GstBaseSink:unlock() and GstBaseSink:unlock_stop()
to be called around them. So there is no other way then timeout way.
GstD3D11Window:consturcted() and GstD3D11Window:unprepare() are the case.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1461>
All subclasses are retrieving list to get target output frame, which
can be done by baseclass. And pass the ownership of the GstH264Picture
to subclass so that subclass can clear implementation dependent resources
before finishing the frame.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1449>
While retrieving supported formats by device, the last return might
not be S_OK in case that it's not supported one by us (e.g., H264, JPEG or so).
But if we've found at least one supported raw video format,
we can keep going on.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1451>
We don't need to duplicate a method for HRESULT error code to string
conversion. This patch is intended to
* Remove duplicated code
* Ensure FormatMessageW (Unicode version) and avoid FormatMessageA
(ANSI version), as the ANSI format is not portable at all.
Note that if "UNICODE" is not defined, FormatMessageA will be aliased
as FormatMessage by default.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1442>
... if target OS version was specified as Windows 10.
When enabled, desktop application can select target capture
implementation between WinRT and Win32
via GST_USE_MF_WINRT_CAPTURE environment
(e,g., GST_USE_MF_WINRT_CAPTURE=1 for WinRT impl.).
Default is Win32 implementation in case of desktop target.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1434>
Because the valid input formats for screen content coding extension is
a subset of input formats for range extension, user must specify the
profile for screen content coding extension in the caps filter
Example:
gst-launch-1.0 videotestsrc ! video/x-raw,format=NV12 ! msdkh265enc
low-power=1 ! video/x-h265,profile=screen-extended-main ! fakesink
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1389>
With the asynchronous slice decoding, we only queue up to 2 slices
per frames. That side effect is that now we are dequeuing bitstream
buffers in both decoding and presentation order. This would lead to
a bitstream buffer from a previous frame being dequeued instead of
the expected last slice buffer and lead to us trying to queue an
already queued bitstream buffer.
We now fix this by tracking pending requests. As request are executed
in decoding order, we marking a request done, we can effectively
dequeue bitstream buffer from all previous request, as they have been
executed already.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1395>
The decoder is not being access from multiple threads, instead it is
always protected by the streaming lock. For this reason, a
GstAtomicQueue for the request pool is overkill and may even introduce
unneeded overhead. Use a GstQueueArray in replacement, the
GstQueueArray is a good fit since the number of item is predictable and
unlikely to vary at run-time.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1395>
In slice mode, we'll do one request per slice. In order to recycle
bitstream buffer, and not run-out, wait for the last pending
request to complete and mark it done.
We only wait after having queued the current slice in order to reduce
that potential driver starvation and maintain performance (using dual
buffering).
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1395>
New plugin with an element for H.264 decoding with VA-API. This novel
approach, different from gstreamer-vaapi, uses gstcodecs library for
state handling.
The code is expected to looks cleaner because it uses VA-API without
further layers or wrappers.
* It uses the first supported DRM device as default VA display (other
displays will be supported through user's GstContext)
* Requires libva >= 1.6
* No multiview/stereo profiles neither interlaced streams because
gstcodecs doesn't handle them yet
* It is incompatible with gstreamer-vaapi
* Even if memory:VAMemory is exposed, it is not handled yet by any
other element
* Caps templates are generated dynamically querying VAAPI, but YV12
and I420 are added for system memory caps because they seem to be
supported for all the drivers when downloading frames onto main
memory, as they are used by xvimagesink and others, avoiding color
conversion.
* Surfaces aren't bounded to context, so they can grow beyond the DBP
size, allowing smooth reverse playback.
* There isn't yet error handling and recovery.
* 10-bit H.264 streams aren't supported by libva.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1379>
Current shader code is not compatible with HLSL profile "ps_4_0_level_9_3"
or lower. So d3dcompiler cannot compile our shader code in that case.
Note that VirtualBox is one known driver which doesn't support currently
implemented shader code.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1343>
It was introduced for later use of its enhanced feature over IMFSourceReader
such as taking photo with video preview, audio/video capturing at
the same time, etc. But currently it's not our use case, and it would
be maintenance burden.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1342>
This element uses the Desktop Duplication API to capture the desktop screen at high speed.
It supports Windows 8 or later.
It has the following features compared to other elements:
* Runs faster.
* It works in High DPI environment.
* Draws an accurate mouse cursor.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/863>
previous code releases GstBuffer too earlier. so we will see
ERROR default gstmsdkvideomemory.c:77:gst_msdk_video_allocator_get_surface: failed to get surface available
ERROR msdkbufferpool gstmsdkbufferpool.c:270:gst_msdk_buffer_pool_alloc_buffer:<msdkbufferpool0> failed to create new MSDK memory
We need to hold GstBuffer reference for msdk if the surfaced locked by msdk.
step to reproduce.
1. ffmpeg -f lavfi -i testsrc=duration=10:size=320x240:rate=30 -pix_fmt yuv420p -c:v libx265 test.265
2. GST_GL_PLATFORM=egl gst-launch-1.0 -v filesrc location=test.265 ! h265parse ! msdkh265dec ! queue ! glimagesink
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1277>
This sequence of event/data flow might happen
1) Initially we have one pending output event
1-1) Then, process the pending output data
2) No pending input event, then we should wait new pending input event
2-1) Wakeup by new pending event (but it's pending output event)
In above case, MFT will not report new pending input event
if pending output is not processed.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1325>
Add a new wasapi implementation mainly to support UWP application.
Basically the core logic of this plugin is almost identical to
existing wasapi plugin, but main target is Windows 10 (+ UWP).
Since this plugin uses WinRT APIs, this plugin most likely might not
work Windows 8 or lower.
Compared with existing wasapi plugin, additional features of this plugin are
* Fully compatible with both Windows 10 desktop and UWP application
* Supports automatic stream routing (auto fallback when device was removed)
* Support device level mute/volume control
But some features of existing wasapi plugin are not implemented
in this plugin yet
* Exclusive streaming mode is not supported
* Loopback feature is not implemented
* Cross-compile is not possible with current mingw toolchain
(meaning that MSVC and Windows 10 SDK are required to build this plugin)
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1264>
d3d11videosink has an advantage over d3dvideosink, such as
* Zero-copy playback with d3d11 decoders
* HDR rendering with 10-bit format/swapchain support
* UWP support
* Any system memory alignment/padding can be supported
* User can select target GPU device
And old d3dvideosink's functionality (e.g., navigation event, overlaycomposition)
can be covered by d3d11videosink
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1311>
all msdk output surfaces come from out_alloc_resp, so the buffer count is not resizable.
we need set min_buffers, max_buffers to same size.
steps to reproduce
1. ffmpeg -f lavfi -i testsrc=duration=10:size=320x240:rate=30:decimals=3 -pix_fmt yuv420p -c:v libx265 ~/bits/hevc/test.265
2. GST_GL_PLATFORM=egl gst-launch-1.0 -v filesrc location=~/bits/hevc/test.265 ! h265parse ! msdkh265dec ! msdkvpp ! queue ! glimagesink
you will see error like this:
ERROR default gstmsdkvideomemory.c:77:gst_msdk_video_allocator_get_surface: failed to get surface available
ERROR msdkbufferpool gstmsdkbufferpool.c:270:gst_msdk_buffer_pool_alloc_buffer:<msdkbufferpool2> failed to create new MSDK memory
ERROR msdkvpp gstmsdkvpp.c:297:create_output_buffer:<msdkvpp0> failed to create output video buffer
ERROR msdkdec gstmsdkdec.c:699:gst_msdkdec_finish_task:<msdkh265dec0> Failed to finish frame
ERROR msdkdec gstmsdkdec.c:1085:gst_msdkdec_handle_frame:<msdkh265dec0> Failed to finish a task
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1278>
Add MediaFoundation AAC encoder element.
Before Windows 10, mono and stereo channels were supported audio channels
configuration by AAC encoder MFT. However, on Windows 10,
5.1 channels support was introduced.
To expose correct range of support format by this element
whatever the OS version is, this element will enumerate
all the supported format by the AAC encoder MFT
and then will configure sink/src templates while plugin init.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1280>
If parent and child windows are running on different thread,
there is always a chance to cause deadlock as DefWindowProc() call
from child window thread might be blocked until the message
is handled by parent's window procedure.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1299>
GObject::dispose method can be called multiple times. As win32 d3d11window
has an internal thread and because GObject::dispose method could be called from the
thread, it might cause problems such as trying to join self-thread
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1299>
New video capture implementation using WinRT Media APIs for UWP app.
Due to the strict permission policy of UWP, device enumeration and
open should be done via new WinRT APIs and to get permission from users,
it will invoke permission dialog on UI.
Strictly saying, this implementation is not a part of MediaFoundation
but structurally it's very similar to MediaFoundation API.
So we can avoid some code duplication by adding this implementation
into MediaFoundation plugin.
This implementation requires UniversalApiContract version >= 6.0
which is part of Windows 10 version 1803 (Redstone 4)
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1241>
There was a missing break for the 4:4:4 case which would break the sizeimage
calculation. We don't currently have hardware that supports 4:4:4, so this
code wasn't tested. This was detected by Coverity.
CID 1463592 1463591
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1283>
The Cedrus driver would otherwise choose 1KB buffer, which is too small.
This follows what some drivers do, which is simply to use the size a
packed raw image would have. Specifications do not really guaranty any minimum
compression ratio.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1268>
This adds support for slice based decoder like the Allwinner/Cedrus driver. In
order to keep things efficient, we hold the sink buffer until we reach the end
of the picture. Note that as we don't know which one is last, we lazy queue the
slices. This effectively introduces one slice latency.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1268>
GstD3D11ColorConverter implementation is able to rescale video as well.
By doing colorspace conversion and rescale at once, we can save
one cycle of shader pipeline which will can save GPU resource.
Since this element can support color space conversion and rescale,
it's renamed as d3d11convert
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1275>
This commit does the following things to fix compilation on FreeBSD:
1. Add required typedefs to linux/types-compat.h.
2. Remove unnecessary include linux/ioctl.h and replace linux/types.h
with linux/types-compat.h. Both files do not exist on FreeBSD.
3. Check the header including makedev macro. FreeBSD does not have
sys/sysmacros.h, and including it unconditionally causes error.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1259>
else gst_video_meta_validate_alignment will report error like
"videometa gstvideometa.c:416:gst_video_meta_validate_alignment: Stride of plane 0 defined in meta (384) is different from the one computed from the alignment (320)"
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1224>
In section 9.3.4 a), segment_feature_mode have 0 for absolute and 1 for delta,
while in 19.2, it says the opposite. But the reference code, which usually
rules over the text state that 1 means absolute:
if (hdr->update_data)
{
hdr->abs = bool_get_bit(bool);
And uses it with that meaning to decide weither to override the existing value
or just add the detla. This fixes multiple decoding issues.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1245>
D3D11_CREATE_DEVICE_DEBUG flag will be used while creating d3d11 device
to activate debug layer. However, if system doesn't support the
debug layer for some reason, we should try to create d3d11 device
without the flag. Debug layer should be optional for device creation.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1217>
The Microsoft Media Foundation (MF) is the successor of DirectShow.
This commit includes two kinds of video capture implementation,
one uses IMFSourceReader interface which is available since Windows Vista
and the other is based on IMFCaptureEngine interface which is available
since Windows 8.
Note that this new video source element cannot be used in UWP app
for now, since device activation using those APIs are not allowed by MS.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/760>
Introduce GST_USE_NV_STATELESS_CODEC environment to allow user to select
primary nvidia decoder implementation. In case the environment
GST_USE_NV_STATELESS_CODEC=h264 was set, old nvidia h264 decoder element
will not be registered. Instead, both nvh264dec and nvh264sldec
factory name will create gstcodecs based new nvidia h264 decoder element.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1198>
Introduce GstH264Decoder based Nvidia H.264 decoder element.
Similar the element factory name of to v4l2 stateless codec,
this element can be configured with factory name "gstnvh264sldec".
Note that "sl" in the name stands for "stateless"
For now, existing nvh264dec covers more profile and formats
(e.g., interlaced stream) than this implementation.
However, this implementation allows us to control lower level
parameters such as decoded picture buffer management and therefore
we can get a chance to improve performance in terms of latency.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1198>
The render width/height and the vinfo was only saved upon renegotiation. This
fixes the problem by saving this metadata at the same time the buffer is
saved. The saved copy of this is needed for expose() and drain() virtual functions.
This fixes various assertion that happens on drain query.
in C, & is weaker than the ! operator and clang is giving the following
error about it.
```
../subprojects/gst-plugins-bad/sys/msdk/gstmsdkdec.c:731:7: error: logical not is only applied to the left hand side of this bitwise operator [-Werror,-Wlogical-not-parentheses]
if (!gst_msdk_context_get_job_type (thiz->context) & GST_MSDK_JOB_DECODER) {
^ ~
../subprojects/gst-plugins-bad/sys/msdk/gstmsdkdec.c:731:7: note: add parentheses after the '!' to evaluate the bitwise operator first
if (!gst_msdk_context_get_job_type (thiz->context) & GST_MSDK_JOB_DECODER) {
^
( )
../subprojects/gst-plugins-bad/sys/msdk/gstmsdkdec.c:731:7: note: add parentheses around left hand side expression to silence this warning
if (!gst_msdk_context_get_job_type (thiz->context) & GST_MSDK_JOB_DECODER) {
^
( )
1 error generated.
```
Too many decode surface would waste GPU memory. Also it seems to be
introducing additional latency depending on stream. Since nvcodec
sdk version 9.0, CUVID parser API has been providing the minimum
required number of surface. By using it, we can save GPU memory
and reduce possible latency.
We should directly check the values of the `debug` and `optimization`
options instead.
`get_option('buildtype')` will return `'custom'` for most combinations
of `-Doptimization` and `-Ddebug`, but those two will always be set
correctly if only `-Dbuildtype` is set. So we should look at those
options directly.
For the two-way mapping between `buildtype` and `optimization`
+ `debug`, see this table:
https://mesonbuild.com/Builtin-options.html#build-type-options
This code add required mechanism to try and allocate (not implemented yet)
otherwise wait for more buffers. This also comes with mechanism to terminate
the wait on flush or PAUSED_TO_READY transitions.
Use a goto to ensure that for all cases we cleanup the current picture state.
And move the src buffer allocation higher, so we don't queue a bitstream
buffer if we don't have a picture buffer to decode into.
This allow negotiating the output format through caps. Some drivers can
pipeline the decoder buffer through an image processor. This only support
colorspace conversion for now.
This implements driver stride support but only for single allocation buffers.
This code is imported from the original v4l2 plugin and adapted to the new
helper context.
In some case, when downstream does not support GstVideoMeta, we need to
normalize the stride and offset of the buffer so that downstream can render
properly with a GstVideoMeta. This code is not called when GstVideoMeta is
supported downstream.
In this patch we strictly set the GstVideoMeta width/height to the coded width
and height. Further patches will add stride support and frame copying when
downstream does not support GstVideoMeta.
Don't let downstream cause a renegotiation at random point in time. This would
lead to spurious renegotiation and the decoder state may not be recoverable.
We now pass the controls, associated to a request, queue the bitstream, qeueue
a picture buffer to decode into and finally queue the request. This now runs
until the buffer pool is exhausted. The next step will be to dequeue.
In this patch we fill the control structure with the bitstream paramter and
copy the bitstream data into V4L2 memory. Slice paramters are only the subset
of what Hantro needs, without any support for interlaced content.
This is a pooling allocator and the buffer pool does nothing other then
reusing the GstBuffer structure. Note that the pool is an internal pool, so
the start/stop/set_config virtual functions are not implemented.
This introduces the skeleton of the H264 decoder. The plugin will list the
devices and register a subclass of the GstV4L2CodecH264Dec base class. The
subclass will pick the required specific information from the GstV4L2Device
stored in the subclass structure.
This is a GstObject which will be used to hold on media and video device file
descriptor and provide abstracted ioctl calls with these descriptor. At the
moment this helper contains just enough to enumerate the supported format.
This part will be used by the plugin to register the CODEC specific elements..
Most of the features we need are very early or not expose yet in the uAPI.
Using an internal copy ensure that we everything we need is defined avoiding
to add load of checks and conditionnal code.
This introduces a GstV4L2CodecDevice structure and helper to retrieve a
list of CODEC device drivers. In order to find the device driver we
enumerate all media devices with UDEV. We then get the media controller
topology and locate a entity with function encoder or decoder and make
sure it is linked to two V4L2 IO entity pointing to the same device
node.
The media driver can support HEVC 8-bit 422 encoding for non-lowpower
mode since ICL[1], so VPP is not needed for this case.
Sample pipeline:
gst-launch-1.0 videotestsrc ! video/x-raw,format=YUY2 ! msdkh265enc ! \
filesink location=output.h265
[1] https://github.com/intel/media-driver#decodingencoding-features
DXVA supports two kinds of texture structure for DPB, one is
"1) texture array" and the other is "2) array of texture".
1) is a type of texture which is single ID3D11Texture2D object having
ArraySize greater than one. So the ID3D11Texture2D itself is a set of texture.
Each sub texture of this type mush have identical resolution, format and so on,
and the number of sub texture in a texture array is fixed.
2) is an array of usual ID3D11Texture2D object. That means each
ID3D11Texture2D is independent each other and might have different resolution as well.
Moreover, we can modify the number of frames of the array dynamically.
This type is more flexible than "1) texture array" in terms of dynamic
behavior and also this type of texture can be used for shader resource view
but "1) texture array" couldn't be.
If "2) array of texture" is supported by driver, DXVA spec is saying that
it's preferred format over "1) texture array" in terms of performance.
The set of supported color space by DXGI is not full combination of
our colorimetry. That means we should convert color space to one
of supported color space by DXGI. This commit modifies the color space
selection step so that d3d11window can find the best matching DXGI color space
first and then the selected input/output color space will be referenced
by shader and/or d3d11videoprocessor.
Adds properties to the devices listed in GstDeviceMonitor by the
applemedia plugin.
These properties are:
- device.api (always set to "avf")
- avf.unique_id
- avf.model_id
- avf.manufacturer (except on iOS)
- avf.has_flash
- avf.has_torch
Everything except device.api is taken directly from the AVCaptureDevice object
provided by AVFoundation.
