Commit graph

32 commits

Author SHA1 Message Date
Seungha Yang
e006366206 d3d11decoder: Enable high precision clock if needed
We've been doing retry with 1ms sleep if DecoderBeginFrame()
returned E_PENDING which means application should call
DecoderBeginFrame() again because GPU is busy.
The 1ms sleep() during retry would result in usually about 15ms delay
in reality because of bad clock precision on Windows.
To improve throughput performance, this commit will enable
high precision clock only for NVIDIA platform since
DecoderBeginFrame() call on the other GPU vendors seems to
succeed without retry.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/2099>
2021-03-23 12:41:43 +00:00
Seungha Yang
5b3e316039 d3d11: Port to C++
Direct3D11 objects are COM, and most COM C APIs are verbose
(C++ is a little better). So, by using C++ APIs, we can make code
shorter and more readable.
Moreover, "ComPtr" helper class (which is C++ only) can be
utilized, that is very helpful for avoiding error-prone COM refcounting
issue/leak.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/2077>
2021-03-14 13:05:22 +09:00
Seungha Yang
cfbc580575 d3d11: Add support for deinterlacing by using ID3D11VideoProcessor interface
Add a new element d3d11deinterlace to support deinterlacing.
Similar to d3d11videosink and d3d11compositor, this element is
a wrapper bin of set of child elements including helpful
conversion elements (upload/download and color convert)
to make this element configurable between non-d3d11 elements.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/2016>
2021-02-17 03:52:19 +09:00
Seungha Yang
60e223f4fd d3d11videosink: Add support for drawing on application's own texture
Add a way to support drawing on application's texture instead of
usual window handle.
To make use of this new feature, application should follow below step.
1) Enable this feature by using "draw-on-shared-texture" property
2) Watch "begin-draw" signal
3) On "begin-draw" signal handler, application can request drawing
   by using "draw" signal action. Note that "draw" signal action
   should be happen before "begin-draw" signal handler is returned

NOTE 1) For texture sharing, creating a texture with
D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX flag is strongly recommend
if possible because we cannot ensure sync a texture
which was created with D3D11_RESOURCE_MISC_SHARED
and it would cause glitch with ID3D11VideoProcessor use case.

NOTE 2) Direct9Ex doesn't support texture sharing which was
created with D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX. In other words,
D3D11_RESOURCE_MISC_SHARED is the only option for Direct3D11/Direct9Ex interop.

NOTE 3) Because of missing synchronization around ID3D11VideoProcessor,
If shared texture was created with D3D11_RESOURCE_MISC_SHARED,
d3d11videosink might use fallback texture to convert DXVA texture
to normal Direct3D texture. Then converted texture will be
copied to user-provided shared texture.

* Why not use generic appsink approach?
In order for application to be able to store video data
which was produced by GStreamer in application's own texture,
there would be two possible approaches,
one is copying our texture into application's own texture,
and the other is drawing on application's own texture directly.
The former (appsink way) cannot be a zero-copy by nature.
In order to support zero-copy processing, we need to draw on
application's own texture directly.

For example, assume that application wants RGBA texture.
Then we can imagine following case.

"d3d11h264dec ! d3d11convert ! video/x-raw(memory:D3D11Memory),format=RGBA ! appsink"
                             ^
                             |_ allocate new Direct3D texture for RGBA format

In above case, d3d11convert will allocate new texture(s) for RGBA format
and then application will copy again the our RGBA texutre into
application's own texture. One texture allocation plus per frame GPU copy will hanppen
in that case therefore.
Moreover, in order for application to be able to access
our texture, we need to allocate texture with additional flags for
application's Direct3D11 device to be able to read texture data.
That would be another implementation burden on our side

But with this MR, we can configure pipeline in this way
"d3d11h264dec ! d3d11videosink".

In that way, we can save at least one texture allocation and
per frame texutre copy since d3d11videosink will convert incoming texture
into application's texture format directly without copy.

* What if we expose texture without conversion and application does
  conversion by itself?
As mentioned above, for application to be able to access our texture
from application's Direct3D11 device, we need to allocate texture
in a special form. But in some case, that might not be possible.
Also, if a texture belongs to decoder DPB, exposing such texture
to application is unsafe and usual Direct3D11 shader cannot handle
such texture. To convert format, ID3D11VideoProcessor API needs to
be used but that would be a implementation burden for application.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1873>
2021-01-26 18:14:37 +00:00
Seungha Yang
6f6a82d0f5 d3d11: Add support for MPEG-2 video decoding
Add DXVA/Direct3D11 API based MPEG-2 decoder element

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1969>
2021-01-25 18:05:53 +00:00
Seungha Yang
d16b5b7f7f d3d11: Allow building UWP features with Desktop features if possible
WINAPI_PARTITION_DESKTOP and WINAPI_PARTITION_APP can coexist.
Although UWP only binaries should be used for production stage,
this change will be useful for development stage

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1962>
2021-01-19 11:23:56 +00:00
Seungha Yang
0f7af4b143 d3d11: Move core methods to gst-libs
Move d3d11 device, memory, buffer pool and minimal method
to gst-libs so that other plugins can access d3d11 resource.
Since Direct3D is primary graphics API on Windows, we need
this infrastructure for various plugins can share GPU resource
without downloading GPU memory.
Note that this implementation is public only for -bad scope
for now.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/464>
2021-01-13 15:01:20 +00:00
Seungha Yang
2ca298d459 d3d11: Re-implement Desktop Duplication source
Add a new video source element "d3d11desktopdupsrc" for capturing desktop image
via Desktop Duplication based on Microsoft's Desktop Duplication sample available at
https://github.com/microsoft/Windows-classic-samples/tree/master/Samples/DXGIDesktopDuplication

This element is expected to be a replacement of existing dxgiscreencapsrc
element in winscreencap plugin.

Currently this element can support (but dxgiscreencapsrc cannot)
- Copying captured D3D11 texture to output buffer without download
- Support desktop session transition
  e.g., can capture desktop without error even in case that
  "Lock desktop" and "Permission dialog"
- Multiple d3d11desktopdupsrc elements can capture the same monitor

Not yet implemented features
- Cropping rect is not implemented, but that can be handled by downstream
- Mult-monitor is not supported. But that is also can be implemented by
  downstream element for example via multiple d3d11desktopdup elements
  with d3d11compositor

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1855>
2020-12-22 19:23:25 +00:00
Seungha Yang
96831233a3 d3d11videosink: Add a property to support rendering statistics data on window
Add a new property "render-stats" to allow rendering statistics
data on window for debugging and/or development purpose.
Text rendering will be accelerated by GPU since this implementation
uses Direct2D/DirectWrite API and Direct3D inter-op for minimal overhead.
Specifically, text data will be rendered on swapchain backbuffer
directly without any copy/allocation of extra texture.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1830>
2020-12-04 05:22:14 +09:00
Seungha Yang
6e30cbb3fc d3d11: Introduce d3d11compositor element
Add new video composition element which is equivalent to compositor
and glvideomixer elements. When d3d11 decoder elements are used,
d3d11compositor can do efficient graphics memory handling
(zero copying or at least copying memory on GPU memory space).

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1323>
2020-10-16 17:02:15 +00:00
Seungha Yang
ad8529caf7 d3d11: Disable D3D11Debug and DXGIDebug layer for UWP build
WACK (Windows App Certification Kit) doesn't seem to be happy with
the DXGIGetDebugInterface1 symbol.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1421>
2020-07-08 21:11:31 +00:00
Seungha Yang
8fe1aae96c d3d11: Load HLSL compiler library using g_module_open
Depending on OS version, available d3dcompiler library name is different.
But for UWP, we can still use the current way

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1355>
2020-06-22 14:36:16 +00:00
Seungha Yang
15365aba70 d3d11decoder: Add VP8 decoder element
New d3d11 VP8 decoder implementation

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1079>
2020-04-22 23:20:44 +00:00
Nirbheek Chauhan
387b6df948 meson: Don't use get_option('buildtype')
We should directly check the values of the `debug` and `optimization`
options instead.

`get_option('buildtype')` will return `'custom'` for most combinations
of `-Doptimization` and `-Ddebug`, but those two will always be set
correctly if only `-Dbuildtype` is set. So we should look at those
options directly.

For the two-way mapping between `buildtype` and `optimization`
+ `debug`, see this table:
https://mesonbuild.com/Builtin-options.html#build-type-options
2020-04-03 17:07:47 +05:30
Nicolas Dufresne
00d04784d3 Move CODEC base classes into it's own library
This introduce a library which contains a set of base classes which
handles the parsing and the state tracking for the purpose of decoding
different CODECs. Currently H264, H265 and VP9 are supported. These
bases classes are used to decode with low level decoding API like DXVA,
NVDEC, VDPAU, VAAPI and V4L2 State Less decoders. The new library is
named gstreamer-codecs-1.0 / libgstcodecs.
2020-03-05 03:06:16 +00:00
Seungha Yang
122a9b93eb d3d11: Add video processor object
ID3D11VideoProcessor interface provides various image conversion
methods. Note that it's analogous to VAAPI VPP.
2020-02-05 00:52:48 +00:00
Seungha Yang
e4daa2ef43 d3d11: Add support for Universal Windows Platform
Initial UWP support via new window (CoreWindow and SwapChainPanel) implementation.
2020-01-06 20:14:51 +09:00
Seungha Yang
a139c8c7e8 d3d11: Add h265 decoder element
Some DPB management implementation is taken from gstreamer-vaapi
2019-12-31 02:13:48 +00:00
Seungha Yang
0e7b6526b8 d3d11: Add vp9 decoder element
Based on gstreamer-vaapi and Chromium implemetation.
2019-12-31 02:13:48 +00:00
Seungha Yang
586390b1ba d3d11: Add h264 decoder element
New decoder implementation based on dxva2 on d3d11 APIs. The DPB
management implementation is taken from Chromium.
2019-12-31 02:13:48 +00:00
Seungha Yang
487a41d312 d3d11videosink: Add support for overlay composition
Add d3d11overlaycompositor object to draw overlay image
on render target using Blend method.
2019-12-24 19:00:45 +09:00
Seungha Yang
a5295509af d3d11: Enable dxgi debug layer and always try to pop d3d11/dxgi debug message if possible
Note that dxgi and d3d11 sdk debug will be enabled on debug build
2019-12-24 19:00:40 +09:00
Seungha Yang
9fd0b62f2d d3d11: Use configuration file for dxgi header version check 2019-12-20 11:15:12 +09:00
Seungha Yang
c9cb08fcd4 d3d11colorconvert: Split color space converter to reuse code 2019-12-05 02:29:18 +00:00
Seungha Yang
42ca123f0f d3d11: Split shader to reuse code 2019-12-05 02:29:18 +00:00
Seungha Yang
4e4fb8a577 d3d11: Add d3d11videosinkbin element
New wrapper element to support d3d11 memory upload, color conversion, and
rendering at once.
2019-12-05 02:29:18 +00:00
Seungha Yang
79ac41ec5b d3d11: Add color space conversion element using shader
Add d3d11colorconvert element for color space conversion.
Currently {RGB, YUV} to RGB conversion is supported.
2019-12-05 02:29:18 +00:00
Seungha Yang
c17c1346ee d3d11: Introduce d3d11 upload/download element
That's equivalent to glupload and gldownload elements but for d3d11
2019-12-05 02:29:18 +00:00
Seungha Yang
6c3311a39e d3d11: Refactor d3d11 memory and dxgi format usage
* Create staging texture only when the CPU access is requested.
Note that we should avoid the CPU access to d3d11 memory as mush as possible.
Incoming d3d11upload and d3d11download will take this GPU memory upload/download.

* Upload/Download texture memory from/to staging only if it needed, similar to
GstGL PBO implementation.

* Define more dxgi formats for future usage (e.g., color conversion, dxva2 decoder).
Because I420_* formats are not supported formats by dxgi, each plane should
be handled likewise GstGL separately, but NV12/P10 formats might be supported ones.
So we decide the number of d3d11memory per GstBuffer for video memory depending on
OS version and dxgi format. For instance, if NV12 is supported by OS,
only one d3d11memory with DXGI_FORMAT_NV12 texture can be allocated by this commit.
One use case of such texture is DXVA. In case DXVA decoder, it might need to produce decoded data
to one DXGI_FORMAT_NV12 instead of seperate Y and UV planes.
Such behavior will be controlled via configuration of GstD3D11BufferPool and
default configuration is separate resources per plane.
2019-12-05 02:29:18 +00:00
Aaron Boxer
6d3429af34 documentation: fixed a heap o' typos 2019-11-05 09:11:25 -05:00
Seungha Yang
58afcf09b4 meson: d3d11: Remove unnecessary dependency
d3d11 never use any API of gstreamer-allocators-1.0
2019-08-15 16:55:51 +09:00
Seungha Yang
5c3879ace6 d3d11videosink: Add new Direct3D11 video render plugin
Direct3D11 was shipped as part of Windows7 and it's obviously
primary graphics API on Windows.

This plugin includes HDR10 rendering if following requirements are satisfied
* IDXGISwapChain4::SetHDRMetaData is available (decleared in dxgi1_5.h)
* Display can support DXGI_COLOR_SPACE_RGB_FULL_G2084_NONE_P2020 color space
* Upstream provides 10 bitdepth format with smpte-st 2084 static metadata
2019-07-08 08:31:47 +00:00