Commit graph

26 commits

Author SHA1 Message Date
Seungha Yang
60e223f4fd d3d11videosink: Add support for drawing on application's own texture
Add a way to support drawing on application's texture instead of
usual window handle.
To make use of this new feature, application should follow below step.
1) Enable this feature by using "draw-on-shared-texture" property
2) Watch "begin-draw" signal
3) On "begin-draw" signal handler, application can request drawing
   by using "draw" signal action. Note that "draw" signal action
   should be happen before "begin-draw" signal handler is returned

NOTE 1) For texture sharing, creating a texture with
D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX flag is strongly recommend
if possible because we cannot ensure sync a texture
which was created with D3D11_RESOURCE_MISC_SHARED
and it would cause glitch with ID3D11VideoProcessor use case.

NOTE 2) Direct9Ex doesn't support texture sharing which was
created with D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX. In other words,
D3D11_RESOURCE_MISC_SHARED is the only option for Direct3D11/Direct9Ex interop.

NOTE 3) Because of missing synchronization around ID3D11VideoProcessor,
If shared texture was created with D3D11_RESOURCE_MISC_SHARED,
d3d11videosink might use fallback texture to convert DXVA texture
to normal Direct3D texture. Then converted texture will be
copied to user-provided shared texture.

* Why not use generic appsink approach?
In order for application to be able to store video data
which was produced by GStreamer in application's own texture,
there would be two possible approaches,
one is copying our texture into application's own texture,
and the other is drawing on application's own texture directly.
The former (appsink way) cannot be a zero-copy by nature.
In order to support zero-copy processing, we need to draw on
application's own texture directly.

For example, assume that application wants RGBA texture.
Then we can imagine following case.

"d3d11h264dec ! d3d11convert ! video/x-raw(memory:D3D11Memory),format=RGBA ! appsink"
                             ^
                             |_ allocate new Direct3D texture for RGBA format

In above case, d3d11convert will allocate new texture(s) for RGBA format
and then application will copy again the our RGBA texutre into
application's own texture. One texture allocation plus per frame GPU copy will hanppen
in that case therefore.
Moreover, in order for application to be able to access
our texture, we need to allocate texture with additional flags for
application's Direct3D11 device to be able to read texture data.
That would be another implementation burden on our side

But with this MR, we can configure pipeline in this way
"d3d11h264dec ! d3d11videosink".

In that way, we can save at least one texture allocation and
per frame texutre copy since d3d11videosink will convert incoming texture
into application's texture format directly without copy.

* What if we expose texture without conversion and application does
  conversion by itself?
As mentioned above, for application to be able to access our texture
from application's Direct3D11 device, we need to allocate texture
in a special form. But in some case, that might not be possible.
Also, if a texture belongs to decoder DPB, exposing such texture
to application is unsafe and usual Direct3D11 shader cannot handle
such texture. To convert format, ID3D11VideoProcessor API needs to
be used but that would be a implementation burden for application.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1873>
2021-01-26 18:14:37 +00:00
Seungha Yang
0f7af4b143 d3d11: Move core methods to gst-libs
Move d3d11 device, memory, buffer pool and minimal method
to gst-libs so that other plugins can access d3d11 resource.
Since Direct3D is primary graphics API on Windows, we need
this infrastructure for various plugins can share GPU resource
without downloading GPU memory.
Note that this implementation is public only for -bad scope
for now.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/464>
2021-01-13 15:01:20 +00:00
Seungha Yang
96831233a3 d3d11videosink: Add a property to support rendering statistics data on window
Add a new property "render-stats" to allow rendering statistics
data on window for debugging and/or development purpose.
Text rendering will be accelerated by GPU since this implementation
uses Direct2D/DirectWrite API and Direct3D inter-op for minimal overhead.
Specifically, text data will be rendered on swapchain backbuffer
directly without any copy/allocation of extra texture.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1830>
2020-12-04 05:22:14 +09:00
Seungha Yang
afe941249e d3d11window: Add unprepare method to clear internal resource
GObject::dispose method can be called multiple times. As win32 d3d11window
has an internal thread and because GObject::dispose method could be called from the
thread, it might cause problems such as trying to join self-thread

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1299>
2020-05-27 04:59:50 +09:00
Seungha Yang
f9f6916aa1 d3d11window: Reorganize display colorspace selection
The set of supported color space by DXGI is not full combination of
our colorimetry. That means we should convert color space to one
of supported color space by DXGI. This commit modifies the color space
selection step so that d3d11window can find the best matching DXGI color space
first and then the selected input/output color space will be referenced
by shader and/or d3d11videoprocessor.
2020-03-28 11:45:48 +00:00
Seungha Yang
9b7c20bfca d3d11videosink: Clarify the meaning of various width and height variables
* Remove redundant variables for width/height and par from GstD3D11Window.
  GstVideoInfo holds all the values.
* Don't need to pass par to gst_d3d11_window_prepare().
  It will be parsed from caps again
* Remove duplicated math

Fixing regression of the commit 9dada90108
2020-03-02 22:01:43 +09:00
Seungha Yang
2aa9f0bd6c d3d11window: Don't create swapchain again per caps change
Creating swapchain is relatively heavy operation. If output dxgi format
is not being chagned, we don't need to destroy and create swachain again.
2020-02-05 00:52:48 +00:00
Seungha Yang
c1d2d9171d d3d11window: Invoke initial resize method from baseclass
... instead of calling from subclass in order for baseclass to handle
more things between swapchain creation and resource creation.
2020-02-05 00:52:48 +00:00
Seungha Yang
7aad9187e4 d3d11videosink: Use ID3D11VideoProcessor interface
...for color space conversion if available

ID3D11VideoProcessor is equivalent to DXVA-HD video processor
which might use specialized blocks for video processing
instead of general GPU resource. In addition to that feature,
we need to use this API for color space conversion of DXVA2 decoder
output memory, because any d3d11 texture arrays that were
created with D3D11_BIND_DECODER cannot be used for shader resource.

This is prework for d3d11decoder zero-copy rendering and also
for conditional HDR tone-map support.
Note that some Intel platform is known to support tone-mapping
at the driver level using this API on Windows 10.
2020-02-05 00:52:48 +00:00
Seungha Yang
a967db3b20 d3d11format: Add util methods for mapping DXGI color space with ours
Move color space mapping and hdr10 metadata conversion methods to
d3d11format in order to reuse the code.
2020-02-05 00:52:48 +00:00
Seungha Yang
e4daa2ef43 d3d11: Add support for Universal Windows Platform
Initial UWP support via new window (CoreWindow and SwapChainPanel) implementation.
2020-01-06 20:14:51 +09:00
Seungha Yang
5298d95195 d3d11window: Make use of partial presentation with IDXGISwapChain1::Present1
Since we might draw on partial area of backbuffer in case of force-aspect-ratio,
presenting only updated area is more efficient way.

See also https://docs.microsoft.com/ko-kr/windows/win32/direct3ddxgi/dxgi-1-2-presentation-improvements
2019-12-24 20:10:54 +09:00
Seungha Yang
487a41d312 d3d11videosink: Add support for overlay composition
Add d3d11overlaycompositor object to draw overlay image
on render target using Blend method.
2019-12-24 19:00:45 +09:00
Seungha Yang
62c8d19e01 d3d11window: Change aspect-ratio mode from window thread
Call DXGI API from window thread as much as possible
2019-12-20 19:21:02 +09:00
Seungha Yang
1b687d556a d3d11videosink: Add support for full screen mode
borderless top-most style full screen mode support.
Basically fullscreen toggle mode is disabled by default. To enable it
use "fullscreen-toggle-mode" property to allow fullscreen mode change
by user input and/or property.
2019-12-20 19:21:02 +09:00
Seungha Yang
0788492461 d3d11: Use GRecMutex to protect immediate context and dxgi API call
In some cases, rendering and dxgi (e.g., swapchain) APIs should be
called from window message pump thread, but current design (dedicated d3d11 thread)
make it impossible. To solve it, change concurrency model to locking based one
from single-thread model.
2019-12-20 19:21:03 +09:00
Seungha Yang
1424de0a4a d3d11window: Don't hold backbuffer interface
We don't need to hold it
2019-12-20 11:15:12 +09:00
Seungha Yang
23b47a7ec1 d3d11window: Use allow-tearing mode if device supports it
As the recommendation from MS.
2019-12-20 11:15:12 +09:00
Seungha Yang
9dada90108 d3d11videosink: Remove resizing window hack and unify resizing flow
In earlier implementation of d3d11videosink where no shader was implemented,
the aspect ratio and render size were adjusted by manipulating the backbuffer size
with unintuitive formula. Since now we do color conversion and resize using
shader, we can remove the hack.
2019-12-20 11:15:12 +09:00
Seungha Yang
64eb1d46b5 d3d11window: Always draw to internal window
... and use SetParent() WIN32 API when external window is used.
Depending on DXGI swap effect, the external window might not be
reusable by another backend. To preserve the external window's property
and setting, drawing to internal window seems to be safer way.
2019-12-16 22:24:29 +09:00
Seungha Yang
9835289239 d3d11window: Do not check shader resource view if not doing conversion
If d3d11window does not convert format internally, shader resource view
is not required. Note that shader resource view is used for
color conversion using shader but when conversion is not required,
we just copy input input texture to backbuffer.
2019-12-10 04:06:28 +00:00
Seungha Yang
ca3ddf7848 d3d11videosink: Add color conversion support
Draw to back buffer texture directly. It would reduce the number of
copy at least once when color conversion is required.
2019-12-05 02:29:18 +00:00
Seungha Yang
59d9589722 d3d11videosink: Draw window with cached texture on resize
This would render nicer than presenting scene with broken aspect ratio,
especially in case of low framerate.
2019-12-05 02:29:18 +00:00
Seungha Yang
44cb5b4ff7 d3d11videosink: Don't try to post error message by d3d11window self
The d3d11window isn't GstElement. To post error message, proxy it
to d3d11videosink instead.
2019-10-04 21:15:15 +09:00
Seungha Yang
5909a495a7 d3d11videosink: Take into account pixel aspect ratio
Fix unexpected cropping with non 1:1 pixel aspect-ratio.

The actual buffer width/height should be passed to gst_d3d11_window_render(),
instead of the calculated resolution. The width/height
values are parameters for copying d3d11 video memory.
Also, aspect-ratio should be considered on resize callback
to decide render rectangle size.
2019-08-15 16:55:37 +09:00
Seungha Yang
5c3879ace6 d3d11videosink: Add new Direct3D11 video render plugin
Direct3D11 was shipped as part of Windows7 and it's obviously
primary graphics API on Windows.

This plugin includes HDR10 rendering if following requirements are satisfied
* IDXGISwapChain4::SetHDRMetaData is available (decleared in dxgi1_5.h)
* Display can support DXGI_COLOR_SPACE_RGB_FULL_G2084_NONE_P2020 color space
* Upstream provides 10 bitdepth format with smpte-st 2084 static metadata
2019-07-08 08:31:47 +00:00