gstreamer/sys/d3d11
Seungha Yang 60e223f4fd d3d11videosink: Add support for drawing on application's own texture
Add a way to support drawing on application's texture instead of
usual window handle.
To make use of this new feature, application should follow below step.
1) Enable this feature by using "draw-on-shared-texture" property
2) Watch "begin-draw" signal
3) On "begin-draw" signal handler, application can request drawing
   by using "draw" signal action. Note that "draw" signal action
   should be happen before "begin-draw" signal handler is returned

NOTE 1) For texture sharing, creating a texture with
D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX flag is strongly recommend
if possible because we cannot ensure sync a texture
which was created with D3D11_RESOURCE_MISC_SHARED
and it would cause glitch with ID3D11VideoProcessor use case.

NOTE 2) Direct9Ex doesn't support texture sharing which was
created with D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX. In other words,
D3D11_RESOURCE_MISC_SHARED is the only option for Direct3D11/Direct9Ex interop.

NOTE 3) Because of missing synchronization around ID3D11VideoProcessor,
If shared texture was created with D3D11_RESOURCE_MISC_SHARED,
d3d11videosink might use fallback texture to convert DXVA texture
to normal Direct3D texture. Then converted texture will be
copied to user-provided shared texture.

* Why not use generic appsink approach?
In order for application to be able to store video data
which was produced by GStreamer in application's own texture,
there would be two possible approaches,
one is copying our texture into application's own texture,
and the other is drawing on application's own texture directly.
The former (appsink way) cannot be a zero-copy by nature.
In order to support zero-copy processing, we need to draw on
application's own texture directly.

For example, assume that application wants RGBA texture.
Then we can imagine following case.

"d3d11h264dec ! d3d11convert ! video/x-raw(memory:D3D11Memory),format=RGBA ! appsink"
                             ^
                             |_ allocate new Direct3D texture for RGBA format

In above case, d3d11convert will allocate new texture(s) for RGBA format
and then application will copy again the our RGBA texutre into
application's own texture. One texture allocation plus per frame GPU copy will hanppen
in that case therefore.
Moreover, in order for application to be able to access
our texture, we need to allocate texture with additional flags for
application's Direct3D11 device to be able to read texture data.
That would be another implementation burden on our side

But with this MR, we can configure pipeline in this way
"d3d11h264dec ! d3d11videosink".

In that way, we can save at least one texture allocation and
per frame texutre copy since d3d11videosink will convert incoming texture
into application's texture format directly without copy.

* What if we expose texture without conversion and application does
  conversion by itself?
As mentioned above, for application to be able to access our texture
from application's Direct3D11 device, we need to allocate texture
in a special form. But in some case, that might not be possible.
Also, if a texture belongs to decoder DPB, exposing such texture
to application is unsafe and usual Direct3D11 shader cannot handle
such texture. To convert format, ID3D11VideoProcessor API needs to
be used but that would be a implementation burden for application.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1873>
2021-01-26 18:14:37 +00:00
..
gstd3d11basefilter.c d3d11: Move core methods to gst-libs 2021-01-13 15:01:20 +00:00
gstd3d11basefilter.h d3d11: Move core methods to gst-libs 2021-01-13 15:01:20 +00:00
gstd3d11colorconvert.c d3d11: Don't use hardcoded maximum resolution value 2021-01-20 19:31:27 +09:00
gstd3d11colorconvert.h d3d11convert: Add new subclasses for only color convert or resize 2020-10-16 17:02:15 +00:00
gstd3d11colorconverter.c d3d11: Move core methods to gst-libs 2021-01-13 15:01:20 +00:00
gstd3d11colorconverter.h d3d11: Move core methods to gst-libs 2021-01-13 15:01:20 +00:00
gstd3d11compositor.c d3d11: Don't use hardcoded maximum resolution value 2021-01-20 19:31:27 +09:00
gstd3d11compositor.h d3d11: Move core methods to gst-libs 2021-01-13 15:01:20 +00:00
gstd3d11compositorbin.c d3d11: Don't use hardcoded maximum resolution value 2021-01-20 19:31:27 +09:00
gstd3d11compositorbin.h d3d11: Move core methods to gst-libs 2021-01-13 15:01:20 +00:00
gstd3d11decoder.c d3d11: Add support for MPEG-2 video decoding 2021-01-25 18:05:53 +00:00
gstd3d11decoder.h d3d11: Add support for MPEG-2 video decoding 2021-01-25 18:05:53 +00:00
gstd3d11desktopdup.cpp d3d11: Move core methods to gst-libs 2021-01-13 15:01:20 +00:00
gstd3d11desktopdup.h d3d11: Move core methods to gst-libs 2021-01-13 15:01:20 +00:00
gstd3d11desktopdupsrc.c d3d11: Don't use hardcoded maximum resolution value 2021-01-20 19:31:27 +09:00
gstd3d11desktopdupsrc.h d3d11: Move core methods to gst-libs 2021-01-13 15:01:20 +00:00
gstd3d11download.c d3d11: Don't use hardcoded maximum resolution value 2021-01-20 19:31:27 +09:00
gstd3d11download.h d3d11: Move to G_DECLARE_FINAL_TYPE 2020-10-04 15:44:03 +00:00
gstd3d11h264dec.c d3d11: Move core methods to gst-libs 2021-01-13 15:01:20 +00:00
gstd3d11h264dec.h d3d11decoder: Register elements per GPU device with capability check 2020-02-18 11:58:45 +00:00
gstd3d11h265dec.c d3d11: Move core methods to gst-libs 2021-01-13 15:01:20 +00:00
gstd3d11h265dec.h d3d11decoder: Register elements per GPU device with capability check 2020-02-18 11:58:45 +00:00
gstd3d11mpeg2dec.c d3d11: Add support for MPEG-2 video decoding 2021-01-25 18:05:53 +00:00
gstd3d11mpeg2dec.h d3d11: Add support for MPEG-2 video decoding 2021-01-25 18:05:53 +00:00
gstd3d11overlaycompositor.c d3d11: Move core methods to gst-libs 2021-01-13 15:01:20 +00:00
gstd3d11overlaycompositor.h d3d11: Move core methods to gst-libs 2021-01-13 15:01:20 +00:00
gstd3d11pluginutils.c d3d11: Don't use hardcoded maximum resolution value 2021-01-20 19:31:27 +09:00
gstd3d11pluginutils.h d3d11: Don't use hardcoded maximum resolution value 2021-01-20 19:31:27 +09:00
gstd3d11shader.c d3d11: Move core methods to gst-libs 2021-01-13 15:01:20 +00:00
gstd3d11shader.h d3d11: Move core methods to gst-libs 2021-01-13 15:01:20 +00:00
gstd3d11upload.c d3d11: Don't use hardcoded maximum resolution value 2021-01-20 19:31:27 +09:00
gstd3d11upload.h d3d11: Move to G_DECLARE_FINAL_TYPE 2020-10-04 15:44:03 +00:00
gstd3d11videoprocessor.c d3d11: Move core methods to gst-libs 2021-01-13 15:01:20 +00:00
gstd3d11videoprocessor.h d3d11: Move core methods to gst-libs 2021-01-13 15:01:20 +00:00
gstd3d11videosink.c d3d11videosink: Add support for drawing on application's own texture 2021-01-26 18:14:37 +00:00
gstd3d11videosink.h d3d11videosink: Add support for drawing on application's own texture 2021-01-26 18:14:37 +00:00
gstd3d11videosinkbin.c d3d11videosink: Add support for drawing on application's own texture 2021-01-26 18:14:37 +00:00
gstd3d11videosinkbin.h d3d11videosink: Add support for drawing on application's own texture 2021-01-26 18:14:37 +00:00
gstd3d11vp8dec.c d3d11: Move core methods to gst-libs 2021-01-13 15:01:20 +00:00
gstd3d11vp8dec.h d3d11decoder: Add VP8 decoder element 2020-04-22 23:20:44 +00:00
gstd3d11vp9dec.c d3d11: Move core methods to gst-libs 2021-01-13 15:01:20 +00:00
gstd3d11vp9dec.h d3d11decoder: Register elements per GPU device with capability check 2020-02-18 11:58:45 +00:00
gstd3d11window.cpp d3d11videosink: Add support for drawing on application's own texture 2021-01-26 18:14:37 +00:00
gstd3d11window.h d3d11videosink: Add support for drawing on application's own texture 2021-01-26 18:14:37 +00:00
gstd3d11window_corewindow.cpp d3d11videosink: Fix MSVC build warnings around UWP code 2021-01-19 11:23:56 +00:00
gstd3d11window_corewindow.h d3d11: Move core methods to gst-libs 2021-01-13 15:01:20 +00:00
gstd3d11window_dummy.cpp d3d11videosink: Add support for drawing on application's own texture 2021-01-26 18:14:37 +00:00
gstd3d11window_dummy.h d3d11videosink: Add support for drawing on application's own texture 2021-01-26 18:14:37 +00:00
gstd3d11window_swapchainpanel.cpp d3d11videosink: Fix MSVC build warnings around UWP code 2021-01-19 11:23:56 +00:00
gstd3d11window_swapchainpanel.h d3d11: Move core methods to gst-libs 2021-01-13 15:01:20 +00:00
gstd3d11window_win32.cpp d3d11videosink: Fix ugly thread name for Win32 window impl. 2021-01-19 11:23:56 +00:00
gstd3d11window_win32.h d3d11: Move core methods to gst-libs 2021-01-13 15:01:20 +00:00
meson.build d3d11videosink: Add support for drawing on application's own texture 2021-01-26 18:14:37 +00:00
plugin.c d3d11: Add support for MPEG-2 video decoding 2021-01-25 18:05:53 +00:00