mirrors/gstreamer
mirrors/gstreamer
1
1
Fork 0
mirror of https://gitlab.freedesktop.org/gstreamer/gstreamer.git synced 2024-12-23 16:50:47 +00:00
Code Issues 1 Projects Releases Wiki Activity
gstreamer/sys/d3d11/gstd3d11window.h

228 lines
8.5 KiB
C
Raw Normal View History

d3d11videosink: Add new Direct3D11 video render plugin Direct3D11 was shipped as part of Windows7 and it's obviously primary graphics API on Windows. This plugin includes HDR10 rendering if following requirements are satisfied * IDXGISwapChain4::SetHDRMetaData is available (decleared in dxgi1_5.h) * Display can support DXGI_COLOR_SPACE_RGB_FULL_G2084_NONE_P2020 color space * Upstream provides 10 bitdepth format with smpte-st 2084 static metadata
2019-01-30 11:07:29 +00:00
/*
* GStreamer
* Copyright (C) 2012 Matthew Waters <ystreet00@gmail.com>
* Copyright (C) 2019 Seungha Yang <seungha.yang@navercorp.com>
*
* This library is free software; you can redistribute it and/or
* modify it under the terms of the GNU Library General Public
* License as published by the Free Software Foundation; either
* version 2 of the License, or (at your option) any later version.
*
* This library is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
* Library General Public License for more details.
*
* You should have received a copy of the GNU Library General Public
* License along with this library; if not, write to the
* Free Software Foundation, Inc., 51 Franklin St, Fifth Floor,
* Boston, MA 02110-1301, USA.
*/
#ifndef __GST_D3D11_WINDOW_H__
#define __GST_D3D11_WINDOW_H__
#include <gst/gst.h>
#include <gst/video/video.h>
d3d11: Move core methods to gst-libs Move d3d11 device, memory, buffer pool and minimal method to gst-libs so that other plugins can access d3d11 resource. Since Direct3D is primary graphics API on Windows, we need this infrastructure for various plugins can share GPU resource without downloading GPU memory. Note that this implementation is public only for -bad scope for now. Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/464>
2020-12-20 17:47:45 +00:00
#include <gst/d3d11/gstd3d11.h>
d3d11: Port to C++ Direct3D11 objects are COM, and most COM C APIs are verbose (C++ is a little better). So, by using C++ APIs, we can make code shorter and more readable. Moreover, "ComPtr" helper class (which is C++ only) can be utilized, that is very helpful for avoiding error-prone COM refcounting issue/leak. Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/2077>
2021-03-13 08:40:57 +00:00
#include "gstd3d11converter.h"
d3d11videosink: Add support for overlay composition Add d3d11overlaycompositor object to draw overlay image on render target using Blend method.
2019-12-24 06:54:57 +00:00
#include "gstd3d11overlaycompositor.h"
d3d11videosink: Use ID3D11VideoProcessor interface ...for color space conversion if available ID3D11VideoProcessor is equivalent to DXVA-HD video processor which might use specialized blocks for video processing instead of general GPU resource. In addition to that feature, we need to use this API for color space conversion of DXVA2 decoder output memory, because any d3d11 texture arrays that were created with D3D11_BIND_DECODER cannot be used for shader resource. This is prework for d3d11decoder zero-copy rendering and also for conditional HDR tone-map support. Note that some Intel platform is known to support tone-mapping at the driver level using this API on Windows 10.
2020-01-29 12:10:00 +00:00
#include "gstd3d11videoprocessor.h"
d3d11: Move core methods to gst-libs Move d3d11 device, memory, buffer pool and minimal method to gst-libs so that other plugins can access d3d11 resource. Since Direct3D is primary graphics API on Windows, we need this infrastructure for various plugins can share GPU resource without downloading GPU memory. Note that this implementation is public only for -bad scope for now. Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/464>
2020-12-20 17:47:45 +00:00
#include "gstd3d11pluginutils.h"
d3d11videosink: Add new Direct3D11 video render plugin Direct3D11 was shipped as part of Windows7 and it's obviously primary graphics API on Windows. This plugin includes HDR10 rendering if following requirements are satisfied * IDXGISwapChain4::SetHDRMetaData is available (decleared in dxgi1_5.h) * Display can support DXGI_COLOR_SPACE_RGB_FULL_G2084_NONE_P2020 color space * Upstream provides 10 bitdepth format with smpte-st 2084 static metadata
2019-01-30 11:07:29 +00:00
G_BEGIN_DECLS
#define GST_TYPE_D3D11_WINDOW (gst_d3d11_window_get_type())
d3d11: Add support for Universal Windows Platform Initial UWP support via new window (CoreWindow and SwapChainPanel) implementation.
2019-12-30 09:58:59 +00:00
#define GST_D3D11_WINDOW(obj) (G_TYPE_CHECK_INSTANCE_CAST((obj),GST_TYPE_D3D11_WINDOW, GstD3D11Window))
#define GST_D3D11_WINDOW_CLASS(klass) (G_TYPE_CHECK_CLASS_CAST((klass), GST_TYPE_D3D11_WINDOW, GstD3D11WindowClass))
#define GST_IS_D3D11_WINDOW(obj) (G_TYPE_CHECK_INSTANCE_TYPE((obj),GST_TYPE_D3D11_WINDOW))
d3d11videosink: Add new Direct3D11 video render plugin Direct3D11 was shipped as part of Windows7 and it's obviously primary graphics API on Windows. This plugin includes HDR10 rendering if following requirements are satisfied * IDXGISwapChain4::SetHDRMetaData is available (decleared in dxgi1_5.h) * Display can support DXGI_COLOR_SPACE_RGB_FULL_G2084_NONE_P2020 color space * Upstream provides 10 bitdepth format with smpte-st 2084 static metadata
2019-01-30 11:07:29 +00:00
#define GST_IS_D3D11_WINDOW_CLASS(klass) (G_TYPE_CHECK_CLASS_TYPE((klass), GST_TYPE_D3D11_WINDOW))
#define GST_D3D11_WINDOW_GET_CLASS(obj) (G_TYPE_INSTANCE_GET_CLASS((obj), GST_TYPE_D3D11_WINDOW, GstD3D11WindowClass))
d3d11videosink: Add support for full screen mode borderless top-most style full screen mode support. Basically fullscreen toggle mode is disabled by default. To enable it use "fullscreen-toggle-mode" property to allow fullscreen mode change by user input and/or property.
2019-12-15 10:29:10 +00:00
#define GST_D3D11_WINDOW_TOGGLE_MODE_GET_TYPE (gst_d3d11_window_fullscreen_toggle_mode_type())
d3d11videosink: Add new Direct3D11 video render plugin Direct3D11 was shipped as part of Windows7 and it's obviously primary graphics API on Windows. This plugin includes HDR10 rendering if following requirements are satisfied * IDXGISwapChain4::SetHDRMetaData is available (decleared in dxgi1_5.h) * Display can support DXGI_COLOR_SPACE_RGB_FULL_G2084_NONE_P2020 color space * Upstream provides 10 bitdepth format with smpte-st 2084 static metadata
2019-01-30 11:07:29 +00:00
typedef struct _GstD3D11Window GstD3D11Window;
typedef struct _GstD3D11WindowClass GstD3D11WindowClass;
d3d11videosink: Add a property to support rendering statistics data on window Add a new property "render-stats" to allow rendering statistics data on window for debugging and/or development purpose. Text rendering will be accelerated by GPU since this implementation uses Direct2D/DirectWrite API and Direct3D inter-op for minimal overhead. Specifically, text data will be rendered on swapchain backbuffer directly without any copy/allocation of extra texture. Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1830>
2020-11-21 19:39:57 +00:00
typedef struct _GstD3D11WindowPrivate GstD3D11WindowPrivate;
d3d11videosink: Add new Direct3D11 video render plugin Direct3D11 was shipped as part of Windows7 and it's obviously primary graphics API on Windows. This plugin includes HDR10 rendering if following requirements are satisfied * IDXGISwapChain4::SetHDRMetaData is available (decleared in dxgi1_5.h) * Display can support DXGI_COLOR_SPACE_RGB_FULL_G2084_NONE_P2020 color space * Upstream provides 10 bitdepth format with smpte-st 2084 static metadata
2019-01-30 11:07:29 +00:00
#define GST_D3D11_WINDOW_FLOW_CLOSED GST_FLOW_CUSTOM_ERROR
d3d11videosink: Add support for full screen mode borderless top-most style full screen mode support. Basically fullscreen toggle mode is disabled by default. To enable it use "fullscreen-toggle-mode" property to allow fullscreen mode change by user input and/or property.
2019-12-15 10:29:10 +00:00
typedef enum
{
GST_D3D11_WINDOW_FULLSCREEN_TOGGLE_MODE_NONE = 0,
GST_D3D11_WINDOW_FULLSCREEN_TOGGLE_MODE_ALT_ENTER = (1 << 1),
GST_D3D11_WINDOW_FULLSCREEN_TOGGLE_MODE_PROPERTY = (1 << 2),
} GstD3D11WindowFullscreenToggleMode;
GType gst_d3d11_window_fullscreen_toggle_mode_type (void);
d3d11: Add support for Universal Windows Platform Initial UWP support via new window (CoreWindow and SwapChainPanel) implementation.
2019-12-30 09:58:59 +00:00
typedef enum
{
GST_D3D11_WINDOW_NATIVE_TYPE_NONE = 0,
GST_D3D11_WINDOW_NATIVE_TYPE_HWND,
GST_D3D11_WINDOW_NATIVE_TYPE_CORE_WINDOW,
GST_D3D11_WINDOW_NATIVE_TYPE_SWAP_CHAIN_PANEL,
} GstD3D11WindowNativeType;
d3d11videosink: Add support for drawing on application's own texture Add a way to support drawing on application's texture instead of usual window handle. To make use of this new feature, application should follow below step. 1) Enable this feature by using "draw-on-shared-texture" property 2) Watch "begin-draw" signal 3) On "begin-draw" signal handler, application can request drawing by using "draw" signal action. Note that "draw" signal action should be happen before "begin-draw" signal handler is returned NOTE 1) For texture sharing, creating a texture with D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX flag is strongly recommend if possible because we cannot ensure sync a texture which was created with D3D11_RESOURCE_MISC_SHARED and it would cause glitch with ID3D11VideoProcessor use case. NOTE 2) Direct9Ex doesn't support texture sharing which was created with D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX. In other words, D3D11_RESOURCE_MISC_SHARED is the only option for Direct3D11/Direct9Ex interop. NOTE 3) Because of missing synchronization around ID3D11VideoProcessor, If shared texture was created with D3D11_RESOURCE_MISC_SHARED, d3d11videosink might use fallback texture to convert DXVA texture to normal Direct3D texture. Then converted texture will be copied to user-provided shared texture. * Why not use generic appsink approach? In order for application to be able to store video data which was produced by GStreamer in application's own texture, there would be two possible approaches, one is copying our texture into application's own texture, and the other is drawing on application's own texture directly. The former (appsink way) cannot be a zero-copy by nature. In order to support zero-copy processing, we need to draw on application's own texture directly. For example, assume that application wants RGBA texture. Then we can imagine following case. "d3d11h264dec ! d3d11convert ! video/x-raw(memory:D3D11Memory),format=RGBA ! appsink" ^ |_ allocate new Direct3D texture for RGBA format In above case, d3d11convert will allocate new texture(s) for RGBA format and then application will copy again the our RGBA texutre into application's own texture. One texture allocation plus per frame GPU copy will hanppen in that case therefore. Moreover, in order for application to be able to access our texture, we need to allocate texture with additional flags for application's Direct3D11 device to be able to read texture data. That would be another implementation burden on our side But with this MR, we can configure pipeline in this way "d3d11h264dec ! d3d11videosink". In that way, we can save at least one texture allocation and per frame texutre copy since d3d11videosink will convert incoming texture into application's texture format directly without copy. * What if we expose texture without conversion and application does conversion by itself? As mentioned above, for application to be able to access our texture from application's Direct3D11 device, we need to allocate texture in a special form. But in some case, that might not be possible. Also, if a texture belongs to decoder DPB, exposing such texture to application is unsafe and usual Direct3D11 shader cannot handle such texture. To convert format, ID3D11VideoProcessor API needs to be used but that would be a implementation burden for application. Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1873>
2020-12-23 14:49:12 +00:00
typedef struct
{
HANDLE shared_handle;
guint texture_misc_flags;
guint64 acquire_key;
guint64 release_key;
ID3D11Texture2D *texture;
IDXGIKeyedMutex *keyed_mutex;
ID3D11VideoProcessorOutputView *pov;
ID3D11RenderTargetView *rtv;
ID3D11VideoProcessorOutputView *fallback_pov;
ID3D11RenderTargetView *fallback_rtv;
} GstD3D11WindowSharedHandleData;
d3d11videosink: Add new Direct3D11 video render plugin Direct3D11 was shipped as part of Windows7 and it's obviously primary graphics API on Windows. This plugin includes HDR10 rendering if following requirements are satisfied * IDXGISwapChain4::SetHDRMetaData is available (decleared in dxgi1_5.h) * Display can support DXGI_COLOR_SPACE_RGB_FULL_G2084_NONE_P2020 color space * Upstream provides 10 bitdepth format with smpte-st 2084 static metadata
2019-01-30 11:07:29 +00:00
struct _GstD3D11Window
{
GstObject parent;
d3d11: Add support for Universal Windows Platform Initial UWP support via new window (CoreWindow and SwapChainPanel) implementation.
2019-12-30 09:58:59 +00:00
/*< protected >*/
gboolean initialized;
GstD3D11Device *device;
guintptr external_handle;
/* properties */
gboolean force_aspect_ratio;
gboolean enable_navigation_events;
GstD3D11WindowFullscreenToggleMode fullscreen_toggle_mode;
gboolean requested_fullscreen;
gboolean fullscreen;
d3d11videosink: Add a property to support rendering statistics data on window Add a new property "render-stats" to allow rendering statistics data on window for debugging and/or development purpose. Text rendering will be accelerated by GPU since this implementation uses Direct2D/DirectWrite API and Direct3D inter-op for minimal overhead. Specifically, text data will be rendered on swapchain backbuffer directly without any copy/allocation of extra texture. Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1830>
2020-11-21 19:39:57 +00:00
gboolean render_stats;
d3d11: Add support for Universal Windows Platform Initial UWP support via new window (CoreWindow and SwapChainPanel) implementation.
2019-12-30 09:58:59 +00:00
d3d11videosink: Add new Direct3D11 video render plugin Direct3D11 was shipped as part of Windows7 and it's obviously primary graphics API on Windows. This plugin includes HDR10 rendering if following requirements are satisfied * IDXGISwapChain4::SetHDRMetaData is available (decleared in dxgi1_5.h) * Display can support DXGI_COLOR_SPACE_RGB_FULL_G2084_NONE_P2020 color space * Upstream provides 10 bitdepth format with smpte-st 2084 static metadata
2019-01-30 11:07:29 +00:00
GstVideoInfo info;
d3d11videosink: Add color conversion support Draw to back buffer texture directly. It would reduce the number of copy at least once when color conversion is required.
2019-12-03 13:54:26 +00:00
GstVideoInfo render_info;
d3d11videosink: Use ID3D11VideoProcessor interface ...for color space conversion if available ID3D11VideoProcessor is equivalent to DXVA-HD video processor which might use specialized blocks for video processing instead of general GPU resource. In addition to that feature, we need to use this API for color space conversion of DXVA2 decoder output memory, because any d3d11 texture arrays that were created with D3D11_BIND_DECODER cannot be used for shader resource. This is prework for d3d11decoder zero-copy rendering and also for conditional HDR tone-map support. Note that some Intel platform is known to support tone-mapping at the driver level using this API on Windows 10.
2020-01-29 12:10:00 +00:00
GstD3D11VideoProcessor *processor;
d3d11: Port to C++ Direct3D11 objects are COM, and most COM C APIs are verbose (C++ is a little better). So, by using C++ APIs, we can make code shorter and more readable. Moreover, "ComPtr" helper class (which is C++ only) can be utilized, that is very helpful for avoiding error-prone COM refcounting issue/leak. Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/2077>
2021-03-13 08:40:57 +00:00
GstD3D11Converter *converter;
d3d11videosink: Add support for overlay composition Add d3d11overlaycompositor object to draw overlay image on render target using Blend method.
2019-12-24 06:54:57 +00:00
GstD3D11OverlayCompositor *compositor;
d3d11videosink: Add color conversion support Draw to back buffer texture directly. It would reduce the number of copy at least once when color conversion is required.
2019-12-03 13:54:26 +00:00
d3d11videosink: Avoid switching conversion tool during playback Decoder might be able to copy decoded texture to the other buffer pool during playback depending on context. In that case, copied one has no D3D11_BIND_DECODER bind flag. If we used ID3D11VideoProcessor previously for decoder texture, and incoming texture supports ID3D11VideoProcessor as well even if it has no D3D11_BIND_DECODER flag (having D3D11_BIND_RENDER_TARGET for example), allow zero-copying instead of using our fallback texture. Frequent conversion tool change (between ID3D11VideoProcessor and generic shader) might result in inconsistent image quality. Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/2084>
2021-03-15 10:48:56 +00:00
gboolean processor_in_use;
d3d11videosink: Draw window with cached texture on resize This would render nicer than presenting scene with broken aspect ratio, especially in case of low framerate.
2019-12-02 14:27:42 +00:00
/* calculated rect with aspect ratio and window area */
d3d11window: Make use of partial presentation with IDXGISwapChain1::Present1 Since we might draw on partial area of backbuffer in case of force-aspect-ratio, presenting only updated area is more efficient way. See also https://docs.microsoft.com/ko-kr/windows/win32/direct3ddxgi/dxgi-1-2-presentation-improvements
2019-12-24 05:00:15 +00:00
RECT render_rect;
d3d11videosink: Add new Direct3D11 video render plugin Direct3D11 was shipped as part of Windows7 and it's obviously primary graphics API on Windows. This plugin includes HDR10 rendering if following requirements are satisfied * IDXGISwapChain4::SetHDRMetaData is available (decleared in dxgi1_5.h) * Display can support DXGI_COLOR_SPACE_RGB_FULL_G2084_NONE_P2020 color space * Upstream provides 10 bitdepth format with smpte-st 2084 static metadata
2019-01-30 11:07:29 +00:00
d3d11videosink: Use ID3D11VideoProcessor interface ...for color space conversion if available ID3D11VideoProcessor is equivalent to DXVA-HD video processor which might use specialized blocks for video processing instead of general GPU resource. In addition to that feature, we need to use this API for color space conversion of DXVA2 decoder output memory, because any d3d11 texture arrays that were created with D3D11_BIND_DECODER cannot be used for shader resource. This is prework for d3d11decoder zero-copy rendering and also for conditional HDR tone-map support. Note that some Intel platform is known to support tone-mapping at the driver level using this API on Windows 10.
2020-01-29 12:10:00 +00:00
/* input resolution */
RECT input_rect;
d3d11videosink: Draw window with cached texture on resize This would render nicer than presenting scene with broken aspect ratio, especially in case of low framerate.
2019-12-02 14:27:42 +00:00
/* requested rect via gst_d3d11_window_render */
GstVideoRectangle rect;
d3d11videosink: Add new Direct3D11 video render plugin Direct3D11 was shipped as part of Windows7 and it's obviously primary graphics API on Windows. This plugin includes HDR10 rendering if following requirements are satisfied * IDXGISwapChain4::SetHDRMetaData is available (decleared in dxgi1_5.h) * Display can support DXGI_COLOR_SPACE_RGB_FULL_G2084_NONE_P2020 color space * Upstream provides 10 bitdepth format with smpte-st 2084 static metadata
2019-01-30 11:07:29 +00:00
guint surface_width;
guint surface_height;
IDXGISwapChain *swap_chain;
ID3D11RenderTargetView *rtv;
d3d11videosink: Use ID3D11VideoProcessor interface ...for color space conversion if available ID3D11VideoProcessor is equivalent to DXVA-HD video processor which might use specialized blocks for video processing instead of general GPU resource. In addition to that feature, we need to use this API for color space conversion of DXVA2 decoder output memory, because any d3d11 texture arrays that were created with D3D11_BIND_DECODER cannot be used for shader resource. This is prework for d3d11decoder zero-copy rendering and also for conditional HDR tone-map support. Note that some Intel platform is known to support tone-mapping at the driver level using this API on Windows 10.
2020-01-29 12:10:00 +00:00
ID3D11VideoProcessorOutputView *pov;
d3d11window: Don't create swapchain again per caps change Creating swapchain is relatively heavy operation. If output dxgi format is not being chagned, we don't need to destroy and create swachain again.
2020-01-30 12:12:31 +00:00
DXGI_FORMAT dxgi_format;
d3d11videosink: Draw window with cached texture on resize This would render nicer than presenting scene with broken aspect ratio, especially in case of low framerate.
2019-12-02 14:27:42 +00:00
GstBuffer *cached_buffer;
d3d11: Add support for Universal Windows Platform Initial UWP support via new window (CoreWindow and SwapChainPanel) implementation.
2019-12-30 09:58:59 +00:00
gboolean first_present;
d3d11window: Use allow-tearing mode if device supports it As the recommendation from MS.
2019-12-17 04:47:24 +00:00
gboolean allow_tearing;
d3d11videosink: Add a property to support rendering statistics data on window Add a new property "render-stats" to allow rendering statistics data on window for debugging and/or development purpose. Text rendering will be accelerated by GPU since this implementation uses Direct2D/DirectWrite API and Direct3D inter-op for minimal overhead. Specifically, text data will be rendered on swapchain backbuffer directly without any copy/allocation of extra texture. Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1830>
2020-11-21 19:39:57 +00:00
GstD3D11WindowPrivate *priv;
d3d11videosink: Add new Direct3D11 video render plugin Direct3D11 was shipped as part of Windows7 and it's obviously primary graphics API on Windows. This plugin includes HDR10 rendering if following requirements are satisfied * IDXGISwapChain4::SetHDRMetaData is available (decleared in dxgi1_5.h) * Display can support DXGI_COLOR_SPACE_RGB_FULL_G2084_NONE_P2020 color space * Upstream provides 10 bitdepth format with smpte-st 2084 static metadata
2019-01-30 11:07:29 +00:00
};
struct _GstD3D11WindowClass
{
GstObjectClass object_class;
d3d11: Add support for Universal Windows Platform Initial UWP support via new window (CoreWindow and SwapChainPanel) implementation.
2019-12-30 09:58:59 +00:00
void (*show) (GstD3D11Window * window);
void (*update_swap_chain) (GstD3D11Window * window);
void (*change_fullscreen_mode) (GstD3D11Window * window);
gboolean (*create_swap_chain) (GstD3D11Window * window,
DXGI_FORMAT format,
guint width,
guint height,
guint swapchain_flags,
IDXGISwapChain ** swap_chain);
GstFlowReturn (*present) (GstD3D11Window * window,
guint present_flags);
gboolean (*unlock) (GstD3D11Window * window);
gboolean (*unlock_stop) (GstD3D11Window * window);
d3d11window: Invoke initial resize method from baseclass ... instead of calling from subclass in order for baseclass to handle more things between swapchain creation and resource creation.
2020-01-10 12:45:43 +00:00
void (*on_resize) (GstD3D11Window * window,
guint width,
guint height);
d3d11window: Add unprepare method to clear internal resource GObject::dispose method can be called multiple times. As win32 d3d11window has an internal thread and because GObject::dispose method could be called from the thread, it might cause problems such as trying to join self-thread Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1299>
2020-05-26 16:52:59 +00:00
d3d11videosink: Add support for drawing on application's own texture Add a way to support drawing on application's texture instead of usual window handle. To make use of this new feature, application should follow below step. 1) Enable this feature by using "draw-on-shared-texture" property 2) Watch "begin-draw" signal 3) On "begin-draw" signal handler, application can request drawing by using "draw" signal action. Note that "draw" signal action should be happen before "begin-draw" signal handler is returned NOTE 1) For texture sharing, creating a texture with D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX flag is strongly recommend if possible because we cannot ensure sync a texture which was created with D3D11_RESOURCE_MISC_SHARED and it would cause glitch with ID3D11VideoProcessor use case. NOTE 2) Direct9Ex doesn't support texture sharing which was created with D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX. In other words, D3D11_RESOURCE_MISC_SHARED is the only option for Direct3D11/Direct9Ex interop. NOTE 3) Because of missing synchronization around ID3D11VideoProcessor, If shared texture was created with D3D11_RESOURCE_MISC_SHARED, d3d11videosink might use fallback texture to convert DXVA texture to normal Direct3D texture. Then converted texture will be copied to user-provided shared texture. * Why not use generic appsink approach? In order for application to be able to store video data which was produced by GStreamer in application's own texture, there would be two possible approaches, one is copying our texture into application's own texture, and the other is drawing on application's own texture directly. The former (appsink way) cannot be a zero-copy by nature. In order to support zero-copy processing, we need to draw on application's own texture directly. For example, assume that application wants RGBA texture. Then we can imagine following case. "d3d11h264dec ! d3d11convert ! video/x-raw(memory:D3D11Memory),format=RGBA ! appsink" ^ |_ allocate new Direct3D texture for RGBA format In above case, d3d11convert will allocate new texture(s) for RGBA format and then application will copy again the our RGBA texutre into application's own texture. One texture allocation plus per frame GPU copy will hanppen in that case therefore. Moreover, in order for application to be able to access our texture, we need to allocate texture with additional flags for application's Direct3D11 device to be able to read texture data. That would be another implementation burden on our side But with this MR, we can configure pipeline in this way "d3d11h264dec ! d3d11videosink". In that way, we can save at least one texture allocation and per frame texutre copy since d3d11videosink will convert incoming texture into application's texture format directly without copy. * What if we expose texture without conversion and application does conversion by itself? As mentioned above, for application to be able to access our texture from application's Direct3D11 device, we need to allocate texture in a special form. But in some case, that might not be possible. Also, if a texture belongs to decoder DPB, exposing such texture to application is unsafe and usual Direct3D11 shader cannot handle such texture. To convert format, ID3D11VideoProcessor API needs to be used but that would be a implementation burden for application. Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1873>
2020-12-23 14:49:12 +00:00
gboolean (*prepare) (GstD3D11Window * window,
guint display_width,
guint display_height,
GstCaps * caps,
gboolean * video_processor_available,
GError ** error);
d3d11window: Add unprepare method to clear internal resource GObject::dispose method can be called multiple times. As win32 d3d11window has an internal thread and because GObject::dispose method could be called from the thread, it might cause problems such as trying to join self-thread Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1299>
2020-05-26 16:52:59 +00:00
void (*unprepare) (GstD3D11Window * window);
d3d11videosink: Add support for drawing on application's own texture Add a way to support drawing on application's texture instead of usual window handle. To make use of this new feature, application should follow below step. 1) Enable this feature by using "draw-on-shared-texture" property 2) Watch "begin-draw" signal 3) On "begin-draw" signal handler, application can request drawing by using "draw" signal action. Note that "draw" signal action should be happen before "begin-draw" signal handler is returned NOTE 1) For texture sharing, creating a texture with D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX flag is strongly recommend if possible because we cannot ensure sync a texture which was created with D3D11_RESOURCE_MISC_SHARED and it would cause glitch with ID3D11VideoProcessor use case. NOTE 2) Direct9Ex doesn't support texture sharing which was created with D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX. In other words, D3D11_RESOURCE_MISC_SHARED is the only option for Direct3D11/Direct9Ex interop. NOTE 3) Because of missing synchronization around ID3D11VideoProcessor, If shared texture was created with D3D11_RESOURCE_MISC_SHARED, d3d11videosink might use fallback texture to convert DXVA texture to normal Direct3D texture. Then converted texture will be copied to user-provided shared texture. * Why not use generic appsink approach? In order for application to be able to store video data which was produced by GStreamer in application's own texture, there would be two possible approaches, one is copying our texture into application's own texture, and the other is drawing on application's own texture directly. The former (appsink way) cannot be a zero-copy by nature. In order to support zero-copy processing, we need to draw on application's own texture directly. For example, assume that application wants RGBA texture. Then we can imagine following case. "d3d11h264dec ! d3d11convert ! video/x-raw(memory:D3D11Memory),format=RGBA ! appsink" ^ |_ allocate new Direct3D texture for RGBA format In above case, d3d11convert will allocate new texture(s) for RGBA format and then application will copy again the our RGBA texutre into application's own texture. One texture allocation plus per frame GPU copy will hanppen in that case therefore. Moreover, in order for application to be able to access our texture, we need to allocate texture with additional flags for application's Direct3D11 device to be able to read texture data. That would be another implementation burden on our side But with this MR, we can configure pipeline in this way "d3d11h264dec ! d3d11videosink". In that way, we can save at least one texture allocation and per frame texutre copy since d3d11videosink will convert incoming texture into application's texture format directly without copy. * What if we expose texture without conversion and application does conversion by itself? As mentioned above, for application to be able to access our texture from application's Direct3D11 device, we need to allocate texture in a special form. But in some case, that might not be possible. Also, if a texture belongs to decoder DPB, exposing such texture to application is unsafe and usual Direct3D11 shader cannot handle such texture. To convert format, ID3D11VideoProcessor API needs to be used but that would be a implementation burden for application. Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1873>
2020-12-23 14:49:12 +00:00
gboolean (*open_shared_handle) (GstD3D11Window * window,
GstD3D11WindowSharedHandleData * data);
gboolean (*release_shared_handle) (GstD3D11Window * window,
GstD3D11WindowSharedHandleData * data);
d3d11videosink: Add new Direct3D11 video render plugin Direct3D11 was shipped as part of Windows7 and it's obviously primary graphics API on Windows. This plugin includes HDR10 rendering if following requirements are satisfied * IDXGISwapChain4::SetHDRMetaData is available (decleared in dxgi1_5.h) * Display can support DXGI_COLOR_SPACE_RGB_FULL_G2084_NONE_P2020 color space * Upstream provides 10 bitdepth format with smpte-st 2084 static metadata
2019-01-30 11:07:29 +00:00
};
d3d11: Add support for Universal Windows Platform Initial UWP support via new window (CoreWindow and SwapChainPanel) implementation.
2019-12-30 09:58:59 +00:00
GType gst_d3d11_window_get_type (void);
void gst_d3d11_window_show (GstD3D11Window * window);
void gst_d3d11_window_set_render_rectangle (GstD3D11Window * window,
gint x, gint y,
gint width, gint height);
gboolean gst_d3d11_window_prepare (GstD3D11Window * window,
d3d11videosink: Clarify the meaning of various width and height variables * Remove redundant variables for width/height and par from GstD3D11Window. GstVideoInfo holds all the values. * Don't need to pass par to gst_d3d11_window_prepare(). It will be parsed from caps again * Remove duplicated math Fixing regression of the commit 9dada901083fec3276cb2df58785db14b87e8f22
2020-03-02 11:55:29 +00:00
guint display_width,
guint display_height,
d3d11: Add support for Universal Windows Platform Initial UWP support via new window (CoreWindow and SwapChainPanel) implementation.
2019-12-30 09:58:59 +00:00
GstCaps * caps,
d3d11videosink: Use ID3D11VideoProcessor interface ...for color space conversion if available ID3D11VideoProcessor is equivalent to DXVA-HD video processor which might use specialized blocks for video processing instead of general GPU resource. In addition to that feature, we need to use this API for color space conversion of DXVA2 decoder output memory, because any d3d11 texture arrays that were created with D3D11_BIND_DECODER cannot be used for shader resource. This is prework for d3d11decoder zero-copy rendering and also for conditional HDR tone-map support. Note that some Intel platform is known to support tone-mapping at the driver level using this API on Windows 10.
2020-01-29 12:10:00 +00:00
gboolean * video_processor_available,
d3d11: Add support for Universal Windows Platform Initial UWP support via new window (CoreWindow and SwapChainPanel) implementation.
2019-12-30 09:58:59 +00:00
GError ** error);
GstFlowReturn gst_d3d11_window_render (GstD3D11Window * window,
GstBuffer * buffer,
d3d11videosink: Add a property to support rendering statistics data on window Add a new property "render-stats" to allow rendering statistics data on window for debugging and/or development purpose. Text rendering will be accelerated by GPU since this implementation uses Direct2D/DirectWrite API and Direct3D inter-op for minimal overhead. Specifically, text data will be rendered on swapchain backbuffer directly without any copy/allocation of extra texture. Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1830>
2020-11-21 19:39:57 +00:00
GstVideoRectangle * src_rect,
GstStructure * stats);
d3d11: Add support for Universal Windows Platform Initial UWP support via new window (CoreWindow and SwapChainPanel) implementation.
2019-12-30 09:58:59 +00:00
d3d11videosink: Add support for drawing on application's own texture Add a way to support drawing on application's texture instead of usual window handle. To make use of this new feature, application should follow below step. 1) Enable this feature by using "draw-on-shared-texture" property 2) Watch "begin-draw" signal 3) On "begin-draw" signal handler, application can request drawing by using "draw" signal action. Note that "draw" signal action should be happen before "begin-draw" signal handler is returned NOTE 1) For texture sharing, creating a texture with D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX flag is strongly recommend if possible because we cannot ensure sync a texture which was created with D3D11_RESOURCE_MISC_SHARED and it would cause glitch with ID3D11VideoProcessor use case. NOTE 2) Direct9Ex doesn't support texture sharing which was created with D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX. In other words, D3D11_RESOURCE_MISC_SHARED is the only option for Direct3D11/Direct9Ex interop. NOTE 3) Because of missing synchronization around ID3D11VideoProcessor, If shared texture was created with D3D11_RESOURCE_MISC_SHARED, d3d11videosink might use fallback texture to convert DXVA texture to normal Direct3D texture. Then converted texture will be copied to user-provided shared texture. * Why not use generic appsink approach? In order for application to be able to store video data which was produced by GStreamer in application's own texture, there would be two possible approaches, one is copying our texture into application's own texture, and the other is drawing on application's own texture directly. The former (appsink way) cannot be a zero-copy by nature. In order to support zero-copy processing, we need to draw on application's own texture directly. For example, assume that application wants RGBA texture. Then we can imagine following case. "d3d11h264dec ! d3d11convert ! video/x-raw(memory:D3D11Memory),format=RGBA ! appsink" ^ |_ allocate new Direct3D texture for RGBA format In above case, d3d11convert will allocate new texture(s) for RGBA format and then application will copy again the our RGBA texutre into application's own texture. One texture allocation plus per frame GPU copy will hanppen in that case therefore. Moreover, in order for application to be able to access our texture, we need to allocate texture with additional flags for application's Direct3D11 device to be able to read texture data. That would be another implementation burden on our side But with this MR, we can configure pipeline in this way "d3d11h264dec ! d3d11videosink". In that way, we can save at least one texture allocation and per frame texutre copy since d3d11videosink will convert incoming texture into application's texture format directly without copy. * What if we expose texture without conversion and application does conversion by itself? As mentioned above, for application to be able to access our texture from application's Direct3D11 device, we need to allocate texture in a special form. But in some case, that might not be possible. Also, if a texture belongs to decoder DPB, exposing such texture to application is unsafe and usual Direct3D11 shader cannot handle such texture. To convert format, ID3D11VideoProcessor API needs to be used but that would be a implementation burden for application. Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1873>
2020-12-23 14:49:12 +00:00
GstFlowReturn gst_d3d11_window_render_on_shared_handle (GstD3D11Window * window,
GstBuffer * buffer,
HANDLE shared_handle,
guint texture_misc_flags,
guint64 acquire_key,
guint64 release_key);
d3d11: Add support for Universal Windows Platform Initial UWP support via new window (CoreWindow and SwapChainPanel) implementation.
2019-12-30 09:58:59 +00:00
gboolean gst_d3d11_window_unlock (GstD3D11Window * window);
d3d11videosink: Add new Direct3D11 video render plugin Direct3D11 was shipped as part of Windows7 and it's obviously primary graphics API on Windows. This plugin includes HDR10 rendering if following requirements are satisfied * IDXGISwapChain4::SetHDRMetaData is available (decleared in dxgi1_5.h) * Display can support DXGI_COLOR_SPACE_RGB_FULL_G2084_NONE_P2020 color space * Upstream provides 10 bitdepth format with smpte-st 2084 static metadata
2019-01-30 11:07:29 +00:00
d3d11: Add support for Universal Windows Platform Initial UWP support via new window (CoreWindow and SwapChainPanel) implementation.
2019-12-30 09:58:59 +00:00
gboolean gst_d3d11_window_unlock_stop (GstD3D11Window * window);
d3d11videosink: Add new Direct3D11 video render plugin Direct3D11 was shipped as part of Windows7 and it's obviously primary graphics API on Windows. This plugin includes HDR10 rendering if following requirements are satisfied * IDXGISwapChain4::SetHDRMetaData is available (decleared in dxgi1_5.h) * Display can support DXGI_COLOR_SPACE_RGB_FULL_G2084_NONE_P2020 color space * Upstream provides 10 bitdepth format with smpte-st 2084 static metadata
2019-01-30 11:07:29 +00:00
d3d11window: Add unprepare method to clear internal resource GObject::dispose method can be called multiple times. As win32 d3d11window has an internal thread and because GObject::dispose method could be called from the thread, it might cause problems such as trying to join self-thread Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1299>
2020-05-26 16:52:59 +00:00
void gst_d3d11_window_unprepare (GstD3D11Window * window);
d3d11: Add support for Universal Windows Platform Initial UWP support via new window (CoreWindow and SwapChainPanel) implementation.
2019-12-30 09:58:59 +00:00
void gst_d3d11_window_on_key_event (GstD3D11Window * window,
const gchar * event,
const gchar * key);
d3d11videosink: Add new Direct3D11 video render plugin Direct3D11 was shipped as part of Windows7 and it's obviously primary graphics API on Windows. This plugin includes HDR10 rendering if following requirements are satisfied * IDXGISwapChain4::SetHDRMetaData is available (decleared in dxgi1_5.h) * Display can support DXGI_COLOR_SPACE_RGB_FULL_G2084_NONE_P2020 color space * Upstream provides 10 bitdepth format with smpte-st 2084 static metadata
2019-01-30 11:07:29 +00:00
d3d11: Add support for Universal Windows Platform Initial UWP support via new window (CoreWindow and SwapChainPanel) implementation.
2019-12-30 09:58:59 +00:00
void gst_d3d11_window_on_mouse_event (GstD3D11Window * window,
const gchar * event,
gint button,
gdouble x,
gdouble y);
d3d11videosink: Add new Direct3D11 video render plugin Direct3D11 was shipped as part of Windows7 and it's obviously primary graphics API on Windows. This plugin includes HDR10 rendering if following requirements are satisfied * IDXGISwapChain4::SetHDRMetaData is available (decleared in dxgi1_5.h) * Display can support DXGI_COLOR_SPACE_RGB_FULL_G2084_NONE_P2020 color space * Upstream provides 10 bitdepth format with smpte-st 2084 static metadata
2019-01-30 11:07:29 +00:00
d3d11: Add support for Universal Windows Platform Initial UWP support via new window (CoreWindow and SwapChainPanel) implementation.
2019-12-30 09:58:59 +00:00
/* utils */
GstD3D11WindowNativeType gst_d3d11_window_get_native_type_from_handle (guintptr handle);
d3d11videosink: Add new Direct3D11 video render plugin Direct3D11 was shipped as part of Windows7 and it's obviously primary graphics API on Windows. This plugin includes HDR10 rendering if following requirements are satisfied * IDXGISwapChain4::SetHDRMetaData is available (decleared in dxgi1_5.h) * Display can support DXGI_COLOR_SPACE_RGB_FULL_G2084_NONE_P2020 color space * Upstream provides 10 bitdepth format with smpte-st 2084 static metadata
2019-01-30 11:07:29 +00:00
d3d11: Add support for Universal Windows Platform Initial UWP support via new window (CoreWindow and SwapChainPanel) implementation.
2019-12-30 09:58:59 +00:00
const gchar * gst_d3d11_window_get_native_type_to_string (GstD3D11WindowNativeType type);
d3d11videosink: Add new Direct3D11 video render plugin Direct3D11 was shipped as part of Windows7 and it's obviously primary graphics API on Windows. This plugin includes HDR10 rendering if following requirements are satisfied * IDXGISwapChain4::SetHDRMetaData is available (decleared in dxgi1_5.h) * Display can support DXGI_COLOR_SPACE_RGB_FULL_G2084_NONE_P2020 color space * Upstream provides 10 bitdepth format with smpte-st 2084 static metadata
2019-01-30 11:07:29 +00:00
d3d11: Add support for Universal Windows Platform Initial UWP support via new window (CoreWindow and SwapChainPanel) implementation.
2019-12-30 09:58:59 +00:00
G_DEFINE_AUTOPTR_CLEANUP_FUNC(GstD3D11Window, gst_object_unref)
d3d11videosink: Draw window with cached texture on resize This would render nicer than presenting scene with broken aspect ratio, especially in case of low framerate.
2019-12-02 14:27:42 +00:00
d3d11videosink: Add new Direct3D11 video render plugin Direct3D11 was shipped as part of Windows7 and it's obviously primary graphics API on Windows. This plugin includes HDR10 rendering if following requirements are satisfied * IDXGISwapChain4::SetHDRMetaData is available (decleared in dxgi1_5.h) * Display can support DXGI_COLOR_SPACE_RGB_FULL_G2084_NONE_P2020 color space * Upstream provides 10 bitdepth format with smpte-st 2084 static metadata
2019-01-30 11:07:29 +00:00
G_END_DECLS
#endif /* __GST_D3D11_WINDOW_H__ */
Reference in a new issue Copy permalink