2019-01-30 11:07:29 +00:00
|
|
|
/* GStreamer
|
|
|
|
|
* Copyright (C) 2019 Seungha Yang <seungha.yang@navercorp.com>
|
d3d11videosink: Add support for drawing on application's own texture
Add a way to support drawing on application's texture instead of
usual window handle.
To make use of this new feature, application should follow below step.
1) Enable this feature by using "draw-on-shared-texture" property
2) Watch "begin-draw" signal
3) On "begin-draw" signal handler, application can request drawing
by using "draw" signal action. Note that "draw" signal action
should be happen before "begin-draw" signal handler is returned
NOTE 1) For texture sharing, creating a texture with
D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX flag is strongly recommend
if possible because we cannot ensure sync a texture
which was created with D3D11_RESOURCE_MISC_SHARED
and it would cause glitch with ID3D11VideoProcessor use case.
NOTE 2) Direct9Ex doesn't support texture sharing which was
created with D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX. In other words,
D3D11_RESOURCE_MISC_SHARED is the only option for Direct3D11/Direct9Ex interop.
NOTE 3) Because of missing synchronization around ID3D11VideoProcessor,
If shared texture was created with D3D11_RESOURCE_MISC_SHARED,
d3d11videosink might use fallback texture to convert DXVA texture
to normal Direct3D texture. Then converted texture will be
copied to user-provided shared texture.
* Why not use generic appsink approach?
In order for application to be able to store video data
which was produced by GStreamer in application's own texture,
there would be two possible approaches,
one is copying our texture into application's own texture,
and the other is drawing on application's own texture directly.
The former (appsink way) cannot be a zero-copy by nature.
In order to support zero-copy processing, we need to draw on
application's own texture directly.
For example, assume that application wants RGBA texture.
Then we can imagine following case.
"d3d11h264dec ! d3d11convert ! video/x-raw(memory:D3D11Memory),format=RGBA ! appsink"
^
|_ allocate new Direct3D texture for RGBA format
In above case, d3d11convert will allocate new texture(s) for RGBA format
and then application will copy again the our RGBA texutre into
application's own texture. One texture allocation plus per frame GPU copy will hanppen
in that case therefore.
Moreover, in order for application to be able to access
our texture, we need to allocate texture with additional flags for
application's Direct3D11 device to be able to read texture data.
That would be another implementation burden on our side
But with this MR, we can configure pipeline in this way
"d3d11h264dec ! d3d11videosink".
In that way, we can save at least one texture allocation and
per frame texutre copy since d3d11videosink will convert incoming texture
into application's texture format directly without copy.
* What if we expose texture without conversion and application does
conversion by itself?
As mentioned above, for application to be able to access our texture
from application's Direct3D11 device, we need to allocate texture
in a special form. But in some case, that might not be possible.
Also, if a texture belongs to decoder DPB, exposing such texture
to application is unsafe and usual Direct3D11 shader cannot handle
such texture. To convert format, ID3D11VideoProcessor API needs to
be used but that would be a implementation burden for application.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1873>
2020-12-23 14:49:12 +00:00
|
|
|
* Copyright (C) 2020 Seungha Yang <seungha@centricular.com>
|
2019-01-30 11:07:29 +00:00
|
|
|
*
|
|
|
|
|
* This library is free software; you can redistribute it and/or
|
|
|
|
|
* modify it under the terms of the GNU Library General Public
|
|
|
|
|
* License as published by the Free Software Foundation; either
|
|
|
|
|
* version 2 of the License, or (at your option) any later version.
|
|
|
|
|
*
|
|
|
|
|
* This library is distributed in the hope that it will be useful,
|
|
|
|
|
* but WITHOUT ANY WARRANTY; without even the implied warranty of
|
|
|
|
|
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
|
|
|
|
|
* Library General Public License for more details.
|
|
|
|
|
*
|
|
|
|
|
* You should have received a copy of the GNU Library General Public
|
|
|
|
|
* License along with this library; if not, write to the
|
|
|
|
|
* Free Software Foundation, Inc., 51 Franklin St, Fifth Floor,
|
|
|
|
|
* Boston, MA 02110-1301, USA.
|
|
|
|
|
*/
|
|
|
|
|
|
2021-02-21 08:38:38 +00:00
|
|
|
/**
|
2021-04-17 11:37:13 +00:00
|
|
|
* SECTION:element-d3d11videosink
|
|
|
|
|
* @title: d3d11videosink
|
2021-02-21 08:38:38 +00:00
|
|
|
*
|
2021-04-17 11:37:13 +00:00
|
|
|
* Direct3D11 based video render element
|
2021-02-21 08:38:38 +00:00
|
|
|
*
|
|
|
|
|
* ## Example launch line
|
|
|
|
|
* ```
|
2021-04-17 11:37:13 +00:00
|
|
|
* gst-launch-1.0 videotestsrc ! d3d11upload ! d3d11videosink
|
2021-02-21 08:38:38 +00:00
|
|
|
* ```
|
2021-04-17 11:37:13 +00:00
|
|
|
* This pipeline will display test video stream on screen via d3d11videosink
|
2021-02-21 08:38:38 +00:00
|
|
|
*
|
|
|
|
|
* Since: 1.18
|
|
|
|
|
*
|
|
|
|
|
*/
|
|
|
|
|
|
2019-01-30 11:07:29 +00:00
|
|
|
#ifdef HAVE_CONFIG_H
|
|
|
|
|
#include "config.h"
|
|
|
|
|
#endif
|
|
|
|
|
|
|
|
|
|
#include "gstd3d11videosink.h"
|
2020-12-20 17:47:45 +00:00
|
|
|
#include "gstd3d11pluginutils.h"
|
2021-09-17 17:27:51 +00:00
|
|
|
#include <string>
|
2019-01-30 11:07:29 +00:00
|
|
|
|
2021-01-18 10:17:14 +00:00
|
|
|
#if GST_D3D11_WINAPI_APP
|
2019-12-30 09:58:59 +00:00
|
|
|
#include "gstd3d11window_corewindow.h"
|
|
|
|
|
#include "gstd3d11window_swapchainpanel.h"
|
2021-01-18 10:17:14 +00:00
|
|
|
#endif
|
|
|
|
|
#if (!GST_D3D11_WINAPI_ONLY_APP)
|
2019-12-30 09:58:59 +00:00
|
|
|
#include "gstd3d11window_win32.h"
|
|
|
|
|
#endif
|
d3d11videosink: Add support for drawing on application's own texture
Add a way to support drawing on application's texture instead of
usual window handle.
To make use of this new feature, application should follow below step.
1) Enable this feature by using "draw-on-shared-texture" property
2) Watch "begin-draw" signal
3) On "begin-draw" signal handler, application can request drawing
by using "draw" signal action. Note that "draw" signal action
should be happen before "begin-draw" signal handler is returned
NOTE 1) For texture sharing, creating a texture with
D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX flag is strongly recommend
if possible because we cannot ensure sync a texture
which was created with D3D11_RESOURCE_MISC_SHARED
and it would cause glitch with ID3D11VideoProcessor use case.
NOTE 2) Direct9Ex doesn't support texture sharing which was
created with D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX. In other words,
D3D11_RESOURCE_MISC_SHARED is the only option for Direct3D11/Direct9Ex interop.
NOTE 3) Because of missing synchronization around ID3D11VideoProcessor,
If shared texture was created with D3D11_RESOURCE_MISC_SHARED,
d3d11videosink might use fallback texture to convert DXVA texture
to normal Direct3D texture. Then converted texture will be
copied to user-provided shared texture.
* Why not use generic appsink approach?
In order for application to be able to store video data
which was produced by GStreamer in application's own texture,
there would be two possible approaches,
one is copying our texture into application's own texture,
and the other is drawing on application's own texture directly.
The former (appsink way) cannot be a zero-copy by nature.
In order to support zero-copy processing, we need to draw on
application's own texture directly.
For example, assume that application wants RGBA texture.
Then we can imagine following case.
"d3d11h264dec ! d3d11convert ! video/x-raw(memory:D3D11Memory),format=RGBA ! appsink"
^
|_ allocate new Direct3D texture for RGBA format
In above case, d3d11convert will allocate new texture(s) for RGBA format
and then application will copy again the our RGBA texutre into
application's own texture. One texture allocation plus per frame GPU copy will hanppen
in that case therefore.
Moreover, in order for application to be able to access
our texture, we need to allocate texture with additional flags for
application's Direct3D11 device to be able to read texture data.
That would be another implementation burden on our side
But with this MR, we can configure pipeline in this way
"d3d11h264dec ! d3d11videosink".
In that way, we can save at least one texture allocation and
per frame texutre copy since d3d11videosink will convert incoming texture
into application's texture format directly without copy.
* What if we expose texture without conversion and application does
conversion by itself?
As mentioned above, for application to be able to access our texture
from application's Direct3D11 device, we need to allocate texture
in a special form. But in some case, that might not be possible.
Also, if a texture belongs to decoder DPB, exposing such texture
to application is unsafe and usual Direct3D11 shader cannot handle
such texture. To convert format, ID3D11VideoProcessor API needs to
be used but that would be a implementation burden for application.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1873>
2020-12-23 14:49:12 +00:00
|
|
|
#include "gstd3d11window_dummy.h"
|
2019-12-30 09:58:59 +00:00
|
|
|
|
2019-01-30 11:07:29 +00:00
|
|
|
enum
|
|
|
|
|
{
|
|
|
|
|
PROP_0,
|
|
|
|
|
PROP_ADAPTER,
|
|
|
|
|
PROP_FORCE_ASPECT_RATIO,
|
2019-12-15 10:29:10 +00:00
|
|
|
PROP_ENABLE_NAVIGATION_EVENTS,
|
|
|
|
|
PROP_FULLSCREEN_TOGGLE_MODE,
|
|
|
|
|
PROP_FULLSCREEN,
|
d3d11videosink: Add support for drawing on application's own texture
Add a way to support drawing on application's texture instead of
usual window handle.
To make use of this new feature, application should follow below step.
1) Enable this feature by using "draw-on-shared-texture" property
2) Watch "begin-draw" signal
3) On "begin-draw" signal handler, application can request drawing
by using "draw" signal action. Note that "draw" signal action
should be happen before "begin-draw" signal handler is returned
NOTE 1) For texture sharing, creating a texture with
D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX flag is strongly recommend
if possible because we cannot ensure sync a texture
which was created with D3D11_RESOURCE_MISC_SHARED
and it would cause glitch with ID3D11VideoProcessor use case.
NOTE 2) Direct9Ex doesn't support texture sharing which was
created with D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX. In other words,
D3D11_RESOURCE_MISC_SHARED is the only option for Direct3D11/Direct9Ex interop.
NOTE 3) Because of missing synchronization around ID3D11VideoProcessor,
If shared texture was created with D3D11_RESOURCE_MISC_SHARED,
d3d11videosink might use fallback texture to convert DXVA texture
to normal Direct3D texture. Then converted texture will be
copied to user-provided shared texture.
* Why not use generic appsink approach?
In order for application to be able to store video data
which was produced by GStreamer in application's own texture,
there would be two possible approaches,
one is copying our texture into application's own texture,
and the other is drawing on application's own texture directly.
The former (appsink way) cannot be a zero-copy by nature.
In order to support zero-copy processing, we need to draw on
application's own texture directly.
For example, assume that application wants RGBA texture.
Then we can imagine following case.
"d3d11h264dec ! d3d11convert ! video/x-raw(memory:D3D11Memory),format=RGBA ! appsink"
^
|_ allocate new Direct3D texture for RGBA format
In above case, d3d11convert will allocate new texture(s) for RGBA format
and then application will copy again the our RGBA texutre into
application's own texture. One texture allocation plus per frame GPU copy will hanppen
in that case therefore.
Moreover, in order for application to be able to access
our texture, we need to allocate texture with additional flags for
application's Direct3D11 device to be able to read texture data.
That would be another implementation burden on our side
But with this MR, we can configure pipeline in this way
"d3d11h264dec ! d3d11videosink".
In that way, we can save at least one texture allocation and
per frame texutre copy since d3d11videosink will convert incoming texture
into application's texture format directly without copy.
* What if we expose texture without conversion and application does
conversion by itself?
As mentioned above, for application to be able to access our texture
from application's Direct3D11 device, we need to allocate texture
in a special form. But in some case, that might not be possible.
Also, if a texture belongs to decoder DPB, exposing such texture
to application is unsafe and usual Direct3D11 shader cannot handle
such texture. To convert format, ID3D11VideoProcessor API needs to
be used but that would be a implementation burden for application.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1873>
2020-12-23 14:49:12 +00:00
|
|
|
PROP_DRAW_ON_SHARED_TEXTURE,
|
2022-07-02 16:22:10 +00:00
|
|
|
PROP_ROTATE_METHOD,
|
2022-08-02 16:47:46 +00:00
|
|
|
PROP_GAMMA_MODE,
|
|
|
|
|
PROP_PRIMARIES_MODE,
|
2022-08-23 15:13:21 +00:00
|
|
|
PROP_DISPLAY_FORMAT,
|
2022-10-13 15:17:52 +00:00
|
|
|
PROP_EMIT_PRESENT,
|
2023-05-14 11:05:25 +00:00
|
|
|
PROP_RENDER_RECTANGE,
|
2019-01-30 11:07:29 +00:00
|
|
|
};
|
|
|
|
|
|
|
|
|
|
#define DEFAULT_ADAPTER -1
|
|
|
|
|
#define DEFAULT_FORCE_ASPECT_RATIO TRUE
|
|
|
|
|
#define DEFAULT_ENABLE_NAVIGATION_EVENTS TRUE
|
2019-12-15 10:29:10 +00:00
|
|
|
#define DEFAULT_FULLSCREEN_TOGGLE_MODE GST_D3D11_WINDOW_FULLSCREEN_TOGGLE_MODE_NONE
|
|
|
|
|
#define DEFAULT_FULLSCREEN FALSE
|
d3d11videosink: Add support for drawing on application's own texture
Add a way to support drawing on application's texture instead of
usual window handle.
To make use of this new feature, application should follow below step.
1) Enable this feature by using "draw-on-shared-texture" property
2) Watch "begin-draw" signal
3) On "begin-draw" signal handler, application can request drawing
by using "draw" signal action. Note that "draw" signal action
should be happen before "begin-draw" signal handler is returned
NOTE 1) For texture sharing, creating a texture with
D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX flag is strongly recommend
if possible because we cannot ensure sync a texture
which was created with D3D11_RESOURCE_MISC_SHARED
and it would cause glitch with ID3D11VideoProcessor use case.
NOTE 2) Direct9Ex doesn't support texture sharing which was
created with D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX. In other words,
D3D11_RESOURCE_MISC_SHARED is the only option for Direct3D11/Direct9Ex interop.
NOTE 3) Because of missing synchronization around ID3D11VideoProcessor,
If shared texture was created with D3D11_RESOURCE_MISC_SHARED,
d3d11videosink might use fallback texture to convert DXVA texture
to normal Direct3D texture. Then converted texture will be
copied to user-provided shared texture.
* Why not use generic appsink approach?
In order for application to be able to store video data
which was produced by GStreamer in application's own texture,
there would be two possible approaches,
one is copying our texture into application's own texture,
and the other is drawing on application's own texture directly.
The former (appsink way) cannot be a zero-copy by nature.
In order to support zero-copy processing, we need to draw on
application's own texture directly.
For example, assume that application wants RGBA texture.
Then we can imagine following case.
"d3d11h264dec ! d3d11convert ! video/x-raw(memory:D3D11Memory),format=RGBA ! appsink"
^
|_ allocate new Direct3D texture for RGBA format
In above case, d3d11convert will allocate new texture(s) for RGBA format
and then application will copy again the our RGBA texutre into
application's own texture. One texture allocation plus per frame GPU copy will hanppen
in that case therefore.
Moreover, in order for application to be able to access
our texture, we need to allocate texture with additional flags for
application's Direct3D11 device to be able to read texture data.
That would be another implementation burden on our side
But with this MR, we can configure pipeline in this way
"d3d11h264dec ! d3d11videosink".
In that way, we can save at least one texture allocation and
per frame texutre copy since d3d11videosink will convert incoming texture
into application's texture format directly without copy.
* What if we expose texture without conversion and application does
conversion by itself?
As mentioned above, for application to be able to access our texture
from application's Direct3D11 device, we need to allocate texture
in a special form. But in some case, that might not be possible.
Also, if a texture belongs to decoder DPB, exposing such texture
to application is unsafe and usual Direct3D11 shader cannot handle
such texture. To convert format, ID3D11VideoProcessor API needs to
be used but that would be a implementation burden for application.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1873>
2020-12-23 14:49:12 +00:00
|
|
|
#define DEFAULT_DRAW_ON_SHARED_TEXTURE FALSE
|
2022-08-02 16:47:46 +00:00
|
|
|
#define DEFAULT_GAMMA_MODE GST_VIDEO_GAMMA_MODE_NONE
|
|
|
|
|
#define DEFAULT_PRIMARIES_MODE GST_VIDEO_PRIMARIES_MODE_NONE
|
2022-08-23 15:13:21 +00:00
|
|
|
#define DEFAULT_DISPLAY_FORMAT DXGI_FORMAT_UNKNOWN
|
2022-10-13 15:17:52 +00:00
|
|
|
#define DEFAULT_EMIT_PRESENT FALSE
|
2022-08-23 15:13:21 +00:00
|
|
|
|
2022-10-21 17:49:00 +00:00
|
|
|
/**
|
|
|
|
|
* GstD3D11VideoSinkDisplayFormat:
|
|
|
|
|
*
|
|
|
|
|
* Swapchain's DXGI format
|
|
|
|
|
*
|
|
|
|
|
* Since: 1.22
|
|
|
|
|
*/
|
2022-08-23 15:13:21 +00:00
|
|
|
#define GST_TYPE_D3D11_VIDEO_SINK_DISPLAY_FORMAT (gst_d3d11_video_sink_display_format_type())
|
|
|
|
|
static GType
|
|
|
|
|
gst_d3d11_video_sink_display_format_type (void)
|
|
|
|
|
{
|
|
|
|
|
static GType format_type = 0;
|
|
|
|
|
|
|
|
|
|
GST_D3D11_CALL_ONCE_BEGIN {
|
|
|
|
|
static const GEnumValue format_types[] = {
|
2022-10-21 17:49:00 +00:00
|
|
|
/**
|
|
|
|
|
* GstD3D11VideoSinkDisplayFormat::unknown:
|
|
|
|
|
*
|
|
|
|
|
* Since: 1.22
|
|
|
|
|
*/
|
2022-08-23 15:13:21 +00:00
|
|
|
{DXGI_FORMAT_UNKNOWN, "DXGI_FORMAT_UNKNOWN", "unknown"},
|
2022-10-21 17:49:00 +00:00
|
|
|
|
|
|
|
|
/**
|
|
|
|
|
* GstD3D11VideoSinkDisplayFormat::r10g10b10a2-unorm:
|
|
|
|
|
*
|
|
|
|
|
* Since: 1.22
|
|
|
|
|
*/
|
2022-08-23 15:13:21 +00:00
|
|
|
{DXGI_FORMAT_R10G10B10A2_UNORM,
|
|
|
|
|
"DXGI_FORMAT_R10G10B10A2_UNORM", "r10g10b10a2-unorm"},
|
2022-10-21 17:49:00 +00:00
|
|
|
|
|
|
|
|
/**
|
|
|
|
|
* GstD3D11VideoSinkDisplayFormat::r8g8b8a8-unorm:
|
|
|
|
|
*
|
|
|
|
|
* Since: 1.22
|
|
|
|
|
*/
|
2022-08-23 15:13:21 +00:00
|
|
|
{DXGI_FORMAT_R8G8B8A8_UNORM,
|
|
|
|
|
"DXGI_FORMAT_R8G8B8A8_UNORM", "r8g8b8a8-unorm"},
|
2022-10-21 17:49:00 +00:00
|
|
|
|
|
|
|
|
/**
|
|
|
|
|
* GstD3D11VideoSinkDisplayFormat::b8g8r8a8-unorm:
|
|
|
|
|
*
|
|
|
|
|
* Since: 1.22
|
|
|
|
|
*/
|
2022-08-23 15:13:21 +00:00
|
|
|
{DXGI_FORMAT_B8G8R8A8_UNORM,
|
|
|
|
|
"DXGI_FORMAT_B8G8R8A8_UNORM", "b8g8r8a8-unorm"},
|
|
|
|
|
{0, nullptr, nullptr},
|
|
|
|
|
};
|
|
|
|
|
|
|
|
|
|
format_type = g_enum_register_static ("GstD3D11VideoSinkDisplayFormat",
|
|
|
|
|
format_types);
|
|
|
|
|
} GST_D3D11_CALL_ONCE_END;
|
|
|
|
|
|
|
|
|
|
return format_type;
|
|
|
|
|
}
|
d3d11videosink: Add support for drawing on application's own texture
Add a way to support drawing on application's texture instead of
usual window handle.
To make use of this new feature, application should follow below step.
1) Enable this feature by using "draw-on-shared-texture" property
2) Watch "begin-draw" signal
3) On "begin-draw" signal handler, application can request drawing
by using "draw" signal action. Note that "draw" signal action
should be happen before "begin-draw" signal handler is returned
NOTE 1) For texture sharing, creating a texture with
D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX flag is strongly recommend
if possible because we cannot ensure sync a texture
which was created with D3D11_RESOURCE_MISC_SHARED
and it would cause glitch with ID3D11VideoProcessor use case.
NOTE 2) Direct9Ex doesn't support texture sharing which was
created with D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX. In other words,
D3D11_RESOURCE_MISC_SHARED is the only option for Direct3D11/Direct9Ex interop.
NOTE 3) Because of missing synchronization around ID3D11VideoProcessor,
If shared texture was created with D3D11_RESOURCE_MISC_SHARED,
d3d11videosink might use fallback texture to convert DXVA texture
to normal Direct3D texture. Then converted texture will be
copied to user-provided shared texture.
* Why not use generic appsink approach?
In order for application to be able to store video data
which was produced by GStreamer in application's own texture,
there would be two possible approaches,
one is copying our texture into application's own texture,
and the other is drawing on application's own texture directly.
The former (appsink way) cannot be a zero-copy by nature.
In order to support zero-copy processing, we need to draw on
application's own texture directly.
For example, assume that application wants RGBA texture.
Then we can imagine following case.
"d3d11h264dec ! d3d11convert ! video/x-raw(memory:D3D11Memory),format=RGBA ! appsink"
^
|_ allocate new Direct3D texture for RGBA format
In above case, d3d11convert will allocate new texture(s) for RGBA format
and then application will copy again the our RGBA texutre into
application's own texture. One texture allocation plus per frame GPU copy will hanppen
in that case therefore.
Moreover, in order for application to be able to access
our texture, we need to allocate texture with additional flags for
application's Direct3D11 device to be able to read texture data.
That would be another implementation burden on our side
But with this MR, we can configure pipeline in this way
"d3d11h264dec ! d3d11videosink".
In that way, we can save at least one texture allocation and
per frame texutre copy since d3d11videosink will convert incoming texture
into application's texture format directly without copy.
* What if we expose texture without conversion and application does
conversion by itself?
As mentioned above, for application to be able to access our texture
from application's Direct3D11 device, we need to allocate texture
in a special form. But in some case, that might not be possible.
Also, if a texture belongs to decoder DPB, exposing such texture
to application is unsafe and usual Direct3D11 shader cannot handle
such texture. To convert format, ID3D11VideoProcessor API needs to
be used but that would be a implementation burden for application.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1873>
2020-12-23 14:49:12 +00:00
|
|
|
|
|
|
|
|
enum
|
|
|
|
|
{
|
|
|
|
|
/* signals */
|
|
|
|
|
SIGNAL_BEGIN_DRAW,
|
2022-08-19 11:25:31 +00:00
|
|
|
SIGNAL_PRESENT,
|
d3d11videosink: Add support for drawing on application's own texture
Add a way to support drawing on application's texture instead of
usual window handle.
To make use of this new feature, application should follow below step.
1) Enable this feature by using "draw-on-shared-texture" property
2) Watch "begin-draw" signal
3) On "begin-draw" signal handler, application can request drawing
by using "draw" signal action. Note that "draw" signal action
should be happen before "begin-draw" signal handler is returned
NOTE 1) For texture sharing, creating a texture with
D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX flag is strongly recommend
if possible because we cannot ensure sync a texture
which was created with D3D11_RESOURCE_MISC_SHARED
and it would cause glitch with ID3D11VideoProcessor use case.
NOTE 2) Direct9Ex doesn't support texture sharing which was
created with D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX. In other words,
D3D11_RESOURCE_MISC_SHARED is the only option for Direct3D11/Direct9Ex interop.
NOTE 3) Because of missing synchronization around ID3D11VideoProcessor,
If shared texture was created with D3D11_RESOURCE_MISC_SHARED,
d3d11videosink might use fallback texture to convert DXVA texture
to normal Direct3D texture. Then converted texture will be
copied to user-provided shared texture.
* Why not use generic appsink approach?
In order for application to be able to store video data
which was produced by GStreamer in application's own texture,
there would be two possible approaches,
one is copying our texture into application's own texture,
and the other is drawing on application's own texture directly.
The former (appsink way) cannot be a zero-copy by nature.
In order to support zero-copy processing, we need to draw on
application's own texture directly.
For example, assume that application wants RGBA texture.
Then we can imagine following case.
"d3d11h264dec ! d3d11convert ! video/x-raw(memory:D3D11Memory),format=RGBA ! appsink"
^
|_ allocate new Direct3D texture for RGBA format
In above case, d3d11convert will allocate new texture(s) for RGBA format
and then application will copy again the our RGBA texutre into
application's own texture. One texture allocation plus per frame GPU copy will hanppen
in that case therefore.
Moreover, in order for application to be able to access
our texture, we need to allocate texture with additional flags for
application's Direct3D11 device to be able to read texture data.
That would be another implementation burden on our side
But with this MR, we can configure pipeline in this way
"d3d11h264dec ! d3d11videosink".
In that way, we can save at least one texture allocation and
per frame texutre copy since d3d11videosink will convert incoming texture
into application's texture format directly without copy.
* What if we expose texture without conversion and application does
conversion by itself?
As mentioned above, for application to be able to access our texture
from application's Direct3D11 device, we need to allocate texture
in a special form. But in some case, that might not be possible.
Also, if a texture belongs to decoder DPB, exposing such texture
to application is unsafe and usual Direct3D11 shader cannot handle
such texture. To convert format, ID3D11VideoProcessor API needs to
be used but that would be a implementation burden for application.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1873>
2020-12-23 14:49:12 +00:00
|
|
|
|
|
|
|
|
/* actions */
|
|
|
|
|
SIGNAL_DRAW,
|
|
|
|
|
|
|
|
|
|
LAST_SIGNAL
|
|
|
|
|
};
|
|
|
|
|
|
|
|
|
|
static guint gst_d3d11_video_sink_signals[LAST_SIGNAL] = { 0, };
|
2019-01-30 11:07:29 +00:00
|
|
|
|
2021-01-19 17:29:43 +00:00
|
|
|
static GstStaticCaps pad_template_caps =
|
2019-08-24 08:43:43 +00:00
|
|
|
GST_STATIC_CAPS (GST_VIDEO_CAPS_MAKE_WITH_FEATURES
|
2021-01-19 17:29:43 +00:00
|
|
|
(GST_CAPS_FEATURE_MEMORY_D3D11_MEMORY, GST_D3D11_SINK_FORMATS) "; "
|
|
|
|
|
GST_VIDEO_CAPS_MAKE_WITH_FEATURES
|
|
|
|
|
(GST_CAPS_FEATURE_MEMORY_D3D11_MEMORY ","
|
2021-04-17 11:37:13 +00:00
|
|
|
GST_CAPS_FEATURE_META_GST_VIDEO_OVERLAY_COMPOSITION,
|
|
|
|
|
GST_D3D11_SINK_FORMATS) ";"
|
|
|
|
|
GST_VIDEO_CAPS_MAKE (GST_D3D11_SINK_FORMATS) "; "
|
|
|
|
|
GST_VIDEO_CAPS_MAKE_WITH_FEATURES
|
|
|
|
|
(GST_CAPS_FEATURE_MEMORY_SYSTEM_MEMORY ","
|
2021-01-19 17:29:43 +00:00
|
|
|
GST_CAPS_FEATURE_META_GST_VIDEO_OVERLAY_COMPOSITION,
|
|
|
|
|
GST_D3D11_SINK_FORMATS));
|
2019-01-30 11:07:29 +00:00
|
|
|
|
|
|
|
|
GST_DEBUG_CATEGORY (d3d11_video_sink_debug);
|
|
|
|
|
#define GST_CAT_DEFAULT d3d11_video_sink_debug
|
|
|
|
|
|
2020-09-25 18:27:39 +00:00
|
|
|
struct _GstD3D11VideoSink
|
|
|
|
|
{
|
|
|
|
|
GstVideoSink parent;
|
|
|
|
|
GstD3D11Device *device;
|
|
|
|
|
GstD3D11Window *window;
|
|
|
|
|
gint video_width;
|
|
|
|
|
gint video_height;
|
|
|
|
|
GstVideoInfo info;
|
|
|
|
|
guintptr window_id;
|
2021-04-23 10:29:55 +00:00
|
|
|
gboolean caps_updated;
|
2022-12-15 17:52:08 +00:00
|
|
|
GstBuffer *prepared_buffer;
|
|
|
|
|
GstBufferPool *pool;
|
2021-04-23 10:29:55 +00:00
|
|
|
|
2020-09-25 18:27:39 +00:00
|
|
|
/* properties */
|
|
|
|
|
gint adapter;
|
|
|
|
|
gboolean force_aspect_ratio;
|
|
|
|
|
gboolean enable_navigation_events;
|
|
|
|
|
GstD3D11WindowFullscreenToggleMode fullscreen_toggle_mode;
|
|
|
|
|
gboolean fullscreen;
|
d3d11videosink: Add support for drawing on application's own texture
Add a way to support drawing on application's texture instead of
usual window handle.
To make use of this new feature, application should follow below step.
1) Enable this feature by using "draw-on-shared-texture" property
2) Watch "begin-draw" signal
3) On "begin-draw" signal handler, application can request drawing
by using "draw" signal action. Note that "draw" signal action
should be happen before "begin-draw" signal handler is returned
NOTE 1) For texture sharing, creating a texture with
D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX flag is strongly recommend
if possible because we cannot ensure sync a texture
which was created with D3D11_RESOURCE_MISC_SHARED
and it would cause glitch with ID3D11VideoProcessor use case.
NOTE 2) Direct9Ex doesn't support texture sharing which was
created with D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX. In other words,
D3D11_RESOURCE_MISC_SHARED is the only option for Direct3D11/Direct9Ex interop.
NOTE 3) Because of missing synchronization around ID3D11VideoProcessor,
If shared texture was created with D3D11_RESOURCE_MISC_SHARED,
d3d11videosink might use fallback texture to convert DXVA texture
to normal Direct3D texture. Then converted texture will be
copied to user-provided shared texture.
* Why not use generic appsink approach?
In order for application to be able to store video data
which was produced by GStreamer in application's own texture,
there would be two possible approaches,
one is copying our texture into application's own texture,
and the other is drawing on application's own texture directly.
The former (appsink way) cannot be a zero-copy by nature.
In order to support zero-copy processing, we need to draw on
application's own texture directly.
For example, assume that application wants RGBA texture.
Then we can imagine following case.
"d3d11h264dec ! d3d11convert ! video/x-raw(memory:D3D11Memory),format=RGBA ! appsink"
^
|_ allocate new Direct3D texture for RGBA format
In above case, d3d11convert will allocate new texture(s) for RGBA format
and then application will copy again the our RGBA texutre into
application's own texture. One texture allocation plus per frame GPU copy will hanppen
in that case therefore.
Moreover, in order for application to be able to access
our texture, we need to allocate texture with additional flags for
application's Direct3D11 device to be able to read texture data.
That would be another implementation burden on our side
But with this MR, we can configure pipeline in this way
"d3d11h264dec ! d3d11videosink".
In that way, we can save at least one texture allocation and
per frame texutre copy since d3d11videosink will convert incoming texture
into application's texture format directly without copy.
* What if we expose texture without conversion and application does
conversion by itself?
As mentioned above, for application to be able to access our texture
from application's Direct3D11 device, we need to allocate texture
in a special form. But in some case, that might not be possible.
Also, if a texture belongs to decoder DPB, exposing such texture
to application is unsafe and usual Direct3D11 shader cannot handle
such texture. To convert format, ID3D11VideoProcessor API needs to
be used but that would be a implementation burden for application.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1873>
2020-12-23 14:49:12 +00:00
|
|
|
gboolean draw_on_shared_texture;
|
2022-08-02 16:47:46 +00:00
|
|
|
GstVideoGammaMode gamma_mode;
|
|
|
|
|
GstVideoPrimariesMode primaries_mode;
|
2022-08-23 15:13:21 +00:00
|
|
|
DXGI_FORMAT display_format;
|
2022-10-13 15:17:52 +00:00
|
|
|
gboolean emit_present;
|
2020-09-25 18:27:39 +00:00
|
|
|
|
|
|
|
|
/* saved render rectangle until we have a window */
|
|
|
|
|
GstVideoRectangle render_rect;
|
|
|
|
|
gboolean pending_render_rect;
|
|
|
|
|
|
d3d11videosink: Add support for drawing on application's own texture
Add a way to support drawing on application's texture instead of
usual window handle.
To make use of this new feature, application should follow below step.
1) Enable this feature by using "draw-on-shared-texture" property
2) Watch "begin-draw" signal
3) On "begin-draw" signal handler, application can request drawing
by using "draw" signal action. Note that "draw" signal action
should be happen before "begin-draw" signal handler is returned
NOTE 1) For texture sharing, creating a texture with
D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX flag is strongly recommend
if possible because we cannot ensure sync a texture
which was created with D3D11_RESOURCE_MISC_SHARED
and it would cause glitch with ID3D11VideoProcessor use case.
NOTE 2) Direct9Ex doesn't support texture sharing which was
created with D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX. In other words,
D3D11_RESOURCE_MISC_SHARED is the only option for Direct3D11/Direct9Ex interop.
NOTE 3) Because of missing synchronization around ID3D11VideoProcessor,
If shared texture was created with D3D11_RESOURCE_MISC_SHARED,
d3d11videosink might use fallback texture to convert DXVA texture
to normal Direct3D texture. Then converted texture will be
copied to user-provided shared texture.
* Why not use generic appsink approach?
In order for application to be able to store video data
which was produced by GStreamer in application's own texture,
there would be two possible approaches,
one is copying our texture into application's own texture,
and the other is drawing on application's own texture directly.
The former (appsink way) cannot be a zero-copy by nature.
In order to support zero-copy processing, we need to draw on
application's own texture directly.
For example, assume that application wants RGBA texture.
Then we can imagine following case.
"d3d11h264dec ! d3d11convert ! video/x-raw(memory:D3D11Memory),format=RGBA ! appsink"
^
|_ allocate new Direct3D texture for RGBA format
In above case, d3d11convert will allocate new texture(s) for RGBA format
and then application will copy again the our RGBA texutre into
application's own texture. One texture allocation plus per frame GPU copy will hanppen
in that case therefore.
Moreover, in order for application to be able to access
our texture, we need to allocate texture with additional flags for
application's Direct3D11 device to be able to read texture data.
That would be another implementation burden on our side
But with this MR, we can configure pipeline in this way
"d3d11h264dec ! d3d11videosink".
In that way, we can save at least one texture allocation and
per frame texutre copy since d3d11videosink will convert incoming texture
into application's texture format directly without copy.
* What if we expose texture without conversion and application does
conversion by itself?
As mentioned above, for application to be able to access our texture
from application's Direct3D11 device, we need to allocate texture
in a special form. But in some case, that might not be possible.
Also, if a texture belongs to decoder DPB, exposing such texture
to application is unsafe and usual Direct3D11 shader cannot handle
such texture. To convert format, ID3D11VideoProcessor API needs to
be used but that would be a implementation burden for application.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1873>
2020-12-23 14:49:12 +00:00
|
|
|
/* For drawing on user texture */
|
|
|
|
|
gboolean drawing;
|
2022-08-18 15:05:17 +00:00
|
|
|
CRITICAL_SECTION lock;
|
2021-09-17 17:27:51 +00:00
|
|
|
|
|
|
|
|
gchar *title;
|
2022-07-02 16:22:10 +00:00
|
|
|
|
|
|
|
|
/* method configured via property */
|
|
|
|
|
GstVideoOrientationMethod method;
|
|
|
|
|
/* method parsed from tag */
|
|
|
|
|
GstVideoOrientationMethod tag_method;
|
|
|
|
|
/* method currently selected based on "method" and "tag_method" */
|
|
|
|
|
GstVideoOrientationMethod selected_method;
|
2020-09-25 18:27:39 +00:00
|
|
|
};
|
|
|
|
|
|
2019-01-30 11:07:29 +00:00
|
|
|
static void gst_d3d11_videosink_set_property (GObject * object, guint prop_id,
|
|
|
|
|
const GValue * value, GParamSpec * pspec);
|
|
|
|
|
static void gst_d3d11_videosink_get_property (GObject * object, guint prop_id,
|
|
|
|
|
GValue * value, GParamSpec * pspec);
|
d3d11videosink: Add support for drawing on application's own texture
Add a way to support drawing on application's texture instead of
usual window handle.
To make use of this new feature, application should follow below step.
1) Enable this feature by using "draw-on-shared-texture" property
2) Watch "begin-draw" signal
3) On "begin-draw" signal handler, application can request drawing
by using "draw" signal action. Note that "draw" signal action
should be happen before "begin-draw" signal handler is returned
NOTE 1) For texture sharing, creating a texture with
D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX flag is strongly recommend
if possible because we cannot ensure sync a texture
which was created with D3D11_RESOURCE_MISC_SHARED
and it would cause glitch with ID3D11VideoProcessor use case.
NOTE 2) Direct9Ex doesn't support texture sharing which was
created with D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX. In other words,
D3D11_RESOURCE_MISC_SHARED is the only option for Direct3D11/Direct9Ex interop.
NOTE 3) Because of missing synchronization around ID3D11VideoProcessor,
If shared texture was created with D3D11_RESOURCE_MISC_SHARED,
d3d11videosink might use fallback texture to convert DXVA texture
to normal Direct3D texture. Then converted texture will be
copied to user-provided shared texture.
* Why not use generic appsink approach?
In order for application to be able to store video data
which was produced by GStreamer in application's own texture,
there would be two possible approaches,
one is copying our texture into application's own texture,
and the other is drawing on application's own texture directly.
The former (appsink way) cannot be a zero-copy by nature.
In order to support zero-copy processing, we need to draw on
application's own texture directly.
For example, assume that application wants RGBA texture.
Then we can imagine following case.
"d3d11h264dec ! d3d11convert ! video/x-raw(memory:D3D11Memory),format=RGBA ! appsink"
^
|_ allocate new Direct3D texture for RGBA format
In above case, d3d11convert will allocate new texture(s) for RGBA format
and then application will copy again the our RGBA texutre into
application's own texture. One texture allocation plus per frame GPU copy will hanppen
in that case therefore.
Moreover, in order for application to be able to access
our texture, we need to allocate texture with additional flags for
application's Direct3D11 device to be able to read texture data.
That would be another implementation burden on our side
But with this MR, we can configure pipeline in this way
"d3d11h264dec ! d3d11videosink".
In that way, we can save at least one texture allocation and
per frame texutre copy since d3d11videosink will convert incoming texture
into application's texture format directly without copy.
* What if we expose texture without conversion and application does
conversion by itself?
As mentioned above, for application to be able to access our texture
from application's Direct3D11 device, we need to allocate texture
in a special form. But in some case, that might not be possible.
Also, if a texture belongs to decoder DPB, exposing such texture
to application is unsafe and usual Direct3D11 shader cannot handle
such texture. To convert format, ID3D11VideoProcessor API needs to
be used but that would be a implementation burden for application.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1873>
2020-12-23 14:49:12 +00:00
|
|
|
static void gst_d3d11_video_sink_finalize (GObject * object);
|
|
|
|
|
static gboolean
|
|
|
|
|
gst_d3d11_video_sink_draw_action (GstD3D11VideoSink * self,
|
|
|
|
|
gpointer shared_handle, guint texture_misc_flags, guint64 acquire_key,
|
|
|
|
|
guint64 release_key);
|
2019-01-30 11:07:29 +00:00
|
|
|
|
|
|
|
|
static void
|
|
|
|
|
gst_d3d11_video_sink_video_overlay_init (GstVideoOverlayInterface * iface);
|
|
|
|
|
static void
|
|
|
|
|
gst_d3d11_video_sink_navigation_init (GstNavigationInterface * iface);
|
|
|
|
|
|
|
|
|
|
static void gst_d3d11_video_sink_set_context (GstElement * element,
|
|
|
|
|
GstContext * context);
|
|
|
|
|
static GstCaps *gst_d3d11_video_sink_get_caps (GstBaseSink * sink,
|
|
|
|
|
GstCaps * filter);
|
|
|
|
|
static gboolean gst_d3d11_video_sink_set_caps (GstBaseSink * sink,
|
|
|
|
|
GstCaps * caps);
|
|
|
|
|
|
|
|
|
|
static gboolean gst_d3d11_video_sink_start (GstBaseSink * sink);
|
|
|
|
|
static gboolean gst_d3d11_video_sink_stop (GstBaseSink * sink);
|
|
|
|
|
static gboolean gst_d3d11_video_sink_propose_allocation (GstBaseSink * sink,
|
|
|
|
|
GstQuery * query);
|
2019-07-18 14:12:13 +00:00
|
|
|
static gboolean gst_d3d11_video_sink_query (GstBaseSink * sink,
|
|
|
|
|
GstQuery * query);
|
2019-12-02 14:27:42 +00:00
|
|
|
static gboolean gst_d3d11_video_sink_unlock (GstBaseSink * sink);
|
2019-12-30 09:58:59 +00:00
|
|
|
static gboolean gst_d3d11_video_sink_unlock_stop (GstBaseSink * sink);
|
2021-09-17 17:27:51 +00:00
|
|
|
static gboolean gst_d3d11_video_sink_event (GstBaseSink * sink,
|
|
|
|
|
GstEvent * event);
|
2022-12-15 17:52:08 +00:00
|
|
|
static GstFlowReturn gst_d3d11_video_sink_prepare (GstBaseSink * sink,
|
|
|
|
|
GstBuffer * buffer);
|
2019-01-30 11:07:29 +00:00
|
|
|
static GstFlowReturn
|
|
|
|
|
gst_d3d11_video_sink_show_frame (GstVideoSink * sink, GstBuffer * buf);
|
2020-12-20 13:12:44 +00:00
|
|
|
static gboolean gst_d3d11_video_sink_prepare_window (GstD3D11VideoSink * self);
|
2022-07-02 16:22:10 +00:00
|
|
|
static void gst_d3d11_video_sink_set_orientation (GstD3D11VideoSink * self,
|
|
|
|
|
GstVideoOrientationMethod method, gboolean from_tag);
|
2019-01-30 11:07:29 +00:00
|
|
|
|
|
|
|
|
#define gst_d3d11_video_sink_parent_class parent_class
|
|
|
|
|
G_DEFINE_TYPE_WITH_CODE (GstD3D11VideoSink, gst_d3d11_video_sink,
|
|
|
|
|
GST_TYPE_VIDEO_SINK,
|
|
|
|
|
G_IMPLEMENT_INTERFACE (GST_TYPE_VIDEO_OVERLAY,
|
|
|
|
|
gst_d3d11_video_sink_video_overlay_init);
|
|
|
|
|
G_IMPLEMENT_INTERFACE (GST_TYPE_NAVIGATION,
|
|
|
|
|
gst_d3d11_video_sink_navigation_init);
|
|
|
|
|
GST_DEBUG_CATEGORY_INIT (d3d11_video_sink_debug,
|
|
|
|
|
"d3d11videosink", 0, "Direct3D11 Video Sink"));
|
|
|
|
|
|
|
|
|
|
static void
|
|
|
|
|
gst_d3d11_video_sink_class_init (GstD3D11VideoSinkClass * klass)
|
|
|
|
|
{
|
|
|
|
|
GObjectClass *gobject_class = G_OBJECT_CLASS (klass);
|
|
|
|
|
GstElementClass *element_class = GST_ELEMENT_CLASS (klass);
|
|
|
|
|
GstBaseSinkClass *basesink_class = GST_BASE_SINK_CLASS (klass);
|
|
|
|
|
GstVideoSinkClass *videosink_class = GST_VIDEO_SINK_CLASS (klass);
|
2021-01-19 17:29:43 +00:00
|
|
|
GstCaps *caps;
|
2019-01-30 11:07:29 +00:00
|
|
|
|
|
|
|
|
gobject_class->set_property = gst_d3d11_videosink_set_property;
|
|
|
|
|
gobject_class->get_property = gst_d3d11_videosink_get_property;
|
d3d11videosink: Add support for drawing on application's own texture
Add a way to support drawing on application's texture instead of
usual window handle.
To make use of this new feature, application should follow below step.
1) Enable this feature by using "draw-on-shared-texture" property
2) Watch "begin-draw" signal
3) On "begin-draw" signal handler, application can request drawing
by using "draw" signal action. Note that "draw" signal action
should be happen before "begin-draw" signal handler is returned
NOTE 1) For texture sharing, creating a texture with
D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX flag is strongly recommend
if possible because we cannot ensure sync a texture
which was created with D3D11_RESOURCE_MISC_SHARED
and it would cause glitch with ID3D11VideoProcessor use case.
NOTE 2) Direct9Ex doesn't support texture sharing which was
created with D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX. In other words,
D3D11_RESOURCE_MISC_SHARED is the only option for Direct3D11/Direct9Ex interop.
NOTE 3) Because of missing synchronization around ID3D11VideoProcessor,
If shared texture was created with D3D11_RESOURCE_MISC_SHARED,
d3d11videosink might use fallback texture to convert DXVA texture
to normal Direct3D texture. Then converted texture will be
copied to user-provided shared texture.
* Why not use generic appsink approach?
In order for application to be able to store video data
which was produced by GStreamer in application's own texture,
there would be two possible approaches,
one is copying our texture into application's own texture,
and the other is drawing on application's own texture directly.
The former (appsink way) cannot be a zero-copy by nature.
In order to support zero-copy processing, we need to draw on
application's own texture directly.
For example, assume that application wants RGBA texture.
Then we can imagine following case.
"d3d11h264dec ! d3d11convert ! video/x-raw(memory:D3D11Memory),format=RGBA ! appsink"
^
|_ allocate new Direct3D texture for RGBA format
In above case, d3d11convert will allocate new texture(s) for RGBA format
and then application will copy again the our RGBA texutre into
application's own texture. One texture allocation plus per frame GPU copy will hanppen
in that case therefore.
Moreover, in order for application to be able to access
our texture, we need to allocate texture with additional flags for
application's Direct3D11 device to be able to read texture data.
That would be another implementation burden on our side
But with this MR, we can configure pipeline in this way
"d3d11h264dec ! d3d11videosink".
In that way, we can save at least one texture allocation and
per frame texutre copy since d3d11videosink will convert incoming texture
into application's texture format directly without copy.
* What if we expose texture without conversion and application does
conversion by itself?
As mentioned above, for application to be able to access our texture
from application's Direct3D11 device, we need to allocate texture
in a special form. But in some case, that might not be possible.
Also, if a texture belongs to decoder DPB, exposing such texture
to application is unsafe and usual Direct3D11 shader cannot handle
such texture. To convert format, ID3D11VideoProcessor API needs to
be used but that would be a implementation burden for application.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1873>
2020-12-23 14:49:12 +00:00
|
|
|
gobject_class->finalize = gst_d3d11_video_sink_finalize;
|
2019-01-30 11:07:29 +00:00
|
|
|
|
|
|
|
|
g_object_class_install_property (gobject_class, PROP_ADAPTER,
|
|
|
|
|
g_param_spec_int ("adapter", "Adapter",
|
|
|
|
|
"Adapter index for creating device (-1 for default)",
|
|
|
|
|
-1, G_MAXINT32, DEFAULT_ADAPTER,
|
2021-03-13 08:40:57 +00:00
|
|
|
(GParamFlags) (G_PARAM_READWRITE | GST_PARAM_MUTABLE_READY |
|
|
|
|
|
G_PARAM_STATIC_STRINGS)));
|
2019-01-30 11:07:29 +00:00
|
|
|
|
|
|
|
|
g_object_class_install_property (gobject_class, PROP_FORCE_ASPECT_RATIO,
|
|
|
|
|
g_param_spec_boolean ("force-aspect-ratio",
|
|
|
|
|
"Force aspect ratio",
|
|
|
|
|
"When enabled, scaling will respect original aspect ratio",
|
|
|
|
|
DEFAULT_FORCE_ASPECT_RATIO,
|
2021-03-13 08:40:57 +00:00
|
|
|
(GParamFlags) (G_PARAM_READWRITE | G_PARAM_STATIC_STRINGS)));
|
2019-01-30 11:07:29 +00:00
|
|
|
|
|
|
|
|
g_object_class_install_property (gobject_class, PROP_ENABLE_NAVIGATION_EVENTS,
|
|
|
|
|
g_param_spec_boolean ("enable-navigation-events",
|
|
|
|
|
"Enable navigation events",
|
|
|
|
|
"When enabled, navigation events are sent upstream",
|
|
|
|
|
DEFAULT_ENABLE_NAVIGATION_EVENTS,
|
2021-03-13 08:40:57 +00:00
|
|
|
(GParamFlags) (G_PARAM_READWRITE | G_PARAM_STATIC_STRINGS)));
|
2019-01-30 11:07:29 +00:00
|
|
|
|
2019-12-15 10:29:10 +00:00
|
|
|
g_object_class_install_property (gobject_class, PROP_FULLSCREEN_TOGGLE_MODE,
|
|
|
|
|
g_param_spec_flags ("fullscreen-toggle-mode",
|
|
|
|
|
"Full screen toggle mode",
|
|
|
|
|
"Full screen toggle mode used to trigger fullscreen mode change",
|
|
|
|
|
GST_D3D11_WINDOW_TOGGLE_MODE_GET_TYPE, DEFAULT_FULLSCREEN_TOGGLE_MODE,
|
2021-03-13 08:40:57 +00:00
|
|
|
(GParamFlags) (G_PARAM_READWRITE | G_PARAM_STATIC_STRINGS)));
|
2019-12-15 10:29:10 +00:00
|
|
|
|
|
|
|
|
g_object_class_install_property (gobject_class, PROP_FULLSCREEN,
|
|
|
|
|
g_param_spec_boolean ("fullscreen",
|
|
|
|
|
"fullscreen",
|
|
|
|
|
"Ignored when \"fullscreen-toggle-mode\" does not include \"property\"",
|
2021-03-13 08:40:57 +00:00
|
|
|
DEFAULT_FULLSCREEN,
|
|
|
|
|
(GParamFlags) (G_PARAM_READWRITE | G_PARAM_STATIC_STRINGS)));
|
2019-12-15 10:29:10 +00:00
|
|
|
|
d3d11videosink: Add support for drawing on application's own texture
Add a way to support drawing on application's texture instead of
usual window handle.
To make use of this new feature, application should follow below step.
1) Enable this feature by using "draw-on-shared-texture" property
2) Watch "begin-draw" signal
3) On "begin-draw" signal handler, application can request drawing
by using "draw" signal action. Note that "draw" signal action
should be happen before "begin-draw" signal handler is returned
NOTE 1) For texture sharing, creating a texture with
D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX flag is strongly recommend
if possible because we cannot ensure sync a texture
which was created with D3D11_RESOURCE_MISC_SHARED
and it would cause glitch with ID3D11VideoProcessor use case.
NOTE 2) Direct9Ex doesn't support texture sharing which was
created with D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX. In other words,
D3D11_RESOURCE_MISC_SHARED is the only option for Direct3D11/Direct9Ex interop.
NOTE 3) Because of missing synchronization around ID3D11VideoProcessor,
If shared texture was created with D3D11_RESOURCE_MISC_SHARED,
d3d11videosink might use fallback texture to convert DXVA texture
to normal Direct3D texture. Then converted texture will be
copied to user-provided shared texture.
* Why not use generic appsink approach?
In order for application to be able to store video data
which was produced by GStreamer in application's own texture,
there would be two possible approaches,
one is copying our texture into application's own texture,
and the other is drawing on application's own texture directly.
The former (appsink way) cannot be a zero-copy by nature.
In order to support zero-copy processing, we need to draw on
application's own texture directly.
For example, assume that application wants RGBA texture.
Then we can imagine following case.
"d3d11h264dec ! d3d11convert ! video/x-raw(memory:D3D11Memory),format=RGBA ! appsink"
^
|_ allocate new Direct3D texture for RGBA format
In above case, d3d11convert will allocate new texture(s) for RGBA format
and then application will copy again the our RGBA texutre into
application's own texture. One texture allocation plus per frame GPU copy will hanppen
in that case therefore.
Moreover, in order for application to be able to access
our texture, we need to allocate texture with additional flags for
application's Direct3D11 device to be able to read texture data.
That would be another implementation burden on our side
But with this MR, we can configure pipeline in this way
"d3d11h264dec ! d3d11videosink".
In that way, we can save at least one texture allocation and
per frame texutre copy since d3d11videosink will convert incoming texture
into application's texture format directly without copy.
* What if we expose texture without conversion and application does
conversion by itself?
As mentioned above, for application to be able to access our texture
from application's Direct3D11 device, we need to allocate texture
in a special form. But in some case, that might not be possible.
Also, if a texture belongs to decoder DPB, exposing such texture
to application is unsafe and usual Direct3D11 shader cannot handle
such texture. To convert format, ID3D11VideoProcessor API needs to
be used but that would be a implementation burden for application.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1873>
2020-12-23 14:49:12 +00:00
|
|
|
/**
|
|
|
|
|
* GstD3D11VideoSink:draw-on-shared-texture:
|
|
|
|
|
*
|
2021-02-21 08:38:38 +00:00
|
|
|
* Instruct the sink to draw on a shared texture provided by user.
|
2021-04-17 11:37:13 +00:00
|
|
|
* User must watch #d3d11videosink::begin-draw signal and should call
|
|
|
|
|
* #d3d11videosink::draw method on the #d3d11videosink::begin-draw
|
2021-02-21 08:38:38 +00:00
|
|
|
* signal handler.
|
d3d11videosink: Add support for drawing on application's own texture
Add a way to support drawing on application's texture instead of
usual window handle.
To make use of this new feature, application should follow below step.
1) Enable this feature by using "draw-on-shared-texture" property
2) Watch "begin-draw" signal
3) On "begin-draw" signal handler, application can request drawing
by using "draw" signal action. Note that "draw" signal action
should be happen before "begin-draw" signal handler is returned
NOTE 1) For texture sharing, creating a texture with
D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX flag is strongly recommend
if possible because we cannot ensure sync a texture
which was created with D3D11_RESOURCE_MISC_SHARED
and it would cause glitch with ID3D11VideoProcessor use case.
NOTE 2) Direct9Ex doesn't support texture sharing which was
created with D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX. In other words,
D3D11_RESOURCE_MISC_SHARED is the only option for Direct3D11/Direct9Ex interop.
NOTE 3) Because of missing synchronization around ID3D11VideoProcessor,
If shared texture was created with D3D11_RESOURCE_MISC_SHARED,
d3d11videosink might use fallback texture to convert DXVA texture
to normal Direct3D texture. Then converted texture will be
copied to user-provided shared texture.
* Why not use generic appsink approach?
In order for application to be able to store video data
which was produced by GStreamer in application's own texture,
there would be two possible approaches,
one is copying our texture into application's own texture,
and the other is drawing on application's own texture directly.
The former (appsink way) cannot be a zero-copy by nature.
In order to support zero-copy processing, we need to draw on
application's own texture directly.
For example, assume that application wants RGBA texture.
Then we can imagine following case.
"d3d11h264dec ! d3d11convert ! video/x-raw(memory:D3D11Memory),format=RGBA ! appsink"
^
|_ allocate new Direct3D texture for RGBA format
In above case, d3d11convert will allocate new texture(s) for RGBA format
and then application will copy again the our RGBA texutre into
application's own texture. One texture allocation plus per frame GPU copy will hanppen
in that case therefore.
Moreover, in order for application to be able to access
our texture, we need to allocate texture with additional flags for
application's Direct3D11 device to be able to read texture data.
That would be another implementation burden on our side
But with this MR, we can configure pipeline in this way
"d3d11h264dec ! d3d11videosink".
In that way, we can save at least one texture allocation and
per frame texutre copy since d3d11videosink will convert incoming texture
into application's texture format directly without copy.
* What if we expose texture without conversion and application does
conversion by itself?
As mentioned above, for application to be able to access our texture
from application's Direct3D11 device, we need to allocate texture
in a special form. But in some case, that might not be possible.
Also, if a texture belongs to decoder DPB, exposing such texture
to application is unsafe and usual Direct3D11 shader cannot handle
such texture. To convert format, ID3D11VideoProcessor API needs to
be used but that would be a implementation burden for application.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1873>
2020-12-23 14:49:12 +00:00
|
|
|
*
|
|
|
|
|
* Currently supported formats for user texture are:
|
|
|
|
|
* - DXGI_FORMAT_R8G8B8A8_UNORM
|
|
|
|
|
* - DXGI_FORMAT_B8G8R8A8_UNORM
|
|
|
|
|
* - DXGI_FORMAT_R10G10B10A2_UNORM
|
|
|
|
|
*
|
|
|
|
|
* Since: 1.20
|
|
|
|
|
*/
|
|
|
|
|
g_object_class_install_property (gobject_class, PROP_DRAW_ON_SHARED_TEXTURE,
|
|
|
|
|
g_param_spec_boolean ("draw-on-shared-texture",
|
|
|
|
|
"Draw on shared texture",
|
|
|
|
|
"Draw on user provided shared texture instead of window. "
|
|
|
|
|
"When enabled, user can pass application's own texture to sink "
|
|
|
|
|
"by using \"draw\" action signal on \"begin-draw\" signal handler, "
|
|
|
|
|
"so that sink can draw video data on application's texture. "
|
|
|
|
|
"Supported texture formats for user texture are "
|
|
|
|
|
"DXGI_FORMAT_R8G8B8A8_UNORM, DXGI_FORMAT_B8G8R8A8_UNORM, and "
|
|
|
|
|
"DXGI_FORMAT_R10G10B10A2_UNORM.",
|
|
|
|
|
DEFAULT_DRAW_ON_SHARED_TEXTURE,
|
2021-03-13 08:40:57 +00:00
|
|
|
(GParamFlags) (G_PARAM_READWRITE | GST_PARAM_MUTABLE_READY |
|
|
|
|
|
G_PARAM_STATIC_STRINGS)));
|
d3d11videosink: Add support for drawing on application's own texture
Add a way to support drawing on application's texture instead of
usual window handle.
To make use of this new feature, application should follow below step.
1) Enable this feature by using "draw-on-shared-texture" property
2) Watch "begin-draw" signal
3) On "begin-draw" signal handler, application can request drawing
by using "draw" signal action. Note that "draw" signal action
should be happen before "begin-draw" signal handler is returned
NOTE 1) For texture sharing, creating a texture with
D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX flag is strongly recommend
if possible because we cannot ensure sync a texture
which was created with D3D11_RESOURCE_MISC_SHARED
and it would cause glitch with ID3D11VideoProcessor use case.
NOTE 2) Direct9Ex doesn't support texture sharing which was
created with D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX. In other words,
D3D11_RESOURCE_MISC_SHARED is the only option for Direct3D11/Direct9Ex interop.
NOTE 3) Because of missing synchronization around ID3D11VideoProcessor,
If shared texture was created with D3D11_RESOURCE_MISC_SHARED,
d3d11videosink might use fallback texture to convert DXVA texture
to normal Direct3D texture. Then converted texture will be
copied to user-provided shared texture.
* Why not use generic appsink approach?
In order for application to be able to store video data
which was produced by GStreamer in application's own texture,
there would be two possible approaches,
one is copying our texture into application's own texture,
and the other is drawing on application's own texture directly.
The former (appsink way) cannot be a zero-copy by nature.
In order to support zero-copy processing, we need to draw on
application's own texture directly.
For example, assume that application wants RGBA texture.
Then we can imagine following case.
"d3d11h264dec ! d3d11convert ! video/x-raw(memory:D3D11Memory),format=RGBA ! appsink"
^
|_ allocate new Direct3D texture for RGBA format
In above case, d3d11convert will allocate new texture(s) for RGBA format
and then application will copy again the our RGBA texutre into
application's own texture. One texture allocation plus per frame GPU copy will hanppen
in that case therefore.
Moreover, in order for application to be able to access
our texture, we need to allocate texture with additional flags for
application's Direct3D11 device to be able to read texture data.
That would be another implementation burden on our side
But with this MR, we can configure pipeline in this way
"d3d11h264dec ! d3d11videosink".
In that way, we can save at least one texture allocation and
per frame texutre copy since d3d11videosink will convert incoming texture
into application's texture format directly without copy.
* What if we expose texture without conversion and application does
conversion by itself?
As mentioned above, for application to be able to access our texture
from application's Direct3D11 device, we need to allocate texture
in a special form. But in some case, that might not be possible.
Also, if a texture belongs to decoder DPB, exposing such texture
to application is unsafe and usual Direct3D11 shader cannot handle
such texture. To convert format, ID3D11VideoProcessor API needs to
be used but that would be a implementation burden for application.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1873>
2020-12-23 14:49:12 +00:00
|
|
|
|
2022-07-02 16:22:10 +00:00
|
|
|
/**
|
|
|
|
|
* GstD3D11VideoSink:rotate-method:
|
|
|
|
|
*
|
|
|
|
|
* Video rotation/flip method to use
|
|
|
|
|
*
|
|
|
|
|
* Since: 1.22
|
|
|
|
|
*/
|
|
|
|
|
g_object_class_install_property (gobject_class, PROP_ROTATE_METHOD,
|
|
|
|
|
g_param_spec_enum ("rotate-method", "Rotate Method",
|
|
|
|
|
"Rotate method to use",
|
|
|
|
|
GST_TYPE_VIDEO_ORIENTATION_METHOD, GST_VIDEO_ORIENTATION_IDENTITY,
|
|
|
|
|
(GParamFlags) (G_PARAM_READWRITE | G_PARAM_STATIC_STRINGS)));
|
|
|
|
|
|
2022-08-02 16:47:46 +00:00
|
|
|
/**
|
|
|
|
|
* GstD3D11VideoSink:gamma-mode:
|
|
|
|
|
*
|
|
|
|
|
* Gamma conversion mode
|
|
|
|
|
*
|
|
|
|
|
* Since: 1.22
|
|
|
|
|
*/
|
|
|
|
|
g_object_class_install_property (gobject_class, PROP_GAMMA_MODE,
|
|
|
|
|
g_param_spec_enum ("gamma-mode", "Gamma mode",
|
|
|
|
|
"Gamma conversion mode", GST_TYPE_VIDEO_GAMMA_MODE,
|
|
|
|
|
DEFAULT_GAMMA_MODE, (GParamFlags) (GST_PARAM_MUTABLE_READY |
|
|
|
|
|
G_PARAM_READWRITE | G_PARAM_STATIC_STRINGS)));
|
|
|
|
|
|
|
|
|
|
/**
|
|
|
|
|
* GstD3D11VideoSink:primaries-mode:
|
|
|
|
|
*
|
|
|
|
|
* Primaries conversion mode
|
|
|
|
|
*
|
|
|
|
|
* Since: 1.22
|
|
|
|
|
*/
|
|
|
|
|
g_object_class_install_property (gobject_class, PROP_PRIMARIES_MODE,
|
|
|
|
|
g_param_spec_enum ("primaries-mode", "Primaries Mode",
|
|
|
|
|
"Primaries conversion mode", GST_TYPE_VIDEO_PRIMARIES_MODE,
|
|
|
|
|
DEFAULT_PRIMARIES_MODE, (GParamFlags) (GST_PARAM_MUTABLE_READY |
|
|
|
|
|
G_PARAM_READWRITE | G_PARAM_STATIC_STRINGS)));
|
|
|
|
|
|
2022-08-23 15:13:21 +00:00
|
|
|
/**
|
|
|
|
|
* GstD3D11VideoSink:display-format:
|
|
|
|
|
*
|
|
|
|
|
* Swapchain display format
|
|
|
|
|
*
|
|
|
|
|
* Since: 1.22
|
|
|
|
|
*/
|
|
|
|
|
g_object_class_install_property (gobject_class, PROP_DISPLAY_FORMAT,
|
|
|
|
|
g_param_spec_enum ("display-format", "Display Format",
|
|
|
|
|
"Swapchain display format", GST_TYPE_D3D11_VIDEO_SINK_DISPLAY_FORMAT,
|
|
|
|
|
DEFAULT_DISPLAY_FORMAT, (GParamFlags) (G_PARAM_READWRITE |
|
|
|
|
|
GST_PARAM_MUTABLE_READY | G_PARAM_STATIC_STRINGS)));
|
|
|
|
|
|
2022-10-13 15:17:52 +00:00
|
|
|
/**
|
|
|
|
|
* GstD3D11VideoSink:emit-present:
|
|
|
|
|
*
|
|
|
|
|
* Emits "present" signal
|
|
|
|
|
*
|
|
|
|
|
* Since: 1.22
|
|
|
|
|
*/
|
|
|
|
|
g_object_class_install_property (gobject_class, PROP_EMIT_PRESENT,
|
|
|
|
|
g_param_spec_boolean ("emit-present", "Emit present",
|
|
|
|
|
"Emits present signal", DEFAULT_EMIT_PRESENT,
|
|
|
|
|
(GParamFlags) (G_PARAM_READWRITE | GST_PARAM_MUTABLE_READY |
|
|
|
|
|
G_PARAM_STATIC_STRINGS)));
|
|
|
|
|
|
2023-05-14 11:05:25 +00:00
|
|
|
/**
|
|
|
|
|
* GstD3D11VideoSink:render-rectangle:
|
|
|
|
|
*
|
|
|
|
|
* Since: 1.24
|
|
|
|
|
*/
|
|
|
|
|
gst_video_overlay_install_properties (gobject_class, PROP_RENDER_RECTANGE);
|
|
|
|
|
|
d3d11videosink: Add support for drawing on application's own texture
Add a way to support drawing on application's texture instead of
usual window handle.
To make use of this new feature, application should follow below step.
1) Enable this feature by using "draw-on-shared-texture" property
2) Watch "begin-draw" signal
3) On "begin-draw" signal handler, application can request drawing
by using "draw" signal action. Note that "draw" signal action
should be happen before "begin-draw" signal handler is returned
NOTE 1) For texture sharing, creating a texture with
D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX flag is strongly recommend
if possible because we cannot ensure sync a texture
which was created with D3D11_RESOURCE_MISC_SHARED
and it would cause glitch with ID3D11VideoProcessor use case.
NOTE 2) Direct9Ex doesn't support texture sharing which was
created with D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX. In other words,
D3D11_RESOURCE_MISC_SHARED is the only option for Direct3D11/Direct9Ex interop.
NOTE 3) Because of missing synchronization around ID3D11VideoProcessor,
If shared texture was created with D3D11_RESOURCE_MISC_SHARED,
d3d11videosink might use fallback texture to convert DXVA texture
to normal Direct3D texture. Then converted texture will be
copied to user-provided shared texture.
* Why not use generic appsink approach?
In order for application to be able to store video data
which was produced by GStreamer in application's own texture,
there would be two possible approaches,
one is copying our texture into application's own texture,
and the other is drawing on application's own texture directly.
The former (appsink way) cannot be a zero-copy by nature.
In order to support zero-copy processing, we need to draw on
application's own texture directly.
For example, assume that application wants RGBA texture.
Then we can imagine following case.
"d3d11h264dec ! d3d11convert ! video/x-raw(memory:D3D11Memory),format=RGBA ! appsink"
^
|_ allocate new Direct3D texture for RGBA format
In above case, d3d11convert will allocate new texture(s) for RGBA format
and then application will copy again the our RGBA texutre into
application's own texture. One texture allocation plus per frame GPU copy will hanppen
in that case therefore.
Moreover, in order for application to be able to access
our texture, we need to allocate texture with additional flags for
application's Direct3D11 device to be able to read texture data.
That would be another implementation burden on our side
But with this MR, we can configure pipeline in this way
"d3d11h264dec ! d3d11videosink".
In that way, we can save at least one texture allocation and
per frame texutre copy since d3d11videosink will convert incoming texture
into application's texture format directly without copy.
* What if we expose texture without conversion and application does
conversion by itself?
As mentioned above, for application to be able to access our texture
from application's Direct3D11 device, we need to allocate texture
in a special form. But in some case, that might not be possible.
Also, if a texture belongs to decoder DPB, exposing such texture
to application is unsafe and usual Direct3D11 shader cannot handle
such texture. To convert format, ID3D11VideoProcessor API needs to
be used but that would be a implementation burden for application.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1873>
2020-12-23 14:49:12 +00:00
|
|
|
/**
|
2021-02-21 08:38:38 +00:00
|
|
|
* GstD3D11VideoSink::begin-draw:
|
2021-04-17 11:37:13 +00:00
|
|
|
* @videosink: the #d3d11videosink
|
d3d11videosink: Add support for drawing on application's own texture
Add a way to support drawing on application's texture instead of
usual window handle.
To make use of this new feature, application should follow below step.
1) Enable this feature by using "draw-on-shared-texture" property
2) Watch "begin-draw" signal
3) On "begin-draw" signal handler, application can request drawing
by using "draw" signal action. Note that "draw" signal action
should be happen before "begin-draw" signal handler is returned
NOTE 1) For texture sharing, creating a texture with
D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX flag is strongly recommend
if possible because we cannot ensure sync a texture
which was created with D3D11_RESOURCE_MISC_SHARED
and it would cause glitch with ID3D11VideoProcessor use case.
NOTE 2) Direct9Ex doesn't support texture sharing which was
created with D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX. In other words,
D3D11_RESOURCE_MISC_SHARED is the only option for Direct3D11/Direct9Ex interop.
NOTE 3) Because of missing synchronization around ID3D11VideoProcessor,
If shared texture was created with D3D11_RESOURCE_MISC_SHARED,
d3d11videosink might use fallback texture to convert DXVA texture
to normal Direct3D texture. Then converted texture will be
copied to user-provided shared texture.
* Why not use generic appsink approach?
In order for application to be able to store video data
which was produced by GStreamer in application's own texture,
there would be two possible approaches,
one is copying our texture into application's own texture,
and the other is drawing on application's own texture directly.
The former (appsink way) cannot be a zero-copy by nature.
In order to support zero-copy processing, we need to draw on
application's own texture directly.
For example, assume that application wants RGBA texture.
Then we can imagine following case.
"d3d11h264dec ! d3d11convert ! video/x-raw(memory:D3D11Memory),format=RGBA ! appsink"
^
|_ allocate new Direct3D texture for RGBA format
In above case, d3d11convert will allocate new texture(s) for RGBA format
and then application will copy again the our RGBA texutre into
application's own texture. One texture allocation plus per frame GPU copy will hanppen
in that case therefore.
Moreover, in order for application to be able to access
our texture, we need to allocate texture with additional flags for
application's Direct3D11 device to be able to read texture data.
That would be another implementation burden on our side
But with this MR, we can configure pipeline in this way
"d3d11h264dec ! d3d11videosink".
In that way, we can save at least one texture allocation and
per frame texutre copy since d3d11videosink will convert incoming texture
into application's texture format directly without copy.
* What if we expose texture without conversion and application does
conversion by itself?
As mentioned above, for application to be able to access our texture
from application's Direct3D11 device, we need to allocate texture
in a special form. But in some case, that might not be possible.
Also, if a texture belongs to decoder DPB, exposing such texture
to application is unsafe and usual Direct3D11 shader cannot handle
such texture. To convert format, ID3D11VideoProcessor API needs to
be used but that would be a implementation burden for application.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1873>
2020-12-23 14:49:12 +00:00
|
|
|
*
|
2021-02-21 08:38:38 +00:00
|
|
|
* Emitted when sink has a texture to draw. Application needs to invoke
|
2021-04-17 11:37:13 +00:00
|
|
|
* #d3d11videosink::draw action signal before returning from
|
|
|
|
|
* #d3d11videosink::begin-draw signal handler.
|
d3d11videosink: Add support for drawing on application's own texture
Add a way to support drawing on application's texture instead of
usual window handle.
To make use of this new feature, application should follow below step.
1) Enable this feature by using "draw-on-shared-texture" property
2) Watch "begin-draw" signal
3) On "begin-draw" signal handler, application can request drawing
by using "draw" signal action. Note that "draw" signal action
should be happen before "begin-draw" signal handler is returned
NOTE 1) For texture sharing, creating a texture with
D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX flag is strongly recommend
if possible because we cannot ensure sync a texture
which was created with D3D11_RESOURCE_MISC_SHARED
and it would cause glitch with ID3D11VideoProcessor use case.
NOTE 2) Direct9Ex doesn't support texture sharing which was
created with D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX. In other words,
D3D11_RESOURCE_MISC_SHARED is the only option for Direct3D11/Direct9Ex interop.
NOTE 3) Because of missing synchronization around ID3D11VideoProcessor,
If shared texture was created with D3D11_RESOURCE_MISC_SHARED,
d3d11videosink might use fallback texture to convert DXVA texture
to normal Direct3D texture. Then converted texture will be
copied to user-provided shared texture.
* Why not use generic appsink approach?
In order for application to be able to store video data
which was produced by GStreamer in application's own texture,
there would be two possible approaches,
one is copying our texture into application's own texture,
and the other is drawing on application's own texture directly.
The former (appsink way) cannot be a zero-copy by nature.
In order to support zero-copy processing, we need to draw on
application's own texture directly.
For example, assume that application wants RGBA texture.
Then we can imagine following case.
"d3d11h264dec ! d3d11convert ! video/x-raw(memory:D3D11Memory),format=RGBA ! appsink"
^
|_ allocate new Direct3D texture for RGBA format
In above case, d3d11convert will allocate new texture(s) for RGBA format
and then application will copy again the our RGBA texutre into
application's own texture. One texture allocation plus per frame GPU copy will hanppen
in that case therefore.
Moreover, in order for application to be able to access
our texture, we need to allocate texture with additional flags for
application's Direct3D11 device to be able to read texture data.
That would be another implementation burden on our side
But with this MR, we can configure pipeline in this way
"d3d11h264dec ! d3d11videosink".
In that way, we can save at least one texture allocation and
per frame texutre copy since d3d11videosink will convert incoming texture
into application's texture format directly without copy.
* What if we expose texture without conversion and application does
conversion by itself?
As mentioned above, for application to be able to access our texture
from application's Direct3D11 device, we need to allocate texture
in a special form. But in some case, that might not be possible.
Also, if a texture belongs to decoder DPB, exposing such texture
to application is unsafe and usual Direct3D11 shader cannot handle
such texture. To convert format, ID3D11VideoProcessor API needs to
be used but that would be a implementation burden for application.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1873>
2020-12-23 14:49:12 +00:00
|
|
|
*
|
|
|
|
|
* Since: 1.20
|
|
|
|
|
*/
|
|
|
|
|
gst_d3d11_video_sink_signals[SIGNAL_BEGIN_DRAW] =
|
|
|
|
|
g_signal_new ("begin-draw", G_TYPE_FROM_CLASS (klass), G_SIGNAL_RUN_LAST,
|
|
|
|
|
G_STRUCT_OFFSET (GstD3D11VideoSinkClass, begin_draw),
|
|
|
|
|
NULL, NULL, NULL, G_TYPE_NONE, 0, G_TYPE_NONE);
|
|
|
|
|
|
|
|
|
|
/**
|
2021-02-21 08:38:38 +00:00
|
|
|
* GstD3D11VideoSink::draw:
|
2021-04-17 11:37:13 +00:00
|
|
|
* @videosink: the #d3d11videosink
|
d3d11videosink: Add support for drawing on application's own texture
Add a way to support drawing on application's texture instead of
usual window handle.
To make use of this new feature, application should follow below step.
1) Enable this feature by using "draw-on-shared-texture" property
2) Watch "begin-draw" signal
3) On "begin-draw" signal handler, application can request drawing
by using "draw" signal action. Note that "draw" signal action
should be happen before "begin-draw" signal handler is returned
NOTE 1) For texture sharing, creating a texture with
D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX flag is strongly recommend
if possible because we cannot ensure sync a texture
which was created with D3D11_RESOURCE_MISC_SHARED
and it would cause glitch with ID3D11VideoProcessor use case.
NOTE 2) Direct9Ex doesn't support texture sharing which was
created with D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX. In other words,
D3D11_RESOURCE_MISC_SHARED is the only option for Direct3D11/Direct9Ex interop.
NOTE 3) Because of missing synchronization around ID3D11VideoProcessor,
If shared texture was created with D3D11_RESOURCE_MISC_SHARED,
d3d11videosink might use fallback texture to convert DXVA texture
to normal Direct3D texture. Then converted texture will be
copied to user-provided shared texture.
* Why not use generic appsink approach?
In order for application to be able to store video data
which was produced by GStreamer in application's own texture,
there would be two possible approaches,
one is copying our texture into application's own texture,
and the other is drawing on application's own texture directly.
The former (appsink way) cannot be a zero-copy by nature.
In order to support zero-copy processing, we need to draw on
application's own texture directly.
For example, assume that application wants RGBA texture.
Then we can imagine following case.
"d3d11h264dec ! d3d11convert ! video/x-raw(memory:D3D11Memory),format=RGBA ! appsink"
^
|_ allocate new Direct3D texture for RGBA format
In above case, d3d11convert will allocate new texture(s) for RGBA format
and then application will copy again the our RGBA texutre into
application's own texture. One texture allocation plus per frame GPU copy will hanppen
in that case therefore.
Moreover, in order for application to be able to access
our texture, we need to allocate texture with additional flags for
application's Direct3D11 device to be able to read texture data.
That would be another implementation burden on our side
But with this MR, we can configure pipeline in this way
"d3d11h264dec ! d3d11videosink".
In that way, we can save at least one texture allocation and
per frame texutre copy since d3d11videosink will convert incoming texture
into application's texture format directly without copy.
* What if we expose texture without conversion and application does
conversion by itself?
As mentioned above, for application to be able to access our texture
from application's Direct3D11 device, we need to allocate texture
in a special form. But in some case, that might not be possible.
Also, if a texture belongs to decoder DPB, exposing such texture
to application is unsafe and usual Direct3D11 shader cannot handle
such texture. To convert format, ID3D11VideoProcessor API needs to
be used but that would be a implementation burden for application.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1873>
2020-12-23 14:49:12 +00:00
|
|
|
* @shard_handle: a pointer to HANDLE
|
|
|
|
|
* @texture_misc_flags: a D3D11_RESOURCE_MISC_FLAG value
|
|
|
|
|
* @acquire_key: a key value used for IDXGIKeyedMutex::AcquireSync
|
|
|
|
|
* @release_key: a key value used for IDXGIKeyedMutex::ReleaseSync
|
|
|
|
|
*
|
2021-02-21 08:38:38 +00:00
|
|
|
* Draws on a shared texture. @shard_handle must be a valid pointer to
|
|
|
|
|
* a HANDLE which was obtained via IDXGIResource::GetSharedHandle or
|
d3d11videosink: Add support for drawing on application's own texture
Add a way to support drawing on application's texture instead of
usual window handle.
To make use of this new feature, application should follow below step.
1) Enable this feature by using "draw-on-shared-texture" property
2) Watch "begin-draw" signal
3) On "begin-draw" signal handler, application can request drawing
by using "draw" signal action. Note that "draw" signal action
should be happen before "begin-draw" signal handler is returned
NOTE 1) For texture sharing, creating a texture with
D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX flag is strongly recommend
if possible because we cannot ensure sync a texture
which was created with D3D11_RESOURCE_MISC_SHARED
and it would cause glitch with ID3D11VideoProcessor use case.
NOTE 2) Direct9Ex doesn't support texture sharing which was
created with D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX. In other words,
D3D11_RESOURCE_MISC_SHARED is the only option for Direct3D11/Direct9Ex interop.
NOTE 3) Because of missing synchronization around ID3D11VideoProcessor,
If shared texture was created with D3D11_RESOURCE_MISC_SHARED,
d3d11videosink might use fallback texture to convert DXVA texture
to normal Direct3D texture. Then converted texture will be
copied to user-provided shared texture.
* Why not use generic appsink approach?
In order for application to be able to store video data
which was produced by GStreamer in application's own texture,
there would be two possible approaches,
one is copying our texture into application's own texture,
and the other is drawing on application's own texture directly.
The former (appsink way) cannot be a zero-copy by nature.
In order to support zero-copy processing, we need to draw on
application's own texture directly.
For example, assume that application wants RGBA texture.
Then we can imagine following case.
"d3d11h264dec ! d3d11convert ! video/x-raw(memory:D3D11Memory),format=RGBA ! appsink"
^
|_ allocate new Direct3D texture for RGBA format
In above case, d3d11convert will allocate new texture(s) for RGBA format
and then application will copy again the our RGBA texutre into
application's own texture. One texture allocation plus per frame GPU copy will hanppen
in that case therefore.
Moreover, in order for application to be able to access
our texture, we need to allocate texture with additional flags for
application's Direct3D11 device to be able to read texture data.
That would be another implementation burden on our side
But with this MR, we can configure pipeline in this way
"d3d11h264dec ! d3d11videosink".
In that way, we can save at least one texture allocation and
per frame texutre copy since d3d11videosink will convert incoming texture
into application's texture format directly without copy.
* What if we expose texture without conversion and application does
conversion by itself?
As mentioned above, for application to be able to access our texture
from application's Direct3D11 device, we need to allocate texture
in a special form. But in some case, that might not be possible.
Also, if a texture belongs to decoder DPB, exposing such texture
to application is unsafe and usual Direct3D11 shader cannot handle
such texture. To convert format, ID3D11VideoProcessor API needs to
be used but that would be a implementation burden for application.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1873>
2020-12-23 14:49:12 +00:00
|
|
|
* IDXGIResource1::CreateSharedHandle.
|
|
|
|
|
*
|
2021-02-21 08:38:38 +00:00
|
|
|
* If the texture was created with D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX flag,
|
d3d11videosink: Add support for drawing on application's own texture
Add a way to support drawing on application's texture instead of
usual window handle.
To make use of this new feature, application should follow below step.
1) Enable this feature by using "draw-on-shared-texture" property
2) Watch "begin-draw" signal
3) On "begin-draw" signal handler, application can request drawing
by using "draw" signal action. Note that "draw" signal action
should be happen before "begin-draw" signal handler is returned
NOTE 1) For texture sharing, creating a texture with
D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX flag is strongly recommend
if possible because we cannot ensure sync a texture
which was created with D3D11_RESOURCE_MISC_SHARED
and it would cause glitch with ID3D11VideoProcessor use case.
NOTE 2) Direct9Ex doesn't support texture sharing which was
created with D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX. In other words,
D3D11_RESOURCE_MISC_SHARED is the only option for Direct3D11/Direct9Ex interop.
NOTE 3) Because of missing synchronization around ID3D11VideoProcessor,
If shared texture was created with D3D11_RESOURCE_MISC_SHARED,
d3d11videosink might use fallback texture to convert DXVA texture
to normal Direct3D texture. Then converted texture will be
copied to user-provided shared texture.
* Why not use generic appsink approach?
In order for application to be able to store video data
which was produced by GStreamer in application's own texture,
there would be two possible approaches,
one is copying our texture into application's own texture,
and the other is drawing on application's own texture directly.
The former (appsink way) cannot be a zero-copy by nature.
In order to support zero-copy processing, we need to draw on
application's own texture directly.
For example, assume that application wants RGBA texture.
Then we can imagine following case.
"d3d11h264dec ! d3d11convert ! video/x-raw(memory:D3D11Memory),format=RGBA ! appsink"
^
|_ allocate new Direct3D texture for RGBA format
In above case, d3d11convert will allocate new texture(s) for RGBA format
and then application will copy again the our RGBA texutre into
application's own texture. One texture allocation plus per frame GPU copy will hanppen
in that case therefore.
Moreover, in order for application to be able to access
our texture, we need to allocate texture with additional flags for
application's Direct3D11 device to be able to read texture data.
That would be another implementation burden on our side
But with this MR, we can configure pipeline in this way
"d3d11h264dec ! d3d11videosink".
In that way, we can save at least one texture allocation and
per frame texutre copy since d3d11videosink will convert incoming texture
into application's texture format directly without copy.
* What if we expose texture without conversion and application does
conversion by itself?
As mentioned above, for application to be able to access our texture
from application's Direct3D11 device, we need to allocate texture
in a special form. But in some case, that might not be possible.
Also, if a texture belongs to decoder DPB, exposing such texture
to application is unsafe and usual Direct3D11 shader cannot handle
such texture. To convert format, ID3D11VideoProcessor API needs to
be used but that would be a implementation burden for application.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1873>
2020-12-23 14:49:12 +00:00
|
|
|
* caller must specify valid @acquire_key and @release_key.
|
|
|
|
|
* Otherwise (i.e., created with D3D11_RESOURCE_MISC_SHARED flag),
|
|
|
|
|
* @acquire_key and @release_key will be ignored.
|
|
|
|
|
*
|
|
|
|
|
* Since: 1.20
|
|
|
|
|
*/
|
|
|
|
|
gst_d3d11_video_sink_signals[SIGNAL_DRAW] =
|
|
|
|
|
g_signal_new ("draw", G_TYPE_FROM_CLASS (klass),
|
2021-03-13 08:40:57 +00:00
|
|
|
(GSignalFlags) (G_SIGNAL_RUN_LAST | G_SIGNAL_ACTION),
|
d3d11videosink: Add support for drawing on application's own texture
Add a way to support drawing on application's texture instead of
usual window handle.
To make use of this new feature, application should follow below step.
1) Enable this feature by using "draw-on-shared-texture" property
2) Watch "begin-draw" signal
3) On "begin-draw" signal handler, application can request drawing
by using "draw" signal action. Note that "draw" signal action
should be happen before "begin-draw" signal handler is returned
NOTE 1) For texture sharing, creating a texture with
D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX flag is strongly recommend
if possible because we cannot ensure sync a texture
which was created with D3D11_RESOURCE_MISC_SHARED
and it would cause glitch with ID3D11VideoProcessor use case.
NOTE 2) Direct9Ex doesn't support texture sharing which was
created with D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX. In other words,
D3D11_RESOURCE_MISC_SHARED is the only option for Direct3D11/Direct9Ex interop.
NOTE 3) Because of missing synchronization around ID3D11VideoProcessor,
If shared texture was created with D3D11_RESOURCE_MISC_SHARED,
d3d11videosink might use fallback texture to convert DXVA texture
to normal Direct3D texture. Then converted texture will be
copied to user-provided shared texture.
* Why not use generic appsink approach?
In order for application to be able to store video data
which was produced by GStreamer in application's own texture,
there would be two possible approaches,
one is copying our texture into application's own texture,
and the other is drawing on application's own texture directly.
The former (appsink way) cannot be a zero-copy by nature.
In order to support zero-copy processing, we need to draw on
application's own texture directly.
For example, assume that application wants RGBA texture.
Then we can imagine following case.
"d3d11h264dec ! d3d11convert ! video/x-raw(memory:D3D11Memory),format=RGBA ! appsink"
^
|_ allocate new Direct3D texture for RGBA format
In above case, d3d11convert will allocate new texture(s) for RGBA format
and then application will copy again the our RGBA texutre into
application's own texture. One texture allocation plus per frame GPU copy will hanppen
in that case therefore.
Moreover, in order for application to be able to access
our texture, we need to allocate texture with additional flags for
application's Direct3D11 device to be able to read texture data.
That would be another implementation burden on our side
But with this MR, we can configure pipeline in this way
"d3d11h264dec ! d3d11videosink".
In that way, we can save at least one texture allocation and
per frame texutre copy since d3d11videosink will convert incoming texture
into application's texture format directly without copy.
* What if we expose texture without conversion and application does
conversion by itself?
As mentioned above, for application to be able to access our texture
from application's Direct3D11 device, we need to allocate texture
in a special form. But in some case, that might not be possible.
Also, if a texture belongs to decoder DPB, exposing such texture
to application is unsafe and usual Direct3D11 shader cannot handle
such texture. To convert format, ID3D11VideoProcessor API needs to
be used but that would be a implementation burden for application.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1873>
2020-12-23 14:49:12 +00:00
|
|
|
G_STRUCT_OFFSET (GstD3D11VideoSinkClass, draw), NULL, NULL, NULL,
|
|
|
|
|
G_TYPE_BOOLEAN, 4, G_TYPE_POINTER, G_TYPE_UINT, G_TYPE_UINT64,
|
|
|
|
|
G_TYPE_UINT64);
|
|
|
|
|
|
2022-08-19 11:25:31 +00:00
|
|
|
/**
|
|
|
|
|
* GstD3D11VideoSink::present
|
|
|
|
|
* @videosink: the #GstD3D11VideoSink
|
2022-10-21 17:49:00 +00:00
|
|
|
* @device: a GstD3D11Device object
|
2022-08-19 11:25:31 +00:00
|
|
|
* @render_target: a ID3D11RenderTargetView handle of swapchain's backbuffer
|
|
|
|
|
*
|
|
|
|
|
* Emitted just before presenting a texture via the IDXGISwapChain::Present.
|
|
|
|
|
* The client can perform additional rendering on the given @render_target,
|
|
|
|
|
* or can read the content already rendered on the swapchain's backbuffer.
|
|
|
|
|
*
|
2022-10-21 17:49:00 +00:00
|
|
|
* This signal will be emitted with gst_d3d11_device_lock taken and
|
2022-08-19 11:25:31 +00:00
|
|
|
* client should perform GPU operation from the thread where this signal
|
|
|
|
|
* emitted.
|
|
|
|
|
*
|
|
|
|
|
* Since: 1.22
|
|
|
|
|
*/
|
|
|
|
|
gst_d3d11_video_sink_signals[SIGNAL_PRESENT] =
|
|
|
|
|
g_signal_new ("present", G_TYPE_FROM_CLASS (klass),
|
|
|
|
|
G_SIGNAL_RUN_LAST, 0, nullptr, nullptr, nullptr,
|
2022-10-21 17:49:00 +00:00
|
|
|
G_TYPE_NONE, 2, GST_TYPE_OBJECT, G_TYPE_POINTER);
|
2022-08-19 11:25:31 +00:00
|
|
|
|
2019-01-30 11:07:29 +00:00
|
|
|
element_class->set_context =
|
|
|
|
|
GST_DEBUG_FUNCPTR (gst_d3d11_video_sink_set_context);
|
|
|
|
|
|
|
|
|
|
gst_element_class_set_static_metadata (element_class,
|
2023-05-12 13:18:22 +00:00
|
|
|
"Direct3D11 Video Sink", "Sink/Video",
|
2019-01-30 11:07:29 +00:00
|
|
|
"A Direct3D11 based videosink",
|
|
|
|
|
"Seungha Yang <seungha.yang@navercorp.com>");
|
|
|
|
|
|
2021-01-19 17:29:43 +00:00
|
|
|
caps = gst_d3d11_get_updated_template_caps (&pad_template_caps);
|
|
|
|
|
gst_element_class_add_pad_template (element_class,
|
|
|
|
|
gst_pad_template_new ("sink", GST_PAD_SINK, GST_PAD_ALWAYS, caps));
|
|
|
|
|
gst_caps_unref (caps);
|
2019-01-30 11:07:29 +00:00
|
|
|
|
|
|
|
|
basesink_class->get_caps = GST_DEBUG_FUNCPTR (gst_d3d11_video_sink_get_caps);
|
|
|
|
|
basesink_class->set_caps = GST_DEBUG_FUNCPTR (gst_d3d11_video_sink_set_caps);
|
|
|
|
|
basesink_class->start = GST_DEBUG_FUNCPTR (gst_d3d11_video_sink_start);
|
|
|
|
|
basesink_class->stop = GST_DEBUG_FUNCPTR (gst_d3d11_video_sink_stop);
|
|
|
|
|
basesink_class->propose_allocation =
|
|
|
|
|
GST_DEBUG_FUNCPTR (gst_d3d11_video_sink_propose_allocation);
|
2019-07-18 14:12:13 +00:00
|
|
|
basesink_class->query = GST_DEBUG_FUNCPTR (gst_d3d11_video_sink_query);
|
2019-12-02 14:27:42 +00:00
|
|
|
basesink_class->unlock = GST_DEBUG_FUNCPTR (gst_d3d11_video_sink_unlock);
|
2019-12-30 09:58:59 +00:00
|
|
|
basesink_class->unlock_stop =
|
|
|
|
|
GST_DEBUG_FUNCPTR (gst_d3d11_video_sink_unlock_stop);
|
2021-09-17 17:27:51 +00:00
|
|
|
basesink_class->event = GST_DEBUG_FUNCPTR (gst_d3d11_video_sink_event);
|
2022-12-15 17:52:08 +00:00
|
|
|
basesink_class->prepare = GST_DEBUG_FUNCPTR (gst_d3d11_video_sink_prepare);
|
2019-01-30 11:07:29 +00:00
|
|
|
|
|
|
|
|
videosink_class->show_frame =
|
|
|
|
|
GST_DEBUG_FUNCPTR (gst_d3d11_video_sink_show_frame);
|
2020-06-08 15:20:08 +00:00
|
|
|
|
d3d11videosink: Add support for drawing on application's own texture
Add a way to support drawing on application's texture instead of
usual window handle.
To make use of this new feature, application should follow below step.
1) Enable this feature by using "draw-on-shared-texture" property
2) Watch "begin-draw" signal
3) On "begin-draw" signal handler, application can request drawing
by using "draw" signal action. Note that "draw" signal action
should be happen before "begin-draw" signal handler is returned
NOTE 1) For texture sharing, creating a texture with
D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX flag is strongly recommend
if possible because we cannot ensure sync a texture
which was created with D3D11_RESOURCE_MISC_SHARED
and it would cause glitch with ID3D11VideoProcessor use case.
NOTE 2) Direct9Ex doesn't support texture sharing which was
created with D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX. In other words,
D3D11_RESOURCE_MISC_SHARED is the only option for Direct3D11/Direct9Ex interop.
NOTE 3) Because of missing synchronization around ID3D11VideoProcessor,
If shared texture was created with D3D11_RESOURCE_MISC_SHARED,
d3d11videosink might use fallback texture to convert DXVA texture
to normal Direct3D texture. Then converted texture will be
copied to user-provided shared texture.
* Why not use generic appsink approach?
In order for application to be able to store video data
which was produced by GStreamer in application's own texture,
there would be two possible approaches,
one is copying our texture into application's own texture,
and the other is drawing on application's own texture directly.
The former (appsink way) cannot be a zero-copy by nature.
In order to support zero-copy processing, we need to draw on
application's own texture directly.
For example, assume that application wants RGBA texture.
Then we can imagine following case.
"d3d11h264dec ! d3d11convert ! video/x-raw(memory:D3D11Memory),format=RGBA ! appsink"
^
|_ allocate new Direct3D texture for RGBA format
In above case, d3d11convert will allocate new texture(s) for RGBA format
and then application will copy again the our RGBA texutre into
application's own texture. One texture allocation plus per frame GPU copy will hanppen
in that case therefore.
Moreover, in order for application to be able to access
our texture, we need to allocate texture with additional flags for
application's Direct3D11 device to be able to read texture data.
That would be another implementation burden on our side
But with this MR, we can configure pipeline in this way
"d3d11h264dec ! d3d11videosink".
In that way, we can save at least one texture allocation and
per frame texutre copy since d3d11videosink will convert incoming texture
into application's texture format directly without copy.
* What if we expose texture without conversion and application does
conversion by itself?
As mentioned above, for application to be able to access our texture
from application's Direct3D11 device, we need to allocate texture
in a special form. But in some case, that might not be possible.
Also, if a texture belongs to decoder DPB, exposing such texture
to application is unsafe and usual Direct3D11 shader cannot handle
such texture. To convert format, ID3D11VideoProcessor API needs to
be used but that would be a implementation burden for application.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1873>
2020-12-23 14:49:12 +00:00
|
|
|
klass->draw = gst_d3d11_video_sink_draw_action;
|
|
|
|
|
|
2021-03-13 08:40:57 +00:00
|
|
|
gst_type_mark_as_plugin_api (GST_D3D11_WINDOW_TOGGLE_MODE_GET_TYPE,
|
|
|
|
|
(GstPluginAPIFlags) 0);
|
2022-08-23 15:13:21 +00:00
|
|
|
gst_type_mark_as_plugin_api (GST_TYPE_D3D11_VIDEO_SINK_DISPLAY_FORMAT,
|
|
|
|
|
(GstPluginAPIFlags) 0);
|
2019-01-30 11:07:29 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
static void
|
|
|
|
|
gst_d3d11_video_sink_init (GstD3D11VideoSink * self)
|
|
|
|
|
{
|
|
|
|
|
self->adapter = DEFAULT_ADAPTER;
|
|
|
|
|
self->force_aspect_ratio = DEFAULT_FORCE_ASPECT_RATIO;
|
|
|
|
|
self->enable_navigation_events = DEFAULT_ENABLE_NAVIGATION_EVENTS;
|
2019-12-15 10:29:10 +00:00
|
|
|
self->fullscreen_toggle_mode = DEFAULT_FULLSCREEN_TOGGLE_MODE;
|
|
|
|
|
self->fullscreen = DEFAULT_FULLSCREEN;
|
d3d11videosink: Add support for drawing on application's own texture
Add a way to support drawing on application's texture instead of
usual window handle.
To make use of this new feature, application should follow below step.
1) Enable this feature by using "draw-on-shared-texture" property
2) Watch "begin-draw" signal
3) On "begin-draw" signal handler, application can request drawing
by using "draw" signal action. Note that "draw" signal action
should be happen before "begin-draw" signal handler is returned
NOTE 1) For texture sharing, creating a texture with
D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX flag is strongly recommend
if possible because we cannot ensure sync a texture
which was created with D3D11_RESOURCE_MISC_SHARED
and it would cause glitch with ID3D11VideoProcessor use case.
NOTE 2) Direct9Ex doesn't support texture sharing which was
created with D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX. In other words,
D3D11_RESOURCE_MISC_SHARED is the only option for Direct3D11/Direct9Ex interop.
NOTE 3) Because of missing synchronization around ID3D11VideoProcessor,
If shared texture was created with D3D11_RESOURCE_MISC_SHARED,
d3d11videosink might use fallback texture to convert DXVA texture
to normal Direct3D texture. Then converted texture will be
copied to user-provided shared texture.
* Why not use generic appsink approach?
In order for application to be able to store video data
which was produced by GStreamer in application's own texture,
there would be two possible approaches,
one is copying our texture into application's own texture,
and the other is drawing on application's own texture directly.
The former (appsink way) cannot be a zero-copy by nature.
In order to support zero-copy processing, we need to draw on
application's own texture directly.
For example, assume that application wants RGBA texture.
Then we can imagine following case.
"d3d11h264dec ! d3d11convert ! video/x-raw(memory:D3D11Memory),format=RGBA ! appsink"
^
|_ allocate new Direct3D texture for RGBA format
In above case, d3d11convert will allocate new texture(s) for RGBA format
and then application will copy again the our RGBA texutre into
application's own texture. One texture allocation plus per frame GPU copy will hanppen
in that case therefore.
Moreover, in order for application to be able to access
our texture, we need to allocate texture with additional flags for
application's Direct3D11 device to be able to read texture data.
That would be another implementation burden on our side
But with this MR, we can configure pipeline in this way
"d3d11h264dec ! d3d11videosink".
In that way, we can save at least one texture allocation and
per frame texutre copy since d3d11videosink will convert incoming texture
into application's texture format directly without copy.
* What if we expose texture without conversion and application does
conversion by itself?
As mentioned above, for application to be able to access our texture
from application's Direct3D11 device, we need to allocate texture
in a special form. But in some case, that might not be possible.
Also, if a texture belongs to decoder DPB, exposing such texture
to application is unsafe and usual Direct3D11 shader cannot handle
such texture. To convert format, ID3D11VideoProcessor API needs to
be used but that would be a implementation burden for application.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1873>
2020-12-23 14:49:12 +00:00
|
|
|
self->draw_on_shared_texture = DEFAULT_DRAW_ON_SHARED_TEXTURE;
|
2022-08-02 16:47:46 +00:00
|
|
|
self->gamma_mode = DEFAULT_GAMMA_MODE;
|
|
|
|
|
self->primaries_mode = DEFAULT_PRIMARIES_MODE;
|
2022-08-23 15:13:21 +00:00
|
|
|
self->display_format = DEFAULT_DISPLAY_FORMAT;
|
2022-10-13 15:17:52 +00:00
|
|
|
self->emit_present = DEFAULT_EMIT_PRESENT;
|
d3d11videosink: Add support for drawing on application's own texture
Add a way to support drawing on application's texture instead of
usual window handle.
To make use of this new feature, application should follow below step.
1) Enable this feature by using "draw-on-shared-texture" property
2) Watch "begin-draw" signal
3) On "begin-draw" signal handler, application can request drawing
by using "draw" signal action. Note that "draw" signal action
should be happen before "begin-draw" signal handler is returned
NOTE 1) For texture sharing, creating a texture with
D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX flag is strongly recommend
if possible because we cannot ensure sync a texture
which was created with D3D11_RESOURCE_MISC_SHARED
and it would cause glitch with ID3D11VideoProcessor use case.
NOTE 2) Direct9Ex doesn't support texture sharing which was
created with D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX. In other words,
D3D11_RESOURCE_MISC_SHARED is the only option for Direct3D11/Direct9Ex interop.
NOTE 3) Because of missing synchronization around ID3D11VideoProcessor,
If shared texture was created with D3D11_RESOURCE_MISC_SHARED,
d3d11videosink might use fallback texture to convert DXVA texture
to normal Direct3D texture. Then converted texture will be
copied to user-provided shared texture.
* Why not use generic appsink approach?
In order for application to be able to store video data
which was produced by GStreamer in application's own texture,
there would be two possible approaches,
one is copying our texture into application's own texture,
and the other is drawing on application's own texture directly.
The former (appsink way) cannot be a zero-copy by nature.
In order to support zero-copy processing, we need to draw on
application's own texture directly.
For example, assume that application wants RGBA texture.
Then we can imagine following case.
"d3d11h264dec ! d3d11convert ! video/x-raw(memory:D3D11Memory),format=RGBA ! appsink"
^
|_ allocate new Direct3D texture for RGBA format
In above case, d3d11convert will allocate new texture(s) for RGBA format
and then application will copy again the our RGBA texutre into
application's own texture. One texture allocation plus per frame GPU copy will hanppen
in that case therefore.
Moreover, in order for application to be able to access
our texture, we need to allocate texture with additional flags for
application's Direct3D11 device to be able to read texture data.
That would be another implementation burden on our side
But with this MR, we can configure pipeline in this way
"d3d11h264dec ! d3d11videosink".
In that way, we can save at least one texture allocation and
per frame texutre copy since d3d11videosink will convert incoming texture
into application's texture format directly without copy.
* What if we expose texture without conversion and application does
conversion by itself?
As mentioned above, for application to be able to access our texture
from application's Direct3D11 device, we need to allocate texture
in a special form. But in some case, that might not be possible.
Also, if a texture belongs to decoder DPB, exposing such texture
to application is unsafe and usual Direct3D11 shader cannot handle
such texture. To convert format, ID3D11VideoProcessor API needs to
be used but that would be a implementation burden for application.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1873>
2020-12-23 14:49:12 +00:00
|
|
|
|
2022-08-18 15:05:17 +00:00
|
|
|
InitializeCriticalSection (&self->lock);
|
2019-01-30 11:07:29 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
static void
|
|
|
|
|
gst_d3d11_videosink_set_property (GObject * object, guint prop_id,
|
|
|
|
|
const GValue * value, GParamSpec * pspec)
|
|
|
|
|
{
|
|
|
|
|
GstD3D11VideoSink *self = GST_D3D11_VIDEO_SINK (object);
|
2022-08-18 15:05:17 +00:00
|
|
|
GstD3D11CSLockGuard lk (&self->lock);
|
2019-01-30 11:07:29 +00:00
|
|
|
|
|
|
|
|
switch (prop_id) {
|
|
|
|
|
case PROP_ADAPTER:
|
|
|
|
|
self->adapter = g_value_get_int (value);
|
|
|
|
|
break;
|
|
|
|
|
case PROP_FORCE_ASPECT_RATIO:
|
|
|
|
|
self->force_aspect_ratio = g_value_get_boolean (value);
|
|
|
|
|
if (self->window)
|
|
|
|
|
g_object_set (self->window,
|
|
|
|
|
"force-aspect-ratio", self->force_aspect_ratio, NULL);
|
|
|
|
|
break;
|
|
|
|
|
case PROP_ENABLE_NAVIGATION_EVENTS:
|
|
|
|
|
self->enable_navigation_events = g_value_get_boolean (value);
|
|
|
|
|
if (self->window) {
|
|
|
|
|
g_object_set (self->window,
|
|
|
|
|
"enable-navigation-events", self->enable_navigation_events, NULL);
|
|
|
|
|
}
|
|
|
|
|
break;
|
2019-12-15 10:29:10 +00:00
|
|
|
case PROP_FULLSCREEN_TOGGLE_MODE:
|
2021-03-13 08:40:57 +00:00
|
|
|
self->fullscreen_toggle_mode =
|
|
|
|
|
(GstD3D11WindowFullscreenToggleMode) g_value_get_flags (value);
|
2019-12-15 10:29:10 +00:00
|
|
|
if (self->window) {
|
|
|
|
|
g_object_set (self->window,
|
|
|
|
|
"fullscreen-toggle-mode", self->fullscreen_toggle_mode, NULL);
|
|
|
|
|
}
|
|
|
|
|
break;
|
|
|
|
|
case PROP_FULLSCREEN:
|
|
|
|
|
self->fullscreen = g_value_get_boolean (value);
|
|
|
|
|
if (self->window) {
|
|
|
|
|
g_object_set (self->window, "fullscreen", self->fullscreen, NULL);
|
|
|
|
|
}
|
|
|
|
|
break;
|
d3d11videosink: Add support for drawing on application's own texture
Add a way to support drawing on application's texture instead of
usual window handle.
To make use of this new feature, application should follow below step.
1) Enable this feature by using "draw-on-shared-texture" property
2) Watch "begin-draw" signal
3) On "begin-draw" signal handler, application can request drawing
by using "draw" signal action. Note that "draw" signal action
should be happen before "begin-draw" signal handler is returned
NOTE 1) For texture sharing, creating a texture with
D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX flag is strongly recommend
if possible because we cannot ensure sync a texture
which was created with D3D11_RESOURCE_MISC_SHARED
and it would cause glitch with ID3D11VideoProcessor use case.
NOTE 2) Direct9Ex doesn't support texture sharing which was
created with D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX. In other words,
D3D11_RESOURCE_MISC_SHARED is the only option for Direct3D11/Direct9Ex interop.
NOTE 3) Because of missing synchronization around ID3D11VideoProcessor,
If shared texture was created with D3D11_RESOURCE_MISC_SHARED,
d3d11videosink might use fallback texture to convert DXVA texture
to normal Direct3D texture. Then converted texture will be
copied to user-provided shared texture.
* Why not use generic appsink approach?
In order for application to be able to store video data
which was produced by GStreamer in application's own texture,
there would be two possible approaches,
one is copying our texture into application's own texture,
and the other is drawing on application's own texture directly.
The former (appsink way) cannot be a zero-copy by nature.
In order to support zero-copy processing, we need to draw on
application's own texture directly.
For example, assume that application wants RGBA texture.
Then we can imagine following case.
"d3d11h264dec ! d3d11convert ! video/x-raw(memory:D3D11Memory),format=RGBA ! appsink"
^
|_ allocate new Direct3D texture for RGBA format
In above case, d3d11convert will allocate new texture(s) for RGBA format
and then application will copy again the our RGBA texutre into
application's own texture. One texture allocation plus per frame GPU copy will hanppen
in that case therefore.
Moreover, in order for application to be able to access
our texture, we need to allocate texture with additional flags for
application's Direct3D11 device to be able to read texture data.
That would be another implementation burden on our side
But with this MR, we can configure pipeline in this way
"d3d11h264dec ! d3d11videosink".
In that way, we can save at least one texture allocation and
per frame texutre copy since d3d11videosink will convert incoming texture
into application's texture format directly without copy.
* What if we expose texture without conversion and application does
conversion by itself?
As mentioned above, for application to be able to access our texture
from application's Direct3D11 device, we need to allocate texture
in a special form. But in some case, that might not be possible.
Also, if a texture belongs to decoder DPB, exposing such texture
to application is unsafe and usual Direct3D11 shader cannot handle
such texture. To convert format, ID3D11VideoProcessor API needs to
be used but that would be a implementation burden for application.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1873>
2020-12-23 14:49:12 +00:00
|
|
|
case PROP_DRAW_ON_SHARED_TEXTURE:
|
|
|
|
|
self->draw_on_shared_texture = g_value_get_boolean (value);
|
|
|
|
|
break;
|
2022-07-02 16:22:10 +00:00
|
|
|
case PROP_ROTATE_METHOD:
|
|
|
|
|
gst_d3d11_video_sink_set_orientation (self,
|
|
|
|
|
(GstVideoOrientationMethod) g_value_get_enum (value), FALSE);
|
|
|
|
|
break;
|
2022-08-02 16:47:46 +00:00
|
|
|
case PROP_GAMMA_MODE:
|
|
|
|
|
self->gamma_mode = (GstVideoGammaMode) g_value_get_enum (value);
|
|
|
|
|
break;
|
|
|
|
|
case PROP_PRIMARIES_MODE:
|
|
|
|
|
self->primaries_mode = (GstVideoPrimariesMode) g_value_get_enum (value);
|
|
|
|
|
break;
|
2022-08-23 15:13:21 +00:00
|
|
|
case PROP_DISPLAY_FORMAT:
|
|
|
|
|
self->display_format = (DXGI_FORMAT) g_value_get_enum (value);
|
|
|
|
|
break;
|
2022-10-13 15:17:52 +00:00
|
|
|
case PROP_EMIT_PRESENT:
|
|
|
|
|
self->emit_present = g_value_get_boolean (value);
|
|
|
|
|
break;
|
2023-05-14 11:05:25 +00:00
|
|
|
case PROP_RENDER_RECTANGE:
|
|
|
|
|
gst_video_overlay_set_property (object, PROP_RENDER_RECTANGE,
|
|
|
|
|
PROP_RENDER_RECTANGE, value);
|
|
|
|
|
break;
|
2019-01-30 11:07:29 +00:00
|
|
|
default:
|
|
|
|
|
G_OBJECT_WARN_INVALID_PROPERTY_ID (object, prop_id, pspec);
|
|
|
|
|
break;
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
static void
|
|
|
|
|
gst_d3d11_videosink_get_property (GObject * object, guint prop_id,
|
|
|
|
|
GValue * value, GParamSpec * pspec)
|
|
|
|
|
{
|
|
|
|
|
GstD3D11VideoSink *self = GST_D3D11_VIDEO_SINK (object);
|
2022-08-18 15:05:17 +00:00
|
|
|
GstD3D11CSLockGuard lk (&self->lock);
|
2019-01-30 11:07:29 +00:00
|
|
|
|
|
|
|
|
switch (prop_id) {
|
|
|
|
|
case PROP_ADAPTER:
|
|
|
|
|
g_value_set_int (value, self->adapter);
|
|
|
|
|
break;
|
|
|
|
|
case PROP_FORCE_ASPECT_RATIO:
|
|
|
|
|
g_value_set_boolean (value, self->force_aspect_ratio);
|
|
|
|
|
break;
|
|
|
|
|
case PROP_ENABLE_NAVIGATION_EVENTS:
|
|
|
|
|
g_value_set_boolean (value, self->enable_navigation_events);
|
|
|
|
|
break;
|
2019-12-15 10:29:10 +00:00
|
|
|
case PROP_FULLSCREEN_TOGGLE_MODE:
|
|
|
|
|
g_value_set_flags (value, self->fullscreen_toggle_mode);
|
|
|
|
|
break;
|
|
|
|
|
case PROP_FULLSCREEN:
|
|
|
|
|
if (self->window) {
|
|
|
|
|
g_object_get_property (G_OBJECT (self->window), pspec->name, value);
|
|
|
|
|
} else {
|
|
|
|
|
g_value_set_boolean (value, self->fullscreen);
|
|
|
|
|
}
|
|
|
|
|
break;
|
d3d11videosink: Add support for drawing on application's own texture
Add a way to support drawing on application's texture instead of
usual window handle.
To make use of this new feature, application should follow below step.
1) Enable this feature by using "draw-on-shared-texture" property
2) Watch "begin-draw" signal
3) On "begin-draw" signal handler, application can request drawing
by using "draw" signal action. Note that "draw" signal action
should be happen before "begin-draw" signal handler is returned
NOTE 1) For texture sharing, creating a texture with
D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX flag is strongly recommend
if possible because we cannot ensure sync a texture
which was created with D3D11_RESOURCE_MISC_SHARED
and it would cause glitch with ID3D11VideoProcessor use case.
NOTE 2) Direct9Ex doesn't support texture sharing which was
created with D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX. In other words,
D3D11_RESOURCE_MISC_SHARED is the only option for Direct3D11/Direct9Ex interop.
NOTE 3) Because of missing synchronization around ID3D11VideoProcessor,
If shared texture was created with D3D11_RESOURCE_MISC_SHARED,
d3d11videosink might use fallback texture to convert DXVA texture
to normal Direct3D texture. Then converted texture will be
copied to user-provided shared texture.
* Why not use generic appsink approach?
In order for application to be able to store video data
which was produced by GStreamer in application's own texture,
there would be two possible approaches,
one is copying our texture into application's own texture,
and the other is drawing on application's own texture directly.
The former (appsink way) cannot be a zero-copy by nature.
In order to support zero-copy processing, we need to draw on
application's own texture directly.
For example, assume that application wants RGBA texture.
Then we can imagine following case.
"d3d11h264dec ! d3d11convert ! video/x-raw(memory:D3D11Memory),format=RGBA ! appsink"
^
|_ allocate new Direct3D texture for RGBA format
In above case, d3d11convert will allocate new texture(s) for RGBA format
and then application will copy again the our RGBA texutre into
application's own texture. One texture allocation plus per frame GPU copy will hanppen
in that case therefore.
Moreover, in order for application to be able to access
our texture, we need to allocate texture with additional flags for
application's Direct3D11 device to be able to read texture data.
That would be another implementation burden on our side
But with this MR, we can configure pipeline in this way
"d3d11h264dec ! d3d11videosink".
In that way, we can save at least one texture allocation and
per frame texutre copy since d3d11videosink will convert incoming texture
into application's texture format directly without copy.
* What if we expose texture without conversion and application does
conversion by itself?
As mentioned above, for application to be able to access our texture
from application's Direct3D11 device, we need to allocate texture
in a special form. But in some case, that might not be possible.
Also, if a texture belongs to decoder DPB, exposing such texture
to application is unsafe and usual Direct3D11 shader cannot handle
such texture. To convert format, ID3D11VideoProcessor API needs to
be used but that would be a implementation burden for application.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1873>
2020-12-23 14:49:12 +00:00
|
|
|
case PROP_DRAW_ON_SHARED_TEXTURE:
|
|
|
|
|
g_value_set_boolean (value, self->draw_on_shared_texture);
|
|
|
|
|
break;
|
2022-07-02 16:22:10 +00:00
|
|
|
case PROP_ROTATE_METHOD:
|
|
|
|
|
g_value_set_enum (value, self->method);
|
|
|
|
|
break;
|
2022-08-02 16:47:46 +00:00
|
|
|
case PROP_GAMMA_MODE:
|
|
|
|
|
g_value_set_enum (value, self->gamma_mode);
|
|
|
|
|
break;
|
|
|
|
|
case PROP_PRIMARIES_MODE:
|
|
|
|
|
g_value_set_enum (value, self->primaries_mode);
|
|
|
|
|
break;
|
2022-08-23 15:13:21 +00:00
|
|
|
case PROP_DISPLAY_FORMAT:
|
|
|
|
|
g_value_set_enum (value, self->display_format);
|
|
|
|
|
break;
|
2022-10-13 15:17:52 +00:00
|
|
|
case PROP_EMIT_PRESENT:
|
|
|
|
|
g_value_set_boolean (value, self->emit_present);
|
|
|
|
|
break;
|
2019-01-30 11:07:29 +00:00
|
|
|
default:
|
|
|
|
|
G_OBJECT_WARN_INVALID_PROPERTY_ID (object, prop_id, pspec);
|
|
|
|
|
break;
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
d3d11videosink: Add support for drawing on application's own texture
Add a way to support drawing on application's texture instead of
usual window handle.
To make use of this new feature, application should follow below step.
1) Enable this feature by using "draw-on-shared-texture" property
2) Watch "begin-draw" signal
3) On "begin-draw" signal handler, application can request drawing
by using "draw" signal action. Note that "draw" signal action
should be happen before "begin-draw" signal handler is returned
NOTE 1) For texture sharing, creating a texture with
D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX flag is strongly recommend
if possible because we cannot ensure sync a texture
which was created with D3D11_RESOURCE_MISC_SHARED
and it would cause glitch with ID3D11VideoProcessor use case.
NOTE 2) Direct9Ex doesn't support texture sharing which was
created with D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX. In other words,
D3D11_RESOURCE_MISC_SHARED is the only option for Direct3D11/Direct9Ex interop.
NOTE 3) Because of missing synchronization around ID3D11VideoProcessor,
If shared texture was created with D3D11_RESOURCE_MISC_SHARED,
d3d11videosink might use fallback texture to convert DXVA texture
to normal Direct3D texture. Then converted texture will be
copied to user-provided shared texture.
* Why not use generic appsink approach?
In order for application to be able to store video data
which was produced by GStreamer in application's own texture,
there would be two possible approaches,
one is copying our texture into application's own texture,
and the other is drawing on application's own texture directly.
The former (appsink way) cannot be a zero-copy by nature.
In order to support zero-copy processing, we need to draw on
application's own texture directly.
For example, assume that application wants RGBA texture.
Then we can imagine following case.
"d3d11h264dec ! d3d11convert ! video/x-raw(memory:D3D11Memory),format=RGBA ! appsink"
^
|_ allocate new Direct3D texture for RGBA format
In above case, d3d11convert will allocate new texture(s) for RGBA format
and then application will copy again the our RGBA texutre into
application's own texture. One texture allocation plus per frame GPU copy will hanppen
in that case therefore.
Moreover, in order for application to be able to access
our texture, we need to allocate texture with additional flags for
application's Direct3D11 device to be able to read texture data.
That would be another implementation burden on our side
But with this MR, we can configure pipeline in this way
"d3d11h264dec ! d3d11videosink".
In that way, we can save at least one texture allocation and
per frame texutre copy since d3d11videosink will convert incoming texture
into application's texture format directly without copy.
* What if we expose texture without conversion and application does
conversion by itself?
As mentioned above, for application to be able to access our texture
from application's Direct3D11 device, we need to allocate texture
in a special form. But in some case, that might not be possible.
Also, if a texture belongs to decoder DPB, exposing such texture
to application is unsafe and usual Direct3D11 shader cannot handle
such texture. To convert format, ID3D11VideoProcessor API needs to
be used but that would be a implementation burden for application.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1873>
2020-12-23 14:49:12 +00:00
|
|
|
static void
|
|
|
|
|
gst_d3d11_video_sink_finalize (GObject * object)
|
|
|
|
|
{
|
|
|
|
|
GstD3D11VideoSink *self = GST_D3D11_VIDEO_SINK (object);
|
|
|
|
|
|
2022-08-18 15:05:17 +00:00
|
|
|
DeleteCriticalSection (&self->lock);
|
2021-09-17 17:27:51 +00:00
|
|
|
g_free (self->title);
|
d3d11videosink: Add support for drawing on application's own texture
Add a way to support drawing on application's texture instead of
usual window handle.
To make use of this new feature, application should follow below step.
1) Enable this feature by using "draw-on-shared-texture" property
2) Watch "begin-draw" signal
3) On "begin-draw" signal handler, application can request drawing
by using "draw" signal action. Note that "draw" signal action
should be happen before "begin-draw" signal handler is returned
NOTE 1) For texture sharing, creating a texture with
D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX flag is strongly recommend
if possible because we cannot ensure sync a texture
which was created with D3D11_RESOURCE_MISC_SHARED
and it would cause glitch with ID3D11VideoProcessor use case.
NOTE 2) Direct9Ex doesn't support texture sharing which was
created with D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX. In other words,
D3D11_RESOURCE_MISC_SHARED is the only option for Direct3D11/Direct9Ex interop.
NOTE 3) Because of missing synchronization around ID3D11VideoProcessor,
If shared texture was created with D3D11_RESOURCE_MISC_SHARED,
d3d11videosink might use fallback texture to convert DXVA texture
to normal Direct3D texture. Then converted texture will be
copied to user-provided shared texture.
* Why not use generic appsink approach?
In order for application to be able to store video data
which was produced by GStreamer in application's own texture,
there would be two possible approaches,
one is copying our texture into application's own texture,
and the other is drawing on application's own texture directly.
The former (appsink way) cannot be a zero-copy by nature.
In order to support zero-copy processing, we need to draw on
application's own texture directly.
For example, assume that application wants RGBA texture.
Then we can imagine following case.
"d3d11h264dec ! d3d11convert ! video/x-raw(memory:D3D11Memory),format=RGBA ! appsink"
^
|_ allocate new Direct3D texture for RGBA format
In above case, d3d11convert will allocate new texture(s) for RGBA format
and then application will copy again the our RGBA texutre into
application's own texture. One texture allocation plus per frame GPU copy will hanppen
in that case therefore.
Moreover, in order for application to be able to access
our texture, we need to allocate texture with additional flags for
application's Direct3D11 device to be able to read texture data.
That would be another implementation burden on our side
But with this MR, we can configure pipeline in this way
"d3d11h264dec ! d3d11videosink".
In that way, we can save at least one texture allocation and
per frame texutre copy since d3d11videosink will convert incoming texture
into application's texture format directly without copy.
* What if we expose texture without conversion and application does
conversion by itself?
As mentioned above, for application to be able to access our texture
from application's Direct3D11 device, we need to allocate texture
in a special form. But in some case, that might not be possible.
Also, if a texture belongs to decoder DPB, exposing such texture
to application is unsafe and usual Direct3D11 shader cannot handle
such texture. To convert format, ID3D11VideoProcessor API needs to
be used but that would be a implementation burden for application.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1873>
2020-12-23 14:49:12 +00:00
|
|
|
|
|
|
|
|
G_OBJECT_CLASS (parent_class)->finalize (object);
|
|
|
|
|
}
|
|
|
|
|
|
2019-01-30 11:07:29 +00:00
|
|
|
static void
|
|
|
|
|
gst_d3d11_video_sink_set_context (GstElement * element, GstContext * context)
|
|
|
|
|
{
|
|
|
|
|
GstD3D11VideoSink *self = GST_D3D11_VIDEO_SINK (element);
|
|
|
|
|
|
2019-08-23 12:41:39 +00:00
|
|
|
gst_d3d11_handle_set_context (element, context, self->adapter, &self->device);
|
2019-01-30 11:07:29 +00:00
|
|
|
|
|
|
|
|
GST_ELEMENT_CLASS (parent_class)->set_context (element, context);
|
|
|
|
|
}
|
|
|
|
|
|
2020-12-20 17:47:45 +00:00
|
|
|
static GstCaps *
|
2021-04-17 11:37:13 +00:00
|
|
|
gst_d3d11_video_sink_get_caps (GstBaseSink * sink, GstCaps * filter)
|
2020-12-20 17:47:45 +00:00
|
|
|
{
|
2021-04-17 11:37:13 +00:00
|
|
|
GstD3D11VideoSink *self = GST_D3D11_VIDEO_SINK (sink);
|
|
|
|
|
GstCaps *caps = NULL;
|
2020-12-20 17:47:45 +00:00
|
|
|
|
2021-04-17 11:37:13 +00:00
|
|
|
caps = gst_pad_get_pad_template_caps (GST_VIDEO_SINK_PAD (sink));
|
2020-12-20 17:47:45 +00:00
|
|
|
|
2021-04-17 11:37:13 +00:00
|
|
|
if (self->device) {
|
|
|
|
|
gboolean is_hardware = FALSE;
|
2020-12-20 17:47:45 +00:00
|
|
|
|
2021-04-17 11:37:13 +00:00
|
|
|
g_object_get (self->device, "hardware", &is_hardware, NULL);
|
2020-12-20 17:47:45 +00:00
|
|
|
|
2021-04-17 11:37:13 +00:00
|
|
|
/* In case of WARP device, conversion via shader would be inefficient than
|
|
|
|
|
* upstream videoconvert. Allow native formats in this case */
|
|
|
|
|
if (!is_hardware) {
|
|
|
|
|
GValue format_list = G_VALUE_INIT;
|
|
|
|
|
GValue format = G_VALUE_INIT;
|
2019-01-30 11:07:29 +00:00
|
|
|
|
2021-04-17 11:37:13 +00:00
|
|
|
g_value_init (&format_list, GST_TYPE_LIST);
|
|
|
|
|
g_value_init (&format, G_TYPE_STRING);
|
2019-12-24 06:54:57 +00:00
|
|
|
|
2021-04-17 11:37:13 +00:00
|
|
|
g_value_set_string (&format, "RGBA");
|
|
|
|
|
gst_value_list_append_and_take_value (&format_list, &format);
|
2019-01-30 11:07:29 +00:00
|
|
|
|
2021-04-17 11:37:13 +00:00
|
|
|
format = G_VALUE_INIT;
|
|
|
|
|
g_value_init (&format, G_TYPE_STRING);
|
|
|
|
|
g_value_set_string (&format, "BGRA");
|
|
|
|
|
gst_value_list_append_and_take_value (&format_list, &format);
|
2019-01-30 11:07:29 +00:00
|
|
|
|
2021-04-17 11:37:13 +00:00
|
|
|
caps = gst_caps_make_writable (caps);
|
|
|
|
|
gst_caps_set_value (caps, "format", &format_list);
|
|
|
|
|
g_value_unset (&format_list);
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
if (filter) {
|
2019-01-30 11:07:29 +00:00
|
|
|
GstCaps *isect;
|
|
|
|
|
isect = gst_caps_intersect_full (filter, caps, GST_CAPS_INTERSECT_FIRST);
|
|
|
|
|
gst_caps_unref (caps);
|
|
|
|
|
caps = isect;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
return caps;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
static gboolean
|
|
|
|
|
gst_d3d11_video_sink_set_caps (GstBaseSink * sink, GstCaps * caps)
|
|
|
|
|
{
|
|
|
|
|
GstD3D11VideoSink *self = GST_D3D11_VIDEO_SINK (sink);
|
2021-04-23 10:29:55 +00:00
|
|
|
|
|
|
|
|
GST_DEBUG_OBJECT (self, "set caps %" GST_PTR_FORMAT, caps);
|
|
|
|
|
|
|
|
|
|
/* We will update window on show_frame() */
|
|
|
|
|
self->caps_updated = TRUE;
|
|
|
|
|
|
|
|
|
|
return TRUE;
|
|
|
|
|
}
|
|
|
|
|
|
2022-12-14 16:15:10 +00:00
|
|
|
static GstFlowReturn
|
2021-04-23 10:29:55 +00:00
|
|
|
gst_d3d11_video_sink_update_window (GstD3D11VideoSink * self, GstCaps * caps)
|
|
|
|
|
{
|
2019-01-30 11:07:29 +00:00
|
|
|
gint video_width, video_height;
|
|
|
|
|
gint video_par_n, video_par_d; /* video's PAR */
|
|
|
|
|
gint display_par_n = 1, display_par_d = 1; /* display's PAR */
|
|
|
|
|
guint num, den;
|
2019-10-04 12:15:15 +00:00
|
|
|
GError *error = NULL;
|
2022-08-02 16:47:46 +00:00
|
|
|
GstStructure *config;
|
2022-12-14 16:15:10 +00:00
|
|
|
GstD3D11Window *window;
|
|
|
|
|
GstFlowReturn ret = GST_FLOW_OK;
|
2022-12-15 17:52:08 +00:00
|
|
|
GstD3D11AllocationParams *params;
|
|
|
|
|
guint bind_flags = D3D11_BIND_SHADER_RESOURCE;
|
|
|
|
|
GstD3D11Format device_format;
|
2019-01-30 11:07:29 +00:00
|
|
|
|
2021-04-23 10:29:55 +00:00
|
|
|
GST_DEBUG_OBJECT (self, "Updating window with caps %" GST_PTR_FORMAT, caps);
|
|
|
|
|
|
|
|
|
|
self->caps_updated = FALSE;
|
2022-12-15 17:52:08 +00:00
|
|
|
if (self->pool) {
|
|
|
|
|
gst_buffer_pool_set_active (self->pool, FALSE);
|
|
|
|
|
gst_clear_object (&self->pool);
|
|
|
|
|
}
|
2019-01-30 11:07:29 +00:00
|
|
|
|
2022-08-18 15:05:17 +00:00
|
|
|
EnterCriticalSection (&self->lock);
|
2022-07-02 16:18:19 +00:00
|
|
|
if (!gst_d3d11_video_sink_prepare_window (self)) {
|
2022-08-18 15:05:17 +00:00
|
|
|
LeaveCriticalSection (&self->lock);
|
2022-07-02 16:18:19 +00:00
|
|
|
|
|
|
|
|
GST_ELEMENT_ERROR (self, RESOURCE, NOT_FOUND, (nullptr),
|
|
|
|
|
("Failed to open window."));
|
|
|
|
|
|
2022-12-14 16:15:10 +00:00
|
|
|
return GST_FLOW_ERROR;
|
2022-07-02 16:18:19 +00:00
|
|
|
}
|
2020-12-20 13:12:44 +00:00
|
|
|
|
2022-07-02 16:18:19 +00:00
|
|
|
if (!gst_video_info_from_caps (&self->info, caps)) {
|
|
|
|
|
GST_DEBUG_OBJECT (self,
|
|
|
|
|
"Could not locate image format from caps %" GST_PTR_FORMAT, caps);
|
2022-08-18 15:05:17 +00:00
|
|
|
LeaveCriticalSection (&self->lock);
|
2022-12-14 16:15:10 +00:00
|
|
|
return GST_FLOW_ERROR;
|
2022-07-02 16:18:19 +00:00
|
|
|
}
|
2019-01-30 11:07:29 +00:00
|
|
|
|
|
|
|
|
video_width = GST_VIDEO_INFO_WIDTH (&self->info);
|
|
|
|
|
video_height = GST_VIDEO_INFO_HEIGHT (&self->info);
|
|
|
|
|
video_par_n = GST_VIDEO_INFO_PAR_N (&self->info);
|
|
|
|
|
video_par_d = GST_VIDEO_INFO_PAR_D (&self->info);
|
|
|
|
|
|
|
|
|
|
/* get aspect ratio from caps if it's present, and
|
|
|
|
|
* convert video width and height to a display width and height
|
|
|
|
|
* using wd / hd = wv / hv * PARv / PARd */
|
|
|
|
|
|
|
|
|
|
if (!gst_video_calculate_display_ratio (&num, &den, video_width,
|
2022-07-02 16:18:19 +00:00
|
|
|
video_height, video_par_n, video_par_d, display_par_n,
|
|
|
|
|
display_par_d)) {
|
2022-08-18 15:05:17 +00:00
|
|
|
LeaveCriticalSection (&self->lock);
|
2022-07-02 16:18:19 +00:00
|
|
|
|
|
|
|
|
GST_ELEMENT_ERROR (self, CORE, NEGOTIATION, (nullptr),
|
|
|
|
|
("Error calculating the output display ratio of the video."));
|
2022-12-14 16:15:10 +00:00
|
|
|
return GST_FLOW_ERROR;
|
2022-07-02 16:18:19 +00:00
|
|
|
}
|
2019-01-30 11:07:29 +00:00
|
|
|
|
2021-04-23 10:29:55 +00:00
|
|
|
GST_DEBUG_OBJECT (self,
|
2019-01-30 11:07:29 +00:00
|
|
|
"video width/height: %dx%d, calculated display ratio: %d/%d format: %s",
|
|
|
|
|
video_width, video_height, num, den,
|
|
|
|
|
gst_video_format_to_string (GST_VIDEO_INFO_FORMAT (&self->info)));
|
|
|
|
|
|
|
|
|
|
/* now find a width x height that respects this display ratio.
|
|
|
|
|
* prefer those that have one of w/h the same as the incoming video
|
|
|
|
|
* using wd / hd = num / den
|
|
|
|
|
*/
|
|
|
|
|
|
|
|
|
|
/* start with same height, because of interlaced video
|
|
|
|
|
* check hd / den is an integer scale factor, and scale wd with the PAR
|
|
|
|
|
*/
|
|
|
|
|
if (video_height % den == 0) {
|
|
|
|
|
GST_DEBUG_OBJECT (self, "keeping video height");
|
|
|
|
|
GST_VIDEO_SINK_WIDTH (self) = (guint)
|
|
|
|
|
gst_util_uint64_scale_int (video_height, num, den);
|
|
|
|
|
GST_VIDEO_SINK_HEIGHT (self) = video_height;
|
|
|
|
|
} else if (video_width % num == 0) {
|
|
|
|
|
GST_DEBUG_OBJECT (self, "keeping video width");
|
|
|
|
|
GST_VIDEO_SINK_WIDTH (self) = video_width;
|
|
|
|
|
GST_VIDEO_SINK_HEIGHT (self) = (guint)
|
|
|
|
|
gst_util_uint64_scale_int (video_width, den, num);
|
|
|
|
|
} else {
|
|
|
|
|
GST_DEBUG_OBJECT (self, "approximating while keeping video height");
|
|
|
|
|
GST_VIDEO_SINK_WIDTH (self) = (guint)
|
|
|
|
|
gst_util_uint64_scale_int (video_height, num, den);
|
|
|
|
|
GST_VIDEO_SINK_HEIGHT (self) = video_height;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
GST_DEBUG_OBJECT (self, "scaling to %dx%d",
|
|
|
|
|
GST_VIDEO_SINK_WIDTH (self), GST_VIDEO_SINK_HEIGHT (self));
|
2019-08-15 07:20:26 +00:00
|
|
|
self->video_width = video_width;
|
|
|
|
|
self->video_height = video_height;
|
2019-01-30 11:07:29 +00:00
|
|
|
|
2022-07-02 16:18:19 +00:00
|
|
|
if (GST_VIDEO_SINK_WIDTH (self) <= 0 || GST_VIDEO_SINK_HEIGHT (self) <= 0) {
|
2022-08-18 15:05:17 +00:00
|
|
|
LeaveCriticalSection (&self->lock);
|
2022-07-02 16:18:19 +00:00
|
|
|
|
|
|
|
|
GST_ELEMENT_ERROR (self, CORE, NEGOTIATION, (nullptr),
|
|
|
|
|
("Error calculating the output display ratio of the video."));
|
2022-12-14 16:15:10 +00:00
|
|
|
return GST_FLOW_ERROR;
|
2022-07-02 16:18:19 +00:00
|
|
|
}
|
2019-01-30 11:07:29 +00:00
|
|
|
|
2021-07-30 15:59:14 +00:00
|
|
|
if (self->pending_render_rect) {
|
|
|
|
|
GstVideoRectangle rect = self->render_rect;
|
2019-01-30 11:07:29 +00:00
|
|
|
|
2021-07-30 15:59:14 +00:00
|
|
|
self->pending_render_rect = FALSE;
|
|
|
|
|
gst_d3d11_window_set_render_rectangle (self->window, &rect);
|
|
|
|
|
}
|
2019-01-30 11:07:29 +00:00
|
|
|
|
2022-08-02 16:47:46 +00:00
|
|
|
config = gst_structure_new ("convert-config",
|
|
|
|
|
GST_D3D11_CONVERTER_OPT_GAMMA_MODE,
|
|
|
|
|
GST_TYPE_VIDEO_GAMMA_MODE, self->gamma_mode,
|
|
|
|
|
GST_D3D11_CONVERTER_OPT_PRIMARIES_MODE,
|
|
|
|
|
GST_TYPE_VIDEO_PRIMARIES_MODE, self->primaries_mode, nullptr);
|
|
|
|
|
|
2022-12-14 16:15:10 +00:00
|
|
|
window = (GstD3D11Window *) gst_object_ref (self->window);
|
|
|
|
|
LeaveCriticalSection (&self->lock);
|
|
|
|
|
|
|
|
|
|
ret = gst_d3d11_window_prepare (window, GST_VIDEO_SINK_WIDTH (self),
|
|
|
|
|
GST_VIDEO_SINK_HEIGHT (self), caps, config, self->display_format, &error);
|
|
|
|
|
if (ret != GST_FLOW_OK) {
|
2019-10-04 12:15:15 +00:00
|
|
|
GstMessage *error_msg;
|
|
|
|
|
|
2022-12-14 16:15:10 +00:00
|
|
|
if (ret == GST_FLOW_FLUSHING) {
|
|
|
|
|
GstD3D11CSLockGuard lk (&self->lock);
|
|
|
|
|
GST_WARNING_OBJECT (self, "Couldn't prepare window but we are flushing");
|
2023-03-27 07:20:27 +00:00
|
|
|
gst_d3d11_window_unprepare (self->window);
|
2022-12-14 16:15:10 +00:00
|
|
|
gst_clear_object (&self->window);
|
|
|
|
|
gst_object_unref (window);
|
|
|
|
|
|
|
|
|
|
return GST_FLOW_FLUSHING;
|
|
|
|
|
}
|
2022-07-02 16:18:19 +00:00
|
|
|
|
2019-01-30 11:07:29 +00:00
|
|
|
GST_ERROR_OBJECT (self, "cannot create swapchain");
|
2019-10-04 12:15:15 +00:00
|
|
|
error_msg = gst_message_new_error (GST_OBJECT_CAST (self),
|
|
|
|
|
error, "Failed to prepare d3d11window");
|
|
|
|
|
g_clear_error (&error);
|
|
|
|
|
gst_element_post_message (GST_ELEMENT (self), error_msg);
|
2022-12-14 16:15:10 +00:00
|
|
|
gst_object_unref (window);
|
2019-10-04 12:15:15 +00:00
|
|
|
|
2022-12-14 16:15:10 +00:00
|
|
|
return GST_FLOW_ERROR;
|
2019-01-30 11:07:29 +00:00
|
|
|
}
|
|
|
|
|
|
2021-09-17 17:27:51 +00:00
|
|
|
if (self->title) {
|
2022-12-14 16:15:10 +00:00
|
|
|
gst_d3d11_window_set_title (window, self->title);
|
2021-09-17 17:27:51 +00:00
|
|
|
g_clear_pointer (&self->title, g_free);
|
|
|
|
|
}
|
|
|
|
|
|
2022-12-14 16:15:10 +00:00
|
|
|
gst_object_unref (window);
|
2019-01-30 11:07:29 +00:00
|
|
|
|
2022-12-15 17:52:08 +00:00
|
|
|
self->pool = gst_d3d11_buffer_pool_new (self->device);
|
|
|
|
|
config = gst_buffer_pool_get_config (self->pool);
|
|
|
|
|
|
|
|
|
|
if (gst_d3d11_device_get_format (self->device,
|
|
|
|
|
GST_VIDEO_INFO_FORMAT (&self->info), &device_format) &&
|
|
|
|
|
(device_format.format_support[0] &
|
|
|
|
|
(guint) D3D11_FORMAT_SUPPORT_RENDER_TARGET) != 0) {
|
|
|
|
|
bind_flags |= D3D11_BIND_RENDER_TARGET;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
params = gst_d3d11_allocation_params_new (self->device, &self->info,
|
|
|
|
|
GST_D3D11_ALLOCATION_FLAG_DEFAULT, bind_flags, 0);
|
|
|
|
|
gst_buffer_pool_config_set_d3d11_allocation_params (config, params);
|
|
|
|
|
gst_d3d11_allocation_params_free (params);
|
|
|
|
|
|
|
|
|
|
gst_buffer_pool_config_set_params (config, caps, self->info.size, 0, 0);
|
|
|
|
|
if (!gst_buffer_pool_set_config (self->pool, config) ||
|
|
|
|
|
!gst_buffer_pool_set_active (self->pool, TRUE)) {
|
|
|
|
|
GST_ERROR_OBJECT (self, "Couldn't setup buffer pool");
|
|
|
|
|
gst_clear_object (&self->pool);
|
|
|
|
|
|
|
|
|
|
GST_ELEMENT_ERROR (self, RESOURCE, FAILED, (nullptr),
|
|
|
|
|
("Couldn't setup buffer pool"));
|
|
|
|
|
return GST_FLOW_ERROR;
|
|
|
|
|
}
|
|
|
|
|
|
2022-12-14 16:15:10 +00:00
|
|
|
return GST_FLOW_OK;
|
2019-01-30 11:07:29 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
static void
|
|
|
|
|
gst_d3d11_video_sink_key_event (GstD3D11Window * window, const gchar * event,
|
|
|
|
|
const gchar * key, GstD3D11VideoSink * self)
|
|
|
|
|
{
|
2022-08-01 19:16:31 +00:00
|
|
|
GstEvent *key_event;
|
|
|
|
|
|
|
|
|
|
if (!self->enable_navigation_events || !event || !key)
|
|
|
|
|
return;
|
|
|
|
|
|
|
|
|
|
GST_LOG_OBJECT (self, "send key event %s, key %s", event, key);
|
|
|
|
|
if (g_strcmp0 ("key-press", event) == 0) {
|
|
|
|
|
key_event = gst_navigation_event_new_key_press (key,
|
|
|
|
|
GST_NAVIGATION_MODIFIER_NONE);
|
|
|
|
|
} else if (g_strcmp0 ("key-release", event) == 0) {
|
|
|
|
|
key_event = gst_navigation_event_new_key_release (key,
|
|
|
|
|
GST_NAVIGATION_MODIFIER_NONE);
|
|
|
|
|
} else {
|
|
|
|
|
return;
|
2019-01-30 11:07:29 +00:00
|
|
|
}
|
2022-08-01 19:16:31 +00:00
|
|
|
|
|
|
|
|
gst_navigation_send_event_simple (GST_NAVIGATION (self), key_event);
|
2019-01-30 11:07:29 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
static void
|
|
|
|
|
gst_d3d11_video_mouse_key_event (GstD3D11Window * window, const gchar * event,
|
|
|
|
|
gint button, gdouble x, gdouble y, GstD3D11VideoSink * self)
|
|
|
|
|
{
|
2022-08-01 19:16:31 +00:00
|
|
|
GstEvent *mouse_event;
|
|
|
|
|
|
|
|
|
|
if (!self->enable_navigation_events || !event)
|
|
|
|
|
return;
|
|
|
|
|
|
|
|
|
|
GST_LOG_OBJECT (self,
|
|
|
|
|
"send mouse event %s, button %d (%.1f, %.1f)", event, button, x, y);
|
|
|
|
|
if (g_strcmp0 ("mouse-button-press", event) == 0) {
|
|
|
|
|
mouse_event = gst_navigation_event_new_mouse_button_press (button, x, y,
|
|
|
|
|
GST_NAVIGATION_MODIFIER_NONE);
|
|
|
|
|
} else if (g_strcmp0 ("mouse-button-release", event) == 0) {
|
|
|
|
|
mouse_event = gst_navigation_event_new_mouse_button_release (button, x, y,
|
|
|
|
|
GST_NAVIGATION_MODIFIER_NONE);
|
|
|
|
|
} else if (g_strcmp0 ("mouse-move", event) == 0) {
|
|
|
|
|
mouse_event = gst_navigation_event_new_mouse_move (x, y,
|
|
|
|
|
GST_NAVIGATION_MODIFIER_NONE);
|
|
|
|
|
} else {
|
|
|
|
|
return;
|
2019-01-30 11:07:29 +00:00
|
|
|
}
|
2022-08-01 19:16:31 +00:00
|
|
|
|
|
|
|
|
gst_navigation_send_event_simple (GST_NAVIGATION (self), mouse_event);
|
2019-01-30 11:07:29 +00:00
|
|
|
}
|
|
|
|
|
|
2022-08-19 11:25:31 +00:00
|
|
|
static void
|
|
|
|
|
gst_d3d11_video_sink_present (GstD3D11Window * window, GstD3D11Device * device,
|
|
|
|
|
ID3D11RenderTargetView * rtv, GstD3D11VideoSink * self)
|
|
|
|
|
{
|
|
|
|
|
g_signal_emit (self, gst_d3d11_video_sink_signals[SIGNAL_PRESENT], 0,
|
|
|
|
|
device, rtv);
|
|
|
|
|
}
|
|
|
|
|
|
2019-01-30 11:07:29 +00:00
|
|
|
static gboolean
|
|
|
|
|
gst_d3d11_video_sink_start (GstBaseSink * sink)
|
|
|
|
|
{
|
|
|
|
|
GstD3D11VideoSink *self = GST_D3D11_VIDEO_SINK (sink);
|
|
|
|
|
|
|
|
|
|
GST_DEBUG_OBJECT (self, "Start");
|
|
|
|
|
|
2019-08-23 12:41:39 +00:00
|
|
|
if (!gst_d3d11_ensure_element_data (GST_ELEMENT_CAST (self), self->adapter,
|
|
|
|
|
&self->device)) {
|
2019-01-30 11:07:29 +00:00
|
|
|
GST_ERROR_OBJECT (sink, "Cannot create d3d11device");
|
|
|
|
|
return FALSE;
|
|
|
|
|
}
|
|
|
|
|
|
2020-12-20 13:12:44 +00:00
|
|
|
return TRUE;
|
|
|
|
|
}
|
|
|
|
|
|
2022-07-02 16:18:19 +00:00
|
|
|
/* called with lock */
|
2020-12-20 13:12:44 +00:00
|
|
|
static gboolean
|
|
|
|
|
gst_d3d11_video_sink_prepare_window (GstD3D11VideoSink * self)
|
|
|
|
|
{
|
|
|
|
|
GstD3D11WindowNativeType window_type = GST_D3D11_WINDOW_NATIVE_TYPE_HWND;
|
|
|
|
|
|
|
|
|
|
if (self->window)
|
|
|
|
|
return TRUE;
|
|
|
|
|
|
d3d11videosink: Add support for drawing on application's own texture
Add a way to support drawing on application's texture instead of
usual window handle.
To make use of this new feature, application should follow below step.
1) Enable this feature by using "draw-on-shared-texture" property
2) Watch "begin-draw" signal
3) On "begin-draw" signal handler, application can request drawing
by using "draw" signal action. Note that "draw" signal action
should be happen before "begin-draw" signal handler is returned
NOTE 1) For texture sharing, creating a texture with
D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX flag is strongly recommend
if possible because we cannot ensure sync a texture
which was created with D3D11_RESOURCE_MISC_SHARED
and it would cause glitch with ID3D11VideoProcessor use case.
NOTE 2) Direct9Ex doesn't support texture sharing which was
created with D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX. In other words,
D3D11_RESOURCE_MISC_SHARED is the only option for Direct3D11/Direct9Ex interop.
NOTE 3) Because of missing synchronization around ID3D11VideoProcessor,
If shared texture was created with D3D11_RESOURCE_MISC_SHARED,
d3d11videosink might use fallback texture to convert DXVA texture
to normal Direct3D texture. Then converted texture will be
copied to user-provided shared texture.
* Why not use generic appsink approach?
In order for application to be able to store video data
which was produced by GStreamer in application's own texture,
there would be two possible approaches,
one is copying our texture into application's own texture,
and the other is drawing on application's own texture directly.
The former (appsink way) cannot be a zero-copy by nature.
In order to support zero-copy processing, we need to draw on
application's own texture directly.
For example, assume that application wants RGBA texture.
Then we can imagine following case.
"d3d11h264dec ! d3d11convert ! video/x-raw(memory:D3D11Memory),format=RGBA ! appsink"
^
|_ allocate new Direct3D texture for RGBA format
In above case, d3d11convert will allocate new texture(s) for RGBA format
and then application will copy again the our RGBA texutre into
application's own texture. One texture allocation plus per frame GPU copy will hanppen
in that case therefore.
Moreover, in order for application to be able to access
our texture, we need to allocate texture with additional flags for
application's Direct3D11 device to be able to read texture data.
That would be another implementation burden on our side
But with this MR, we can configure pipeline in this way
"d3d11h264dec ! d3d11videosink".
In that way, we can save at least one texture allocation and
per frame texutre copy since d3d11videosink will convert incoming texture
into application's texture format directly without copy.
* What if we expose texture without conversion and application does
conversion by itself?
As mentioned above, for application to be able to access our texture
from application's Direct3D11 device, we need to allocate texture
in a special form. But in some case, that might not be possible.
Also, if a texture belongs to decoder DPB, exposing such texture
to application is unsafe and usual Direct3D11 shader cannot handle
such texture. To convert format, ID3D11VideoProcessor API needs to
be used but that would be a implementation burden for application.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1873>
2020-12-23 14:49:12 +00:00
|
|
|
if (self->draw_on_shared_texture) {
|
|
|
|
|
GST_INFO_OBJECT (self,
|
|
|
|
|
"Create dummy window for rendering on shared texture");
|
|
|
|
|
self->window = gst_d3d11_window_dummy_new (self->device);
|
2022-06-28 08:40:56 +00:00
|
|
|
goto done;
|
d3d11videosink: Add support for drawing on application's own texture
Add a way to support drawing on application's texture instead of
usual window handle.
To make use of this new feature, application should follow below step.
1) Enable this feature by using "draw-on-shared-texture" property
2) Watch "begin-draw" signal
3) On "begin-draw" signal handler, application can request drawing
by using "draw" signal action. Note that "draw" signal action
should be happen before "begin-draw" signal handler is returned
NOTE 1) For texture sharing, creating a texture with
D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX flag is strongly recommend
if possible because we cannot ensure sync a texture
which was created with D3D11_RESOURCE_MISC_SHARED
and it would cause glitch with ID3D11VideoProcessor use case.
NOTE 2) Direct9Ex doesn't support texture sharing which was
created with D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX. In other words,
D3D11_RESOURCE_MISC_SHARED is the only option for Direct3D11/Direct9Ex interop.
NOTE 3) Because of missing synchronization around ID3D11VideoProcessor,
If shared texture was created with D3D11_RESOURCE_MISC_SHARED,
d3d11videosink might use fallback texture to convert DXVA texture
to normal Direct3D texture. Then converted texture will be
copied to user-provided shared texture.
* Why not use generic appsink approach?
In order for application to be able to store video data
which was produced by GStreamer in application's own texture,
there would be two possible approaches,
one is copying our texture into application's own texture,
and the other is drawing on application's own texture directly.
The former (appsink way) cannot be a zero-copy by nature.
In order to support zero-copy processing, we need to draw on
application's own texture directly.
For example, assume that application wants RGBA texture.
Then we can imagine following case.
"d3d11h264dec ! d3d11convert ! video/x-raw(memory:D3D11Memory),format=RGBA ! appsink"
^
|_ allocate new Direct3D texture for RGBA format
In above case, d3d11convert will allocate new texture(s) for RGBA format
and then application will copy again the our RGBA texutre into
application's own texture. One texture allocation plus per frame GPU copy will hanppen
in that case therefore.
Moreover, in order for application to be able to access
our texture, we need to allocate texture with additional flags for
application's Direct3D11 device to be able to read texture data.
That would be another implementation burden on our side
But with this MR, we can configure pipeline in this way
"d3d11h264dec ! d3d11videosink".
In that way, we can save at least one texture allocation and
per frame texutre copy since d3d11videosink will convert incoming texture
into application's texture format directly without copy.
* What if we expose texture without conversion and application does
conversion by itself?
As mentioned above, for application to be able to access our texture
from application's Direct3D11 device, we need to allocate texture
in a special form. But in some case, that might not be possible.
Also, if a texture belongs to decoder DPB, exposing such texture
to application is unsafe and usual Direct3D11 shader cannot handle
such texture. To convert format, ID3D11VideoProcessor API needs to
be used but that would be a implementation burden for application.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1873>
2020-12-23 14:49:12 +00:00
|
|
|
}
|
|
|
|
|
|
2019-12-30 09:58:59 +00:00
|
|
|
if (!self->window_id)
|
|
|
|
|
gst_video_overlay_prepare_window_handle (GST_VIDEO_OVERLAY (self));
|
|
|
|
|
|
|
|
|
|
if (self->window_id) {
|
|
|
|
|
window_type =
|
|
|
|
|
gst_d3d11_window_get_native_type_from_handle (self->window_id);
|
|
|
|
|
|
|
|
|
|
if (window_type != GST_D3D11_WINDOW_NATIVE_TYPE_NONE) {
|
2020-12-20 13:12:44 +00:00
|
|
|
GST_DEBUG_OBJECT (self, "Have window handle %" G_GUINTPTR_FORMAT,
|
|
|
|
|
self->window_id);
|
2019-12-30 09:58:59 +00:00
|
|
|
gst_video_overlay_got_window_handle (GST_VIDEO_OVERLAY (self),
|
|
|
|
|
self->window_id);
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
GST_DEBUG_OBJECT (self, "Create window (type: %s)",
|
|
|
|
|
gst_d3d11_window_get_native_type_to_string (window_type));
|
|
|
|
|
|
|
|
|
|
#if GST_D3D11_WINAPI_ONLY_APP
|
|
|
|
|
if (window_type != GST_D3D11_WINDOW_NATIVE_TYPE_CORE_WINDOW &&
|
|
|
|
|
window_type != GST_D3D11_WINDOW_NATIVE_TYPE_SWAP_CHAIN_PANEL) {
|
2021-01-18 10:23:30 +00:00
|
|
|
GST_ERROR_OBJECT (self, "Overlay handle must be set before READY state");
|
2019-01-30 11:07:29 +00:00
|
|
|
return FALSE;
|
|
|
|
|
}
|
2019-12-30 09:58:59 +00:00
|
|
|
#endif
|
2019-01-30 11:07:29 +00:00
|
|
|
|
2019-12-30 09:58:59 +00:00
|
|
|
switch (window_type) {
|
|
|
|
|
#if (!GST_D3D11_WINAPI_ONLY_APP)
|
|
|
|
|
case GST_D3D11_WINDOW_NATIVE_TYPE_HWND:
|
|
|
|
|
self->window = gst_d3d11_window_win32_new (self->device, self->window_id);
|
|
|
|
|
break;
|
2021-01-18 10:17:14 +00:00
|
|
|
#endif
|
|
|
|
|
#if GST_D3D11_WINAPI_APP
|
2019-12-30 09:58:59 +00:00
|
|
|
case GST_D3D11_WINDOW_NATIVE_TYPE_CORE_WINDOW:
|
|
|
|
|
self->window = gst_d3d11_window_core_window_new (self->device,
|
|
|
|
|
self->window_id);
|
|
|
|
|
break;
|
|
|
|
|
case GST_D3D11_WINDOW_NATIVE_TYPE_SWAP_CHAIN_PANEL:
|
|
|
|
|
self->window = gst_d3d11_window_swap_chain_panel_new (self->device,
|
|
|
|
|
self->window_id);
|
|
|
|
|
break;
|
|
|
|
|
#endif
|
|
|
|
|
default:
|
|
|
|
|
break;
|
|
|
|
|
}
|
2019-12-03 13:54:26 +00:00
|
|
|
|
2022-06-28 08:40:56 +00:00
|
|
|
done:
|
2019-12-30 09:58:59 +00:00
|
|
|
if (!self->window) {
|
2020-12-20 13:12:44 +00:00
|
|
|
GST_ERROR_OBJECT (self, "Cannot create d3d11window");
|
2019-12-30 09:58:59 +00:00
|
|
|
return FALSE;
|
2019-12-03 13:54:26 +00:00
|
|
|
}
|
|
|
|
|
|
2019-01-30 11:07:29 +00:00
|
|
|
g_object_set (self->window,
|
2019-12-30 09:58:59 +00:00
|
|
|
"force-aspect-ratio", self->force_aspect_ratio,
|
|
|
|
|
"fullscreen-toggle-mode", self->fullscreen_toggle_mode,
|
|
|
|
|
"fullscreen", self->fullscreen,
|
2022-08-19 11:25:31 +00:00
|
|
|
"enable-navigation-events", self->enable_navigation_events,
|
2022-10-13 15:17:52 +00:00
|
|
|
"emit-present", self->emit_present, nullptr);
|
2019-01-30 11:07:29 +00:00
|
|
|
|
2022-07-02 16:22:10 +00:00
|
|
|
gst_d3d11_window_set_orientation (self->window, self->selected_method);
|
|
|
|
|
|
2019-01-30 11:07:29 +00:00
|
|
|
g_signal_connect (self->window, "key-event",
|
|
|
|
|
G_CALLBACK (gst_d3d11_video_sink_key_event), self);
|
|
|
|
|
g_signal_connect (self->window, "mouse-event",
|
|
|
|
|
G_CALLBACK (gst_d3d11_video_mouse_key_event), self);
|
2022-08-19 11:25:31 +00:00
|
|
|
g_signal_connect (self->window, "present",
|
|
|
|
|
G_CALLBACK (gst_d3d11_video_sink_present), self);
|
2019-12-30 09:58:59 +00:00
|
|
|
|
2022-12-14 16:15:10 +00:00
|
|
|
GST_DEBUG_OBJECT (self,
|
|
|
|
|
"Have prepared window %" GST_PTR_FORMAT, self->window);
|
|
|
|
|
|
2019-01-30 11:07:29 +00:00
|
|
|
return TRUE;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
static gboolean
|
|
|
|
|
gst_d3d11_video_sink_stop (GstBaseSink * sink)
|
|
|
|
|
{
|
|
|
|
|
GstD3D11VideoSink *self = GST_D3D11_VIDEO_SINK (sink);
|
2022-08-18 15:05:17 +00:00
|
|
|
GstD3D11CSLockGuard lk (&self->lock);
|
2019-01-30 11:07:29 +00:00
|
|
|
|
|
|
|
|
GST_DEBUG_OBJECT (self, "Stop");
|
|
|
|
|
|
2022-12-15 17:52:08 +00:00
|
|
|
gst_clear_buffer (&self->prepared_buffer);
|
|
|
|
|
if (self->pool) {
|
|
|
|
|
gst_buffer_pool_set_active (self->pool, FALSE);
|
|
|
|
|
gst_clear_object (&self->pool);
|
|
|
|
|
}
|
|
|
|
|
|
2020-05-26 16:52:59 +00:00
|
|
|
if (self->window)
|
|
|
|
|
gst_d3d11_window_unprepare (self->window);
|
|
|
|
|
|
2019-01-30 11:07:29 +00:00
|
|
|
gst_clear_object (&self->window);
|
2022-07-02 16:18:19 +00:00
|
|
|
gst_clear_object (&self->device);
|
2021-09-17 17:27:51 +00:00
|
|
|
g_clear_pointer (&self->title, g_free);
|
|
|
|
|
|
2019-01-30 11:07:29 +00:00
|
|
|
return TRUE;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
static gboolean
|
|
|
|
|
gst_d3d11_video_sink_propose_allocation (GstBaseSink * sink, GstQuery * query)
|
|
|
|
|
{
|
|
|
|
|
GstD3D11VideoSink *self = GST_D3D11_VIDEO_SINK (sink);
|
|
|
|
|
GstCaps *caps;
|
|
|
|
|
GstBufferPool *pool = NULL;
|
|
|
|
|
GstVideoInfo info;
|
|
|
|
|
guint size;
|
|
|
|
|
gboolean need_pool;
|
|
|
|
|
|
2021-09-18 14:37:59 +00:00
|
|
|
if (!self->device)
|
2019-01-30 11:07:29 +00:00
|
|
|
return FALSE;
|
|
|
|
|
|
|
|
|
|
gst_query_parse_allocation (query, &caps, &need_pool);
|
|
|
|
|
|
|
|
|
|
if (caps == NULL)
|
|
|
|
|
goto no_caps;
|
|
|
|
|
|
|
|
|
|
if (!gst_video_info_from_caps (&info, caps))
|
|
|
|
|
goto invalid_caps;
|
|
|
|
|
|
|
|
|
|
/* the normal size of a frame */
|
|
|
|
|
size = info.size;
|
|
|
|
|
|
|
|
|
|
if (need_pool) {
|
2021-04-17 11:37:13 +00:00
|
|
|
GstCapsFeatures *features;
|
|
|
|
|
GstStructure *config;
|
2022-06-23 12:43:11 +00:00
|
|
|
gboolean is_d3d11 = FALSE;
|
2021-04-17 11:37:13 +00:00
|
|
|
|
|
|
|
|
features = gst_caps_get_features (caps, 0);
|
|
|
|
|
if (features
|
|
|
|
|
&& gst_caps_features_contains (features,
|
|
|
|
|
GST_CAPS_FEATURE_MEMORY_D3D11_MEMORY)) {
|
|
|
|
|
GST_DEBUG_OBJECT (self, "upstream support d3d11 memory");
|
|
|
|
|
pool = gst_d3d11_buffer_pool_new (self->device);
|
2022-06-23 12:43:11 +00:00
|
|
|
is_d3d11 = TRUE;
|
2021-04-17 11:37:13 +00:00
|
|
|
} else {
|
2022-06-23 12:43:11 +00:00
|
|
|
pool = gst_video_buffer_pool_new ();
|
2021-04-17 11:37:13 +00:00
|
|
|
}
|
2019-12-03 13:54:26 +00:00
|
|
|
|
2021-04-17 11:37:13 +00:00
|
|
|
config = gst_buffer_pool_get_config (pool);
|
|
|
|
|
gst_buffer_pool_config_add_option (config,
|
|
|
|
|
GST_BUFFER_POOL_OPTION_VIDEO_META);
|
2022-06-23 12:43:11 +00:00
|
|
|
if (!is_d3d11) {
|
|
|
|
|
gst_buffer_pool_config_add_option (config,
|
|
|
|
|
GST_BUFFER_POOL_OPTION_VIDEO_ALIGNMENT);
|
|
|
|
|
}
|
2019-01-30 11:07:29 +00:00
|
|
|
|
2021-04-17 11:37:13 +00:00
|
|
|
size = GST_VIDEO_INFO_SIZE (&info);
|
|
|
|
|
if (is_d3d11) {
|
|
|
|
|
GstD3D11AllocationParams *d3d11_params;
|
|
|
|
|
|
|
|
|
|
d3d11_params =
|
|
|
|
|
gst_d3d11_allocation_params_new (self->device,
|
2022-06-23 16:21:48 +00:00
|
|
|
&info, GST_D3D11_ALLOCATION_FLAG_DEFAULT, D3D11_BIND_SHADER_RESOURCE,
|
|
|
|
|
0);
|
2021-04-17 11:37:13 +00:00
|
|
|
|
|
|
|
|
gst_buffer_pool_config_set_d3d11_allocation_params (config, d3d11_params);
|
|
|
|
|
gst_d3d11_allocation_params_free (d3d11_params);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
gst_buffer_pool_config_set_params (config, caps, (guint) size, 2, 0);
|
|
|
|
|
|
|
|
|
|
if (!gst_buffer_pool_set_config (pool, config)) {
|
|
|
|
|
GST_ERROR_OBJECT (pool, "Couldn't set config");
|
|
|
|
|
gst_object_unref (pool);
|
2019-12-03 13:54:26 +00:00
|
|
|
|
2020-12-19 16:06:24 +00:00
|
|
|
return FALSE;
|
2019-01-30 11:07:29 +00:00
|
|
|
}
|
2020-12-19 16:06:24 +00:00
|
|
|
|
2022-02-05 16:06:36 +00:00
|
|
|
/* d3d11 buffer pool will update buffer size based on allocated texture,
|
|
|
|
|
* get size from config again */
|
|
|
|
|
config = gst_buffer_pool_get_config (pool);
|
|
|
|
|
gst_buffer_pool_config_get_params (config, nullptr, &size, nullptr,
|
|
|
|
|
nullptr);
|
|
|
|
|
gst_structure_free (config);
|
2021-09-18 14:37:20 +00:00
|
|
|
|
2022-02-05 16:06:36 +00:00
|
|
|
if (is_d3d11) {
|
2021-09-18 14:37:20 +00:00
|
|
|
/* In case of system memory, we will upload video frame to GPU memory,
|
|
|
|
|
* (which is copy in any case), so crop meta support for system memory
|
|
|
|
|
* is almost pointless */
|
|
|
|
|
gst_query_add_allocation_meta (query,
|
|
|
|
|
GST_VIDEO_CROP_META_API_TYPE, nullptr);
|
2021-08-21 15:33:58 +00:00
|
|
|
}
|
2019-01-30 11:07:29 +00:00
|
|
|
}
|
|
|
|
|
|
2021-01-27 17:02:28 +00:00
|
|
|
/* We need at least 2 buffers because we hold on to the last one for redrawing
|
|
|
|
|
* on window-resize event */
|
|
|
|
|
gst_query_add_allocation_pool (query, pool, size, 2, 0);
|
2019-01-30 11:07:29 +00:00
|
|
|
if (pool)
|
2022-11-28 17:06:11 +00:00
|
|
|
gst_object_unref (pool);
|
2019-01-30 11:07:29 +00:00
|
|
|
|
|
|
|
|
gst_query_add_allocation_meta (query, GST_VIDEO_META_API_TYPE, NULL);
|
2019-12-24 06:54:57 +00:00
|
|
|
gst_query_add_allocation_meta (query,
|
|
|
|
|
GST_VIDEO_OVERLAY_COMPOSITION_META_API_TYPE, NULL);
|
2019-01-30 11:07:29 +00:00
|
|
|
|
|
|
|
|
return TRUE;
|
|
|
|
|
|
|
|
|
|
/* ERRORS */
|
|
|
|
|
no_caps:
|
|
|
|
|
{
|
|
|
|
|
GST_WARNING_OBJECT (self, "no caps specified");
|
|
|
|
|
return FALSE;
|
|
|
|
|
}
|
|
|
|
|
invalid_caps:
|
|
|
|
|
{
|
|
|
|
|
GST_WARNING_OBJECT (self, "invalid caps specified");
|
|
|
|
|
return FALSE;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
return TRUE;
|
|
|
|
|
}
|
|
|
|
|
|
2019-07-18 14:12:13 +00:00
|
|
|
static gboolean
|
|
|
|
|
gst_d3d11_video_sink_query (GstBaseSink * sink, GstQuery * query)
|
|
|
|
|
{
|
|
|
|
|
GstD3D11VideoSink *self = GST_D3D11_VIDEO_SINK (sink);
|
|
|
|
|
|
|
|
|
|
switch (GST_QUERY_TYPE (query)) {
|
|
|
|
|
case GST_QUERY_CONTEXT:
|
|
|
|
|
if (gst_d3d11_handle_context_query (GST_ELEMENT (self), query,
|
|
|
|
|
self->device)) {
|
|
|
|
|
return TRUE;
|
|
|
|
|
}
|
|
|
|
|
break;
|
|
|
|
|
default:
|
|
|
|
|
break;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
return GST_BASE_SINK_CLASS (parent_class)->query (sink, query);
|
|
|
|
|
}
|
|
|
|
|
|
2019-12-02 14:27:42 +00:00
|
|
|
static gboolean
|
|
|
|
|
gst_d3d11_video_sink_unlock (GstBaseSink * sink)
|
2019-01-30 11:07:29 +00:00
|
|
|
{
|
2019-12-02 14:27:42 +00:00
|
|
|
GstD3D11VideoSink *self = GST_D3D11_VIDEO_SINK (sink);
|
2022-08-18 15:05:17 +00:00
|
|
|
GstD3D11CSLockGuard lk (&self->lock);
|
2019-01-30 11:07:29 +00:00
|
|
|
|
2019-12-02 14:27:42 +00:00
|
|
|
if (self->window)
|
2019-12-30 09:58:59 +00:00
|
|
|
gst_d3d11_window_unlock (self->window);
|
|
|
|
|
|
|
|
|
|
return TRUE;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
static gboolean
|
|
|
|
|
gst_d3d11_video_sink_unlock_stop (GstBaseSink * sink)
|
|
|
|
|
{
|
|
|
|
|
GstD3D11VideoSink *self = GST_D3D11_VIDEO_SINK (sink);
|
2022-08-18 15:05:17 +00:00
|
|
|
GstD3D11CSLockGuard lk (&self->lock);
|
2019-12-30 09:58:59 +00:00
|
|
|
|
|
|
|
|
if (self->window)
|
|
|
|
|
gst_d3d11_window_unlock_stop (self->window);
|
2019-01-30 11:07:29 +00:00
|
|
|
|
2019-12-02 14:27:42 +00:00
|
|
|
return TRUE;
|
2019-01-30 11:07:29 +00:00
|
|
|
}
|
|
|
|
|
|
2021-09-17 17:27:51 +00:00
|
|
|
static gboolean
|
|
|
|
|
gst_d3d11_video_sink_event (GstBaseSink * sink, GstEvent * event)
|
|
|
|
|
{
|
|
|
|
|
GstD3D11VideoSink *self = GST_D3D11_VIDEO_SINK (sink);
|
|
|
|
|
|
|
|
|
|
switch (GST_EVENT_TYPE (event)) {
|
|
|
|
|
case GST_EVENT_TAG:{
|
|
|
|
|
GstTagList *taglist;
|
|
|
|
|
gchar *title = nullptr;
|
2022-07-02 16:22:10 +00:00
|
|
|
GstVideoOrientationMethod method = GST_VIDEO_ORIENTATION_IDENTITY;
|
2021-09-17 17:27:51 +00:00
|
|
|
|
|
|
|
|
gst_event_parse_tag (event, &taglist);
|
|
|
|
|
gst_tag_list_get_string (taglist, GST_TAG_TITLE, &title);
|
|
|
|
|
|
|
|
|
|
if (title) {
|
|
|
|
|
const gchar *app_name = g_get_application_name ();
|
|
|
|
|
std::string title_string;
|
2022-08-18 15:05:17 +00:00
|
|
|
GstD3D11CSLockGuard lk (&self->lock);
|
2021-09-17 17:27:51 +00:00
|
|
|
|
|
|
|
|
if (app_name) {
|
|
|
|
|
title_string = std::string (title) + " : " + std::string (app_name);
|
|
|
|
|
} else {
|
|
|
|
|
title_string = std::string (title);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
if (self->window) {
|
|
|
|
|
gst_d3d11_window_set_title (self->window, title_string.c_str ());
|
|
|
|
|
} else {
|
|
|
|
|
g_free (self->title);
|
|
|
|
|
self->title = g_strdup (title_string.c_str ());
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
g_free (title);
|
|
|
|
|
}
|
2022-07-02 16:22:10 +00:00
|
|
|
|
|
|
|
|
if (gst_video_orientation_from_tag (taglist, &method)) {
|
2022-08-18 15:05:17 +00:00
|
|
|
GstD3D11CSLockGuard lk (&self->lock);
|
2022-07-02 16:22:10 +00:00
|
|
|
gst_d3d11_video_sink_set_orientation (self, method, TRUE);
|
|
|
|
|
}
|
2021-09-17 17:27:51 +00:00
|
|
|
break;
|
|
|
|
|
}
|
|
|
|
|
default:
|
|
|
|
|
break;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
return GST_BASE_SINK_CLASS (parent_class)->event (sink, event);
|
|
|
|
|
}
|
|
|
|
|
|
2022-07-02 16:22:10 +00:00
|
|
|
/* called with lock */
|
|
|
|
|
static void
|
|
|
|
|
gst_d3d11_video_sink_set_orientation (GstD3D11VideoSink * self,
|
|
|
|
|
GstVideoOrientationMethod method, gboolean from_tag)
|
|
|
|
|
{
|
|
|
|
|
if (method == GST_VIDEO_ORIENTATION_CUSTOM) {
|
|
|
|
|
GST_WARNING_OBJECT (self, "Unsupported custom orientation");
|
|
|
|
|
return;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
if (from_tag)
|
|
|
|
|
self->tag_method = method;
|
|
|
|
|
else
|
|
|
|
|
self->method = method;
|
|
|
|
|
|
|
|
|
|
if (self->method == GST_VIDEO_ORIENTATION_AUTO) {
|
|
|
|
|
self->selected_method = self->tag_method;
|
|
|
|
|
} else {
|
|
|
|
|
self->selected_method = self->method;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
if (self->window)
|
|
|
|
|
gst_d3d11_window_set_orientation (self->window, self->selected_method);
|
|
|
|
|
}
|
|
|
|
|
|
2021-04-27 12:52:31 +00:00
|
|
|
static void
|
|
|
|
|
gst_d3d11_video_sink_check_device_update (GstD3D11VideoSink * self,
|
|
|
|
|
GstBuffer * buf)
|
|
|
|
|
{
|
|
|
|
|
GstMemory *mem;
|
|
|
|
|
GstD3D11Memory *dmem;
|
|
|
|
|
gboolean update_device = FALSE;
|
|
|
|
|
|
|
|
|
|
/* We have configured window already, cannot update device */
|
|
|
|
|
if (self->window)
|
|
|
|
|
return;
|
|
|
|
|
|
|
|
|
|
mem = gst_buffer_peek_memory (buf, 0);
|
|
|
|
|
if (!gst_is_d3d11_memory (mem))
|
|
|
|
|
return;
|
|
|
|
|
|
|
|
|
|
dmem = GST_D3D11_MEMORY_CAST (mem);
|
|
|
|
|
/* Same device, nothing to do */
|
|
|
|
|
if (dmem->device == self->device)
|
|
|
|
|
return;
|
|
|
|
|
|
|
|
|
|
if (self->adapter < 0) {
|
|
|
|
|
update_device = TRUE;
|
|
|
|
|
} else {
|
|
|
|
|
guint adapter = 0;
|
|
|
|
|
|
|
|
|
|
g_object_get (dmem->device, "adapter", &adapter, NULL);
|
|
|
|
|
/* The same GPU as what user wanted, update */
|
|
|
|
|
if (adapter == (guint) self->adapter)
|
|
|
|
|
update_device = TRUE;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
if (!update_device)
|
|
|
|
|
return;
|
|
|
|
|
|
|
|
|
|
GST_INFO_OBJECT (self, "Updating device %" GST_PTR_FORMAT " -> %"
|
|
|
|
|
GST_PTR_FORMAT, self->device, dmem->device);
|
|
|
|
|
|
|
|
|
|
gst_object_unref (self->device);
|
|
|
|
|
self->device = (GstD3D11Device *) gst_object_ref (dmem->device);
|
|
|
|
|
}
|
|
|
|
|
|
2023-06-14 15:47:11 +00:00
|
|
|
static gboolean
|
|
|
|
|
gst_d3d11_video_sink_foreach_meta (GstBuffer * buffer, GstMeta ** meta,
|
|
|
|
|
GstBuffer * uploaded)
|
|
|
|
|
{
|
|
|
|
|
GstVideoOverlayCompositionMeta *cmeta;
|
|
|
|
|
|
|
|
|
|
if ((*meta)->info->api != GST_VIDEO_OVERLAY_COMPOSITION_META_API_TYPE)
|
|
|
|
|
return TRUE;
|
|
|
|
|
|
|
|
|
|
cmeta = (GstVideoOverlayCompositionMeta *) (*meta);
|
|
|
|
|
if (!cmeta->overlay)
|
|
|
|
|
return TRUE;
|
|
|
|
|
|
|
|
|
|
if (gst_video_overlay_composition_n_rectangles (cmeta->overlay) == 0)
|
|
|
|
|
return TRUE;
|
|
|
|
|
|
|
|
|
|
gst_buffer_add_video_overlay_composition_meta (uploaded, cmeta->overlay);
|
|
|
|
|
|
|
|
|
|
return TRUE;
|
|
|
|
|
}
|
|
|
|
|
|
2019-01-30 11:07:29 +00:00
|
|
|
static GstFlowReturn
|
2022-12-15 17:52:08 +00:00
|
|
|
gst_d3d11_video_sink_prepare (GstBaseSink * sink, GstBuffer * buffer)
|
2019-01-30 11:07:29 +00:00
|
|
|
{
|
|
|
|
|
GstD3D11VideoSink *self = GST_D3D11_VIDEO_SINK (sink);
|
2022-12-15 17:52:08 +00:00
|
|
|
GstFlowReturn ret;
|
2019-12-03 13:54:26 +00:00
|
|
|
|
2022-12-15 17:52:08 +00:00
|
|
|
gst_clear_buffer (&self->prepared_buffer);
|
2021-04-27 12:52:31 +00:00
|
|
|
|
2022-12-15 17:52:08 +00:00
|
|
|
gst_d3d11_video_sink_check_device_update (self, buffer);
|
2021-04-23 10:29:55 +00:00
|
|
|
if (self->caps_updated || !self->window) {
|
|
|
|
|
GstCaps *caps = gst_pad_get_current_caps (GST_BASE_SINK_PAD (sink));
|
|
|
|
|
|
|
|
|
|
/* shouldn't happen */
|
|
|
|
|
if (!caps)
|
|
|
|
|
return GST_FLOW_NOT_NEGOTIATED;
|
|
|
|
|
|
2022-12-14 16:15:10 +00:00
|
|
|
ret = gst_d3d11_video_sink_update_window (self, caps);
|
2021-04-23 10:29:55 +00:00
|
|
|
gst_caps_unref (caps);
|
|
|
|
|
|
2022-12-14 16:15:10 +00:00
|
|
|
if (ret != GST_FLOW_OK)
|
|
|
|
|
return ret;
|
2021-04-23 10:29:55 +00:00
|
|
|
}
|
|
|
|
|
|
2022-12-15 17:52:08 +00:00
|
|
|
if (!gst_is_d3d11_buffer (buffer)) {
|
|
|
|
|
ret = gst_buffer_pool_acquire_buffer (self->pool, &self->prepared_buffer,
|
|
|
|
|
nullptr);
|
|
|
|
|
if (ret != GST_FLOW_OK)
|
|
|
|
|
return ret;
|
|
|
|
|
|
|
|
|
|
gst_d3d11_buffer_copy_into (self->prepared_buffer, buffer, &self->info);
|
|
|
|
|
/* Upload to default texture */
|
|
|
|
|
for (guint i = 0; i < gst_buffer_n_memory (self->prepared_buffer); i++) {
|
|
|
|
|
GstMemory *mem = gst_buffer_peek_memory (self->prepared_buffer, i);
|
|
|
|
|
GstMapInfo info;
|
|
|
|
|
if (!gst_memory_map (mem,
|
|
|
|
|
&info, (GstMapFlags) (GST_MAP_READ | GST_MAP_D3D11))) {
|
|
|
|
|
GST_ERROR_OBJECT (self, "Couldn't map fallback buffer");
|
|
|
|
|
gst_clear_buffer (&self->prepared_buffer);
|
|
|
|
|
return GST_FLOW_ERROR;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
gst_memory_unmap (mem, &info);
|
|
|
|
|
}
|
|
|
|
|
|
2023-06-14 15:47:11 +00:00
|
|
|
gst_buffer_foreach_meta (buffer,
|
|
|
|
|
(GstBufferForeachMetaFunc) gst_d3d11_video_sink_foreach_meta,
|
|
|
|
|
self->prepared_buffer);
|
2022-12-15 17:52:08 +00:00
|
|
|
} else {
|
|
|
|
|
self->prepared_buffer = gst_buffer_ref (buffer);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
return GST_FLOW_OK;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
static GstFlowReturn
|
|
|
|
|
gst_d3d11_video_sink_show_frame (GstVideoSink * sink, GstBuffer * buf)
|
|
|
|
|
{
|
|
|
|
|
GstD3D11VideoSink *self = GST_D3D11_VIDEO_SINK (sink);
|
|
|
|
|
GstFlowReturn ret = GST_FLOW_OK;
|
|
|
|
|
|
|
|
|
|
if (!self->prepared_buffer) {
|
|
|
|
|
GST_ERROR_OBJECT (self, "No prepared buffer");
|
|
|
|
|
return GST_FLOW_ERROR;
|
|
|
|
|
}
|
2019-01-30 11:07:29 +00:00
|
|
|
|
d3d11videosink: Add support for drawing on application's own texture
Add a way to support drawing on application's texture instead of
usual window handle.
To make use of this new feature, application should follow below step.
1) Enable this feature by using "draw-on-shared-texture" property
2) Watch "begin-draw" signal
3) On "begin-draw" signal handler, application can request drawing
by using "draw" signal action. Note that "draw" signal action
should be happen before "begin-draw" signal handler is returned
NOTE 1) For texture sharing, creating a texture with
D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX flag is strongly recommend
if possible because we cannot ensure sync a texture
which was created with D3D11_RESOURCE_MISC_SHARED
and it would cause glitch with ID3D11VideoProcessor use case.
NOTE 2) Direct9Ex doesn't support texture sharing which was
created with D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX. In other words,
D3D11_RESOURCE_MISC_SHARED is the only option for Direct3D11/Direct9Ex interop.
NOTE 3) Because of missing synchronization around ID3D11VideoProcessor,
If shared texture was created with D3D11_RESOURCE_MISC_SHARED,
d3d11videosink might use fallback texture to convert DXVA texture
to normal Direct3D texture. Then converted texture will be
copied to user-provided shared texture.
* Why not use generic appsink approach?
In order for application to be able to store video data
which was produced by GStreamer in application's own texture,
there would be two possible approaches,
one is copying our texture into application's own texture,
and the other is drawing on application's own texture directly.
The former (appsink way) cannot be a zero-copy by nature.
In order to support zero-copy processing, we need to draw on
application's own texture directly.
For example, assume that application wants RGBA texture.
Then we can imagine following case.
"d3d11h264dec ! d3d11convert ! video/x-raw(memory:D3D11Memory),format=RGBA ! appsink"
^
|_ allocate new Direct3D texture for RGBA format
In above case, d3d11convert will allocate new texture(s) for RGBA format
and then application will copy again the our RGBA texutre into
application's own texture. One texture allocation plus per frame GPU copy will hanppen
in that case therefore.
Moreover, in order for application to be able to access
our texture, we need to allocate texture with additional flags for
application's Direct3D11 device to be able to read texture data.
That would be another implementation burden on our side
But with this MR, we can configure pipeline in this way
"d3d11h264dec ! d3d11videosink".
In that way, we can save at least one texture allocation and
per frame texutre copy since d3d11videosink will convert incoming texture
into application's texture format directly without copy.
* What if we expose texture without conversion and application does
conversion by itself?
As mentioned above, for application to be able to access our texture
from application's Direct3D11 device, we need to allocate texture
in a special form. But in some case, that might not be possible.
Also, if a texture belongs to decoder DPB, exposing such texture
to application is unsafe and usual Direct3D11 shader cannot handle
such texture. To convert format, ID3D11VideoProcessor API needs to
be used but that would be a implementation burden for application.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1873>
2020-12-23 14:49:12 +00:00
|
|
|
if (self->draw_on_shared_texture) {
|
2022-08-18 15:05:17 +00:00
|
|
|
GstD3D11CSLockGuard lk (&self->lock);
|
|
|
|
|
|
d3d11videosink: Add support for drawing on application's own texture
Add a way to support drawing on application's texture instead of
usual window handle.
To make use of this new feature, application should follow below step.
1) Enable this feature by using "draw-on-shared-texture" property
2) Watch "begin-draw" signal
3) On "begin-draw" signal handler, application can request drawing
by using "draw" signal action. Note that "draw" signal action
should be happen before "begin-draw" signal handler is returned
NOTE 1) For texture sharing, creating a texture with
D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX flag is strongly recommend
if possible because we cannot ensure sync a texture
which was created with D3D11_RESOURCE_MISC_SHARED
and it would cause glitch with ID3D11VideoProcessor use case.
NOTE 2) Direct9Ex doesn't support texture sharing which was
created with D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX. In other words,
D3D11_RESOURCE_MISC_SHARED is the only option for Direct3D11/Direct9Ex interop.
NOTE 3) Because of missing synchronization around ID3D11VideoProcessor,
If shared texture was created with D3D11_RESOURCE_MISC_SHARED,
d3d11videosink might use fallback texture to convert DXVA texture
to normal Direct3D texture. Then converted texture will be
copied to user-provided shared texture.
* Why not use generic appsink approach?
In order for application to be able to store video data
which was produced by GStreamer in application's own texture,
there would be two possible approaches,
one is copying our texture into application's own texture,
and the other is drawing on application's own texture directly.
The former (appsink way) cannot be a zero-copy by nature.
In order to support zero-copy processing, we need to draw on
application's own texture directly.
For example, assume that application wants RGBA texture.
Then we can imagine following case.
"d3d11h264dec ! d3d11convert ! video/x-raw(memory:D3D11Memory),format=RGBA ! appsink"
^
|_ allocate new Direct3D texture for RGBA format
In above case, d3d11convert will allocate new texture(s) for RGBA format
and then application will copy again the our RGBA texutre into
application's own texture. One texture allocation plus per frame GPU copy will hanppen
in that case therefore.
Moreover, in order for application to be able to access
our texture, we need to allocate texture with additional flags for
application's Direct3D11 device to be able to read texture data.
That would be another implementation burden on our side
But with this MR, we can configure pipeline in this way
"d3d11h264dec ! d3d11videosink".
In that way, we can save at least one texture allocation and
per frame texutre copy since d3d11videosink will convert incoming texture
into application's texture format directly without copy.
* What if we expose texture without conversion and application does
conversion by itself?
As mentioned above, for application to be able to access our texture
from application's Direct3D11 device, we need to allocate texture
in a special form. But in some case, that might not be possible.
Also, if a texture belongs to decoder DPB, exposing such texture
to application is unsafe and usual Direct3D11 shader cannot handle
such texture. To convert format, ID3D11VideoProcessor API needs to
be used but that would be a implementation burden for application.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1873>
2020-12-23 14:49:12 +00:00
|
|
|
self->drawing = TRUE;
|
|
|
|
|
|
|
|
|
|
GST_LOG_OBJECT (self, "Begin drawing");
|
|
|
|
|
|
|
|
|
|
/* Application should call draw method on this callback */
|
2021-04-17 11:37:13 +00:00
|
|
|
g_signal_emit (self, gst_d3d11_video_sink_signals[SIGNAL_BEGIN_DRAW], 0,
|
|
|
|
|
NULL);
|
d3d11videosink: Add support for drawing on application's own texture
Add a way to support drawing on application's texture instead of
usual window handle.
To make use of this new feature, application should follow below step.
1) Enable this feature by using "draw-on-shared-texture" property
2) Watch "begin-draw" signal
3) On "begin-draw" signal handler, application can request drawing
by using "draw" signal action. Note that "draw" signal action
should be happen before "begin-draw" signal handler is returned
NOTE 1) For texture sharing, creating a texture with
D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX flag is strongly recommend
if possible because we cannot ensure sync a texture
which was created with D3D11_RESOURCE_MISC_SHARED
and it would cause glitch with ID3D11VideoProcessor use case.
NOTE 2) Direct9Ex doesn't support texture sharing which was
created with D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX. In other words,
D3D11_RESOURCE_MISC_SHARED is the only option for Direct3D11/Direct9Ex interop.
NOTE 3) Because of missing synchronization around ID3D11VideoProcessor,
If shared texture was created with D3D11_RESOURCE_MISC_SHARED,
d3d11videosink might use fallback texture to convert DXVA texture
to normal Direct3D texture. Then converted texture will be
copied to user-provided shared texture.
* Why not use generic appsink approach?
In order for application to be able to store video data
which was produced by GStreamer in application's own texture,
there would be two possible approaches,
one is copying our texture into application's own texture,
and the other is drawing on application's own texture directly.
The former (appsink way) cannot be a zero-copy by nature.
In order to support zero-copy processing, we need to draw on
application's own texture directly.
For example, assume that application wants RGBA texture.
Then we can imagine following case.
"d3d11h264dec ! d3d11convert ! video/x-raw(memory:D3D11Memory),format=RGBA ! appsink"
^
|_ allocate new Direct3D texture for RGBA format
In above case, d3d11convert will allocate new texture(s) for RGBA format
and then application will copy again the our RGBA texutre into
application's own texture. One texture allocation plus per frame GPU copy will hanppen
in that case therefore.
Moreover, in order for application to be able to access
our texture, we need to allocate texture with additional flags for
application's Direct3D11 device to be able to read texture data.
That would be another implementation burden on our side
But with this MR, we can configure pipeline in this way
"d3d11h264dec ! d3d11videosink".
In that way, we can save at least one texture allocation and
per frame texutre copy since d3d11videosink will convert incoming texture
into application's texture format directly without copy.
* What if we expose texture without conversion and application does
conversion by itself?
As mentioned above, for application to be able to access our texture
from application's Direct3D11 device, we need to allocate texture
in a special form. But in some case, that might not be possible.
Also, if a texture belongs to decoder DPB, exposing such texture
to application is unsafe and usual Direct3D11 shader cannot handle
such texture. To convert format, ID3D11VideoProcessor API needs to
be used but that would be a implementation burden for application.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1873>
2020-12-23 14:49:12 +00:00
|
|
|
|
|
|
|
|
GST_LOG_OBJECT (self, "End drawing");
|
|
|
|
|
self->drawing = FALSE;
|
|
|
|
|
} else {
|
2023-02-20 15:27:27 +00:00
|
|
|
gst_d3d11_window_show (self->window);
|
2022-12-15 17:52:08 +00:00
|
|
|
ret = gst_d3d11_window_render (self->window, self->prepared_buffer);
|
d3d11videosink: Add support for drawing on application's own texture
Add a way to support drawing on application's texture instead of
usual window handle.
To make use of this new feature, application should follow below step.
1) Enable this feature by using "draw-on-shared-texture" property
2) Watch "begin-draw" signal
3) On "begin-draw" signal handler, application can request drawing
by using "draw" signal action. Note that "draw" signal action
should be happen before "begin-draw" signal handler is returned
NOTE 1) For texture sharing, creating a texture with
D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX flag is strongly recommend
if possible because we cannot ensure sync a texture
which was created with D3D11_RESOURCE_MISC_SHARED
and it would cause glitch with ID3D11VideoProcessor use case.
NOTE 2) Direct9Ex doesn't support texture sharing which was
created with D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX. In other words,
D3D11_RESOURCE_MISC_SHARED is the only option for Direct3D11/Direct9Ex interop.
NOTE 3) Because of missing synchronization around ID3D11VideoProcessor,
If shared texture was created with D3D11_RESOURCE_MISC_SHARED,
d3d11videosink might use fallback texture to convert DXVA texture
to normal Direct3D texture. Then converted texture will be
copied to user-provided shared texture.
* Why not use generic appsink approach?
In order for application to be able to store video data
which was produced by GStreamer in application's own texture,
there would be two possible approaches,
one is copying our texture into application's own texture,
and the other is drawing on application's own texture directly.
The former (appsink way) cannot be a zero-copy by nature.
In order to support zero-copy processing, we need to draw on
application's own texture directly.
For example, assume that application wants RGBA texture.
Then we can imagine following case.
"d3d11h264dec ! d3d11convert ! video/x-raw(memory:D3D11Memory),format=RGBA ! appsink"
^
|_ allocate new Direct3D texture for RGBA format
In above case, d3d11convert will allocate new texture(s) for RGBA format
and then application will copy again the our RGBA texutre into
application's own texture. One texture allocation plus per frame GPU copy will hanppen
in that case therefore.
Moreover, in order for application to be able to access
our texture, we need to allocate texture with additional flags for
application's Direct3D11 device to be able to read texture data.
That would be another implementation burden on our side
But with this MR, we can configure pipeline in this way
"d3d11h264dec ! d3d11videosink".
In that way, we can save at least one texture allocation and
per frame texutre copy since d3d11videosink will convert incoming texture
into application's texture format directly without copy.
* What if we expose texture without conversion and application does
conversion by itself?
As mentioned above, for application to be able to access our texture
from application's Direct3D11 device, we need to allocate texture
in a special form. But in some case, that might not be possible.
Also, if a texture belongs to decoder DPB, exposing such texture
to application is unsafe and usual Direct3D11 shader cannot handle
such texture. To convert format, ID3D11VideoProcessor API needs to
be used but that would be a implementation burden for application.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1873>
2020-12-23 14:49:12 +00:00
|
|
|
}
|
2020-11-21 19:39:57 +00:00
|
|
|
|
2019-01-30 11:07:29 +00:00
|
|
|
if (ret == GST_D3D11_WINDOW_FLOW_CLOSED) {
|
|
|
|
|
GST_ELEMENT_ERROR (self, RESOURCE, NOT_FOUND,
|
|
|
|
|
("Output window was closed"), (NULL));
|
|
|
|
|
|
|
|
|
|
ret = GST_FLOW_ERROR;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
return ret;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/* VideoOverlay interface */
|
|
|
|
|
static void
|
|
|
|
|
gst_d3d11_video_sink_set_window_handle (GstVideoOverlay * overlay,
|
|
|
|
|
guintptr window_id)
|
|
|
|
|
{
|
|
|
|
|
GstD3D11VideoSink *self = GST_D3D11_VIDEO_SINK (overlay);
|
|
|
|
|
|
|
|
|
|
GST_DEBUG ("set window handle %" G_GUINTPTR_FORMAT, window_id);
|
|
|
|
|
|
|
|
|
|
self->window_id = window_id;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
static void
|
|
|
|
|
gst_d3d11_video_sink_set_render_rectangle (GstVideoOverlay * overlay, gint x,
|
|
|
|
|
gint y, gint width, gint height)
|
|
|
|
|
{
|
|
|
|
|
GstD3D11VideoSink *self = GST_D3D11_VIDEO_SINK (overlay);
|
2022-08-18 15:05:17 +00:00
|
|
|
GstD3D11CSLockGuard lk (&self->lock);
|
2019-01-30 11:07:29 +00:00
|
|
|
|
|
|
|
|
GST_DEBUG_OBJECT (self,
|
|
|
|
|
"render rect x: %d, y: %d, width: %d, height %d", x, y, width, height);
|
|
|
|
|
|
|
|
|
|
if (self->window) {
|
2021-07-30 15:59:14 +00:00
|
|
|
GstVideoRectangle rect;
|
|
|
|
|
|
|
|
|
|
rect.x = x;
|
|
|
|
|
rect.y = y;
|
|
|
|
|
rect.w = width;
|
|
|
|
|
rect.h = height;
|
|
|
|
|
|
|
|
|
|
self->render_rect = rect;
|
|
|
|
|
|
|
|
|
|
gst_d3d11_window_set_render_rectangle (self->window, &rect);
|
2019-01-30 11:07:29 +00:00
|
|
|
} else {
|
|
|
|
|
self->render_rect.x = x;
|
|
|
|
|
self->render_rect.y = y;
|
|
|
|
|
self->render_rect.w = width;
|
|
|
|
|
self->render_rect.h = height;
|
|
|
|
|
self->pending_render_rect = TRUE;
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
static void
|
|
|
|
|
gst_d3d11_video_sink_expose (GstVideoOverlay * overlay)
|
|
|
|
|
{
|
|
|
|
|
GstD3D11VideoSink *self = GST_D3D11_VIDEO_SINK (overlay);
|
2022-08-18 15:05:17 +00:00
|
|
|
GstD3D11CSLockGuard lk (&self->lock);
|
2019-01-30 11:07:29 +00:00
|
|
|
|
2022-07-02 16:18:19 +00:00
|
|
|
if (self->window && self->window->swap_chain)
|
|
|
|
|
gst_d3d11_window_render (self->window, nullptr);
|
2019-01-30 11:07:29 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
static void
|
|
|
|
|
gst_d3d11_video_sink_video_overlay_init (GstVideoOverlayInterface * iface)
|
|
|
|
|
{
|
|
|
|
|
iface->set_window_handle = gst_d3d11_video_sink_set_window_handle;
|
|
|
|
|
iface->set_render_rectangle = gst_d3d11_video_sink_set_render_rectangle;
|
|
|
|
|
iface->expose = gst_d3d11_video_sink_expose;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/* Navigation interface */
|
|
|
|
|
static void
|
|
|
|
|
gst_d3d11_video_sink_navigation_send_event (GstNavigation * navigation,
|
2022-02-14 15:08:23 +00:00
|
|
|
GstEvent * event)
|
2019-01-30 11:07:29 +00:00
|
|
|
{
|
|
|
|
|
GstD3D11VideoSink *self = GST_D3D11_VIDEO_SINK (navigation);
|
|
|
|
|
|
2021-07-30 16:05:47 +00:00
|
|
|
/* TODO: add support for translating native coordinate and video coordinate
|
|
|
|
|
* when force-aspect-ratio is set */
|
2019-01-30 11:07:29 +00:00
|
|
|
if (event) {
|
2021-07-30 16:05:47 +00:00
|
|
|
gboolean handled;
|
|
|
|
|
|
2019-01-30 11:07:29 +00:00
|
|
|
gst_event_ref (event);
|
|
|
|
|
handled = gst_pad_push_event (GST_VIDEO_SINK_PAD (self), event);
|
|
|
|
|
|
|
|
|
|
if (!handled)
|
|
|
|
|
gst_element_post_message (GST_ELEMENT_CAST (self),
|
|
|
|
|
gst_navigation_message_new_event (GST_OBJECT_CAST (self), event));
|
|
|
|
|
|
|
|
|
|
gst_event_unref (event);
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
static void
|
|
|
|
|
gst_d3d11_video_sink_navigation_init (GstNavigationInterface * iface)
|
|
|
|
|
{
|
2022-02-14 15:08:23 +00:00
|
|
|
iface->send_event_simple = gst_d3d11_video_sink_navigation_send_event;
|
2019-01-30 11:07:29 +00:00
|
|
|
}
|
d3d11videosink: Add support for drawing on application's own texture
Add a way to support drawing on application's texture instead of
usual window handle.
To make use of this new feature, application should follow below step.
1) Enable this feature by using "draw-on-shared-texture" property
2) Watch "begin-draw" signal
3) On "begin-draw" signal handler, application can request drawing
by using "draw" signal action. Note that "draw" signal action
should be happen before "begin-draw" signal handler is returned
NOTE 1) For texture sharing, creating a texture with
D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX flag is strongly recommend
if possible because we cannot ensure sync a texture
which was created with D3D11_RESOURCE_MISC_SHARED
and it would cause glitch with ID3D11VideoProcessor use case.
NOTE 2) Direct9Ex doesn't support texture sharing which was
created with D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX. In other words,
D3D11_RESOURCE_MISC_SHARED is the only option for Direct3D11/Direct9Ex interop.
NOTE 3) Because of missing synchronization around ID3D11VideoProcessor,
If shared texture was created with D3D11_RESOURCE_MISC_SHARED,
d3d11videosink might use fallback texture to convert DXVA texture
to normal Direct3D texture. Then converted texture will be
copied to user-provided shared texture.
* Why not use generic appsink approach?
In order for application to be able to store video data
which was produced by GStreamer in application's own texture,
there would be two possible approaches,
one is copying our texture into application's own texture,
and the other is drawing on application's own texture directly.
The former (appsink way) cannot be a zero-copy by nature.
In order to support zero-copy processing, we need to draw on
application's own texture directly.
For example, assume that application wants RGBA texture.
Then we can imagine following case.
"d3d11h264dec ! d3d11convert ! video/x-raw(memory:D3D11Memory),format=RGBA ! appsink"
^
|_ allocate new Direct3D texture for RGBA format
In above case, d3d11convert will allocate new texture(s) for RGBA format
and then application will copy again the our RGBA texutre into
application's own texture. One texture allocation plus per frame GPU copy will hanppen
in that case therefore.
Moreover, in order for application to be able to access
our texture, we need to allocate texture with additional flags for
application's Direct3D11 device to be able to read texture data.
That would be another implementation burden on our side
But with this MR, we can configure pipeline in this way
"d3d11h264dec ! d3d11videosink".
In that way, we can save at least one texture allocation and
per frame texutre copy since d3d11videosink will convert incoming texture
into application's texture format directly without copy.
* What if we expose texture without conversion and application does
conversion by itself?
As mentioned above, for application to be able to access our texture
from application's Direct3D11 device, we need to allocate texture
in a special form. But in some case, that might not be possible.
Also, if a texture belongs to decoder DPB, exposing such texture
to application is unsafe and usual Direct3D11 shader cannot handle
such texture. To convert format, ID3D11VideoProcessor API needs to
be used but that would be a implementation burden for application.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1873>
2020-12-23 14:49:12 +00:00
|
|
|
|
|
|
|
|
static gboolean
|
|
|
|
|
gst_d3d11_video_sink_draw_action (GstD3D11VideoSink * self,
|
|
|
|
|
gpointer shared_handle, guint texture_misc_flags,
|
|
|
|
|
guint64 acquire_key, guint64 release_key)
|
|
|
|
|
{
|
|
|
|
|
GstFlowReturn ret;
|
|
|
|
|
g_return_val_if_fail (shared_handle != NULL, FALSE);
|
|
|
|
|
|
|
|
|
|
if (!self->draw_on_shared_texture) {
|
|
|
|
|
GST_ERROR_OBJECT (self, "Invalid draw call, we are drawing on window");
|
|
|
|
|
return FALSE;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
if (!shared_handle) {
|
|
|
|
|
GST_ERROR_OBJECT (self, "Invalid handle");
|
|
|
|
|
return FALSE;
|
|
|
|
|
}
|
|
|
|
|
|
2022-08-18 15:05:17 +00:00
|
|
|
GstD3D11CSLockGuard lk (&self->lock);
|
2022-12-15 17:52:08 +00:00
|
|
|
if (!self->drawing || !self->prepared_buffer) {
|
d3d11videosink: Add support for drawing on application's own texture
Add a way to support drawing on application's texture instead of
usual window handle.
To make use of this new feature, application should follow below step.
1) Enable this feature by using "draw-on-shared-texture" property
2) Watch "begin-draw" signal
3) On "begin-draw" signal handler, application can request drawing
by using "draw" signal action. Note that "draw" signal action
should be happen before "begin-draw" signal handler is returned
NOTE 1) For texture sharing, creating a texture with
D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX flag is strongly recommend
if possible because we cannot ensure sync a texture
which was created with D3D11_RESOURCE_MISC_SHARED
and it would cause glitch with ID3D11VideoProcessor use case.
NOTE 2) Direct9Ex doesn't support texture sharing which was
created with D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX. In other words,
D3D11_RESOURCE_MISC_SHARED is the only option for Direct3D11/Direct9Ex interop.
NOTE 3) Because of missing synchronization around ID3D11VideoProcessor,
If shared texture was created with D3D11_RESOURCE_MISC_SHARED,
d3d11videosink might use fallback texture to convert DXVA texture
to normal Direct3D texture. Then converted texture will be
copied to user-provided shared texture.
* Why not use generic appsink approach?
In order for application to be able to store video data
which was produced by GStreamer in application's own texture,
there would be two possible approaches,
one is copying our texture into application's own texture,
and the other is drawing on application's own texture directly.
The former (appsink way) cannot be a zero-copy by nature.
In order to support zero-copy processing, we need to draw on
application's own texture directly.
For example, assume that application wants RGBA texture.
Then we can imagine following case.
"d3d11h264dec ! d3d11convert ! video/x-raw(memory:D3D11Memory),format=RGBA ! appsink"
^
|_ allocate new Direct3D texture for RGBA format
In above case, d3d11convert will allocate new texture(s) for RGBA format
and then application will copy again the our RGBA texutre into
application's own texture. One texture allocation plus per frame GPU copy will hanppen
in that case therefore.
Moreover, in order for application to be able to access
our texture, we need to allocate texture with additional flags for
application's Direct3D11 device to be able to read texture data.
That would be another implementation burden on our side
But with this MR, we can configure pipeline in this way
"d3d11h264dec ! d3d11videosink".
In that way, we can save at least one texture allocation and
per frame texutre copy since d3d11videosink will convert incoming texture
into application's texture format directly without copy.
* What if we expose texture without conversion and application does
conversion by itself?
As mentioned above, for application to be able to access our texture
from application's Direct3D11 device, we need to allocate texture
in a special form. But in some case, that might not be possible.
Also, if a texture belongs to decoder DPB, exposing such texture
to application is unsafe and usual Direct3D11 shader cannot handle
such texture. To convert format, ID3D11VideoProcessor API needs to
be used but that would be a implementation burden for application.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1873>
2020-12-23 14:49:12 +00:00
|
|
|
GST_WARNING_OBJECT (self, "Nothing to draw");
|
|
|
|
|
return FALSE;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
GST_LOG_OBJECT (self, "Drawing on shared handle %p, MiscFlags: 0x%x"
|
|
|
|
|
", acquire key: %" G_GUINT64_FORMAT ", release key: %"
|
|
|
|
|
G_GUINT64_FORMAT, shared_handle, texture_misc_flags, acquire_key,
|
|
|
|
|
release_key);
|
|
|
|
|
|
|
|
|
|
ret = gst_d3d11_window_render_on_shared_handle (self->window,
|
2022-12-15 17:52:08 +00:00
|
|
|
self->prepared_buffer, shared_handle, texture_misc_flags, acquire_key,
|
d3d11videosink: Add support for drawing on application's own texture
Add a way to support drawing on application's texture instead of
usual window handle.
To make use of this new feature, application should follow below step.
1) Enable this feature by using "draw-on-shared-texture" property
2) Watch "begin-draw" signal
3) On "begin-draw" signal handler, application can request drawing
by using "draw" signal action. Note that "draw" signal action
should be happen before "begin-draw" signal handler is returned
NOTE 1) For texture sharing, creating a texture with
D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX flag is strongly recommend
if possible because we cannot ensure sync a texture
which was created with D3D11_RESOURCE_MISC_SHARED
and it would cause glitch with ID3D11VideoProcessor use case.
NOTE 2) Direct9Ex doesn't support texture sharing which was
created with D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX. In other words,
D3D11_RESOURCE_MISC_SHARED is the only option for Direct3D11/Direct9Ex interop.
NOTE 3) Because of missing synchronization around ID3D11VideoProcessor,
If shared texture was created with D3D11_RESOURCE_MISC_SHARED,
d3d11videosink might use fallback texture to convert DXVA texture
to normal Direct3D texture. Then converted texture will be
copied to user-provided shared texture.
* Why not use generic appsink approach?
In order for application to be able to store video data
which was produced by GStreamer in application's own texture,
there would be two possible approaches,
one is copying our texture into application's own texture,
and the other is drawing on application's own texture directly.
The former (appsink way) cannot be a zero-copy by nature.
In order to support zero-copy processing, we need to draw on
application's own texture directly.
For example, assume that application wants RGBA texture.
Then we can imagine following case.
"d3d11h264dec ! d3d11convert ! video/x-raw(memory:D3D11Memory),format=RGBA ! appsink"
^
|_ allocate new Direct3D texture for RGBA format
In above case, d3d11convert will allocate new texture(s) for RGBA format
and then application will copy again the our RGBA texutre into
application's own texture. One texture allocation plus per frame GPU copy will hanppen
in that case therefore.
Moreover, in order for application to be able to access
our texture, we need to allocate texture with additional flags for
application's Direct3D11 device to be able to read texture data.
That would be another implementation burden on our side
But with this MR, we can configure pipeline in this way
"d3d11h264dec ! d3d11videosink".
In that way, we can save at least one texture allocation and
per frame texutre copy since d3d11videosink will convert incoming texture
into application's texture format directly without copy.
* What if we expose texture without conversion and application does
conversion by itself?
As mentioned above, for application to be able to access our texture
from application's Direct3D11 device, we need to allocate texture
in a special form. But in some case, that might not be possible.
Also, if a texture belongs to decoder DPB, exposing such texture
to application is unsafe and usual Direct3D11 shader cannot handle
such texture. To convert format, ID3D11VideoProcessor API needs to
be used but that would be a implementation burden for application.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1873>
2020-12-23 14:49:12 +00:00
|
|
|
release_key);
|
|
|
|
|
|
|
|
|
|
return ret == GST_FLOW_OK;
|
|
|
|
|
}
|
