Current shader code is not compatible with HLSL profile "ps_4_0_level_9_3"
or lower. So d3dcompiler cannot compile our shader code in that case.
Note that VirtualBox is one known driver which doesn't support currently
implemented shader code.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1343>
d3d11videosink has an advantage over d3dvideosink, such as
* Zero-copy playback with d3d11 decoders
* HDR rendering with 10-bit format/swapchain support
* UWP support
* Any system memory alignment/padding can be supported
* User can select target GPU device
And old d3dvideosink's functionality (e.g., navigation event, overlaycomposition)
can be covered by d3d11videosink
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1311>
If parent and child windows are running on different thread,
there is always a chance to cause deadlock as DefWindowProc() call
from child window thread might be blocked until the message
is handled by parent's window procedure.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1299>
GObject::dispose method can be called multiple times. As win32 d3d11window
has an internal thread and because GObject::dispose method could be called from the
thread, it might cause problems such as trying to join self-thread
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1299>
GstD3D11ColorConverter implementation is able to rescale video as well.
By doing colorspace conversion and rescale at once, we can save
one cycle of shader pipeline which will can save GPU resource.
Since this element can support color space conversion and rescale,
it's renamed as d3d11convert
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1275>
D3D11_CREATE_DEVICE_DEBUG flag will be used while creating d3d11 device
to activate debug layer. However, if system doesn't support the
debug layer for some reason, we should try to create d3d11 device
without the flag. Debug layer should be optional for device creation.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1217>
We should directly check the values of the `debug` and `optimization`
options instead.
`get_option('buildtype')` will return `'custom'` for most combinations
of `-Doptimization` and `-Ddebug`, but those two will always be set
correctly if only `-Dbuildtype` is set. So we should look at those
options directly.
For the two-way mapping between `buildtype` and `optimization`
+ `debug`, see this table:
https://mesonbuild.com/Builtin-options.html#build-type-options
DXVA supports two kinds of texture structure for DPB, one is
"1) texture array" and the other is "2) array of texture".
1) is a type of texture which is single ID3D11Texture2D object having
ArraySize greater than one. So the ID3D11Texture2D itself is a set of texture.
Each sub texture of this type mush have identical resolution, format and so on,
and the number of sub texture in a texture array is fixed.
2) is an array of usual ID3D11Texture2D object. That means each
ID3D11Texture2D is independent each other and might have different resolution as well.
Moreover, we can modify the number of frames of the array dynamically.
This type is more flexible than "1) texture array" in terms of dynamic
behavior and also this type of texture can be used for shader resource view
but "1) texture array" couldn't be.
If "2) array of texture" is supported by driver, DXVA spec is saying that
it's preferred format over "1) texture array" in terms of performance.
The set of supported color space by DXGI is not full combination of
our colorimetry. That means we should convert color space to one
of supported color space by DXGI. This commit modifies the color space
selection step so that d3d11window can find the best matching DXGI color space
first and then the selected input/output color space will be referenced
by shader and/or d3d11videoprocessor.
VP9 codec allows resizing reference frame by spec. Handling this case
is a bit tricky especially when the resizing happens on non-keyframe,
because pre-allocated decoder textures (i.e., dpb) have negotiated
resolution and to change resolution meanwhile decoding on non-keyframe,
each texture might need to be re-created, copied to new dpb somehow,
and re-negotiated with downstream.
Due to the complicated requirement of negotiation driven
resizing handling, this commit adds shader into d3d11decoder object
to resize only corresponding frames. Note that if the resolution change
is detected on keyframe, decoder will re-negotiate with downstream.
Not only any textures for decoder output view, any destination texture
which would be copied from decoder output texture need to be aligned too.
Otherwise driver sometimes crashed/hung (not sure why).
Resolution of NV12, P010, and P016 formats must be multiple of two.
Otherwise texture cannot be created. Instead of doing this alignment
per API consumer side, do this in buffer pool for simplicity.
Now that the system_frame_number is saved on the pictures we can use
gst_video_decoder_get_frame() helper instead of getting the full list
and looping over it.
On new_segment, the decoder is expected to negotiate. The decoder may want to
pre-allocate the needed buffers. Pass the max_dpb_size as this is needed to
determin how many buffers should be allocated.
This introduce a library which contains a set of base classes which
handles the parsing and the state tracking for the purpose of decoding
different CODECs. Currently H264, H265 and VP9 are supported. These
bases classes are used to decode with low level decoding API like DXVA,
NVDEC, VDPAU, VAAPI and V4L2 State Less decoders. The new library is
named gstreamer-codecs-1.0 / libgstcodecs.
This commit moves parsing code for superframe and frame header into
handle_frame() method, and removes parse() implementation from vp9decoder
baseclass.
The combination of
- multiple frames are packed in a given input buffer (i.e., superframe)
- reverse playback
seems to be complicated and also it doesn't work as intended in some case
* Remove redundant variables for width/height and par from GstD3D11Window.
GstVideoInfo holds all the values.
* Don't need to pass par to gst_d3d11_window_prepare().
It will be parsed from caps again
* Remove duplicated math
Fixing regression of the commit 9dada90108
gst_d3d11_result() will print warning message when HRESULT != S_OK.
However, since the retry is trivial stuff, check hr == E_PENDING first
and do not warn it.
The DXGI_PRESENT_ALLOW_TEARING flag might cause unexpected tearing
side effect. Setting it in fullscreen mode only seems to be
the correct usage as in the Microsoft's direct3d examples.
DXVA spec is saying that the size of bitstream buffer provided by hardware decoder
should be 128 bytes aligned. And also the host software decoder should
align the size of written buffer to 128 bytes. That means if the slice
(or frame in case of VP9) size is not aligned with 128 bytes,
the rest of non 128 bytes aligned memory should be zero-padded.
In addition to aligning implementation, some variables are renamed
to be more intuitive by this commit.
This implementation is similar to what we've done for nvcodec plugin.
Since supported resolution, profiles, and formats are device dependent ones,
single template caps cannot represent them, so this modification
will help autoplugging and fallback.
Note that the legacy gpu list and list of resolution to query were
taken from chromium's code.
gst_video_frame_copy will copy input frame to stating texture
of fallback frame. Then, we need to map fallback texture with GST_MAP_D3D11
flag to upload the staging texture to render texture. Otherwise
the render texture wouldn't be updated.
Source texture (decoder view) might be larger than destination (staging) texture.
In that case, D3D11_BOX structure should be passed to CopySubresourceRegion method
in order to specify the exact target area.
DXGI_SWAP_EFFECT_DISCARD cannot be used with dirty rect drawing feature
of IDXGISwapChain1::Present().
Note that IDXGISwapChain1 interface is available on Platform Update for Windows 7
and DXGI_SWAP_EFFECT_FLIP_SEQUENTIAL is also the case.
Use resolution specified in caps for input_rect instead of
passed width and height value. The width and height might be modified
ones by d3d11videosink, then frame resolution might be different.
* Move decoding process to handle_frame
* Remove GstVideoDecoder::parse implementation
* Clarify flush/drain/finish usage
In forward playback case, have_frame() call will be followed by
handle_frame() but reverse playback is not the case.
To ensure GstVideoCodecFrame, the decoding process should be placed inside
of handle_frame(), instead of parse().
Since we don't support alignment=nal, the parse() implementation is not worth.
In order to fix broken reverse playback, let's remove the parse()
implementation and revisit it when adding alignment=nal support.
... and remove unused start, stop method from subclass.
Current implementation does not require subclass specific behavior
for the handle_frame() method.
Actually our buffer pool size and the number of backbuffer are
independent. In case of reverse playback, upstream might request
a lot of buffers (up to GOP size).
d3d11window holds one buffer to redraw client area per resize event.
When the input format is being changed, this buffer should be cleared
to avoid mismatch beween newly configured shader/videoprocessor and
the format of previously cached buffer.
Because the size of texture array cannot be updated dynamically,
allocator should block the allocation request. This cannot be
done at buffer pool side if this d3d11 memory is shared among
multiple buffer objects. Note that setting NO_SHARE flag to
d3d11 memory is very inefficient. It would cause most likey
copy of the d3d11 texture.
...for color space conversion if available
ID3D11VideoProcessor is equivalent to DXVA-HD video processor
which might use specialized blocks for video processing
instead of general GPU resource. In addition to that feature,
we need to use this API for color space conversion of DXVA2 decoder
output memory, because any d3d11 texture arrays that were
created with D3D11_BIND_DECODER cannot be used for shader resource.
This is prework for d3d11decoder zero-copy rendering and also
for conditional HDR tone-map support.
Note that some Intel platform is known to support tone-mapping
at the driver level using this API on Windows 10.
Don't specify the resolution of backbuffer. Then dxgi will let us know the
actual client area. When upstream resolution is chagned, updating the size
of backbuffer without the consideration for client size would cause mismatch
between them.
Use consistent memory layout between dxva and other shader use case.
For example, use DXGI_FORMAT_NV12 texture format instead of
two textures with DXGI_FORMAT_R8_UNORM and DXGI_FORMAT_R8G8_UNORM.
This reverts commit ddd13fc7c0
Dynamic usage can reduce the number of copy per frame but make
things complicated and the benefit seems to not significant.
Also since we don't provide _map() method for the dynamic usage,
application cannot read buffers which make "last-sample" property
unusable in case of d3d11videosink.
Although the target platform of D3D11 decoding API are both desktop and UWP app,
DXVA header is blocked by "WINAPI_FAMILY_PARTITION(WINAPI_PARTITION_DESKTOP)"
which is meaning that that's only for desktop app.
To workaround this inconsistent annoyingness, we need to define WINAPI_PARTITION_DESKTOP
regardless of target WinAPI partition.
A ID3D11Texture2D memory can consist of multiple planes with array.
For array typed memory, GstD3D11Allocator will allocate new GstD3D11Memory
with increased reference count to the ID3D11Texture2D but different array index.
Even if one of downstream d3d11 elements can support dynamic-usage memory,
another one might not support it. Also, to support dynamic-usage,
both upstream and downstream d3d11device must be the same object.
If d3d11colorconvert element is configured, do color space conversion
regardless of the device type whether it's S/W emulation or real H/W.
Since d3d11colorconvert is no more a child of d3d11videosinkbin,
we don't need this behavior. Note that previous code was added to
avoid color space conversion from d3d11videosink if no hardware
device is available (S/W emulation of d3d11 is too slow).
d3d11upload should be able to support upstream d3d11 memory, not only system memory.
Fix for following pipeline
d3d11upload ! "video/x-raw(memory:D3D11Memory)" ! d3d11videosink
borderless top-most style full screen mode support.
Basically fullscreen toggle mode is disabled by default. To enable it
use "fullscreen-toggle-mode" property to allow fullscreen mode change
by user input and/or property.
In some cases, rendering and dxgi (e.g., swapchain) APIs should be
called from window message pump thread, but current design (dedicated d3d11 thread)
make it impossible. To solve it, change concurrency model to locking based one
from single-thread model.
In earlier implementation of d3d11videosink where no shader was implemented,
the aspect ratio and render size were adjusted by manipulating the backbuffer size
with unintuitive formula. Since now we do color conversion and resize using
shader, we can remove the hack.
... and use SetParent() WIN32 API when external window is used.
Depending on DXGI swap effect, the external window might not be
reusable by another backend. To preserve the external window's property
and setting, drawing to internal window seems to be safer way.
If d3d11window does not convert format internally, shader resource view
is not required. Note that shader resource view is used for
color conversion using shader but when conversion is not required,
we just copy input input texture to backbuffer.
D3D11 dynamic texture is a special memory type, which is mainly used for
frequent CPU write access to the texture. For now, this texture type
does not support gst_memory_{map,unmap}
* Create staging texture only when the CPU access is requested.
Note that we should avoid the CPU access to d3d11 memory as mush as possible.
Incoming d3d11upload and d3d11download will take this GPU memory upload/download.
* Upload/Download texture memory from/to staging only if it needed, similar to
GstGL PBO implementation.
* Define more dxgi formats for future usage (e.g., color conversion, dxva2 decoder).
Because I420_* formats are not supported formats by dxgi, each plane should
be handled likewise GstGL separately, but NV12/P10 formats might be supported ones.
So we decide the number of d3d11memory per GstBuffer for video memory depending on
OS version and dxgi format. For instance, if NV12 is supported by OS,
only one d3d11memory with DXGI_FORMAT_NV12 texture can be allocated by this commit.
One use case of such texture is DXVA. In case DXVA decoder, it might need to produce decoded data
to one DXGI_FORMAT_NV12 instead of seperate Y and UV planes.
Such behavior will be controlled via configuration of GstD3D11BufferPool and
default configuration is separate resources per plane.
Depending on selected feature level, d3d11 API usage can be different.
Instead of querying the selected feature level by user whenever required,
store it once by d3d11device.
Do not accept any GstD3D11Device context which has different adapter
index from the required one. For example, if a d3d11 element is expecting
d3d11 device with adapter 1 (i.e., the second GPU), any d3d11 device
context having different adapter could not be shared with
the d3d11 element.
Make them consistent with cuda context utils functions.
Put in-only parameter before all in-out parameters, and add _handle()
suffix to native handle getter functions.
By passing NULL to `g_signal_new` instead of a marshaller, GLib will
actually internally optimize the signal (if the marshaller is available
in GLib itself) by also setting the valist marshaller. This makes the
signal emission a bit more performant than the regular marshalling,
which still needs to box into `GValue` and call libffi in case of a
generic marshaller.
Note that for custom marshallers, one would use
`g_signal_set_va_marshaller()` with the valist marshaller instead.
... instead of calculated display ratio from given PAR and DAR.
d3d11window calculates output display ratio
to decide padding area per window resize event. In the formula,
actual PAR is required to handle both 1:1 PAR and non-1:1 PAR.
Fix unexpected cropping with non 1:1 pixel aspect-ratio.
The actual buffer width/height should be passed to gst_d3d11_window_render(),
instead of the calculated resolution. The width/height
values are parameters for copying d3d11 video memory.
Also, aspect-ratio should be considered on resize callback
to decide render rectangle size.
Direct3D11 was shipped as part of Windows7 and it's obviously
primary graphics API on Windows.
This plugin includes HDR10 rendering if following requirements are satisfied
* IDXGISwapChain4::SetHDRMetaData is available (decleared in dxgi1_5.h)
* Display can support DXGI_COLOR_SPACE_RGB_FULL_G2084_NONE_P2020 color space
* Upstream provides 10 bitdepth format with smpte-st 2084 static metadata
