Enable zero-copy if downstream proposed pool and therefore decoder
can know the amount of buffer required by downstream.
Otherwise decoder will copy when our DPB pool has no sufficient
buffers for later decoding operation.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/2097>
Major changes:
* GstD3D11Allocator: This allocator is now device-independent object
which can allocate GstD3D11Memory object for any GstD3D11Device.
User can get this object via gst_allocator_find(GST_D3D11_MEMORY_NAME)
* GstD3D11PoolAllocator: A new allocator implementation for texture pool.
From now on GstD3D11BufferPool will make use of this memory pool allocator
to avoid frequent texture reallocation. That usually happens because
of buffer copy (gst_buffer_make_writable for example)
In addition to that, GstD3D11BufferPool will provide GstBuffer with
GstVideoMeta, because CPU access to a GstD3D11Memory without GstVideoMeta
is almost impossible since GPU drivers needs padding for stride alignment.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/2097>
We've been doing retry with 1ms sleep if DecoderBeginFrame()
returned E_PENDING which means application should call
DecoderBeginFrame() again because GPU is busy.
The 1ms sleep() during retry would result in usually about 15ms delay
in reality because of bad clock precision on Windows.
To improve throughput performance, this commit will enable
high precision clock only for NVIDIA platform since
DecoderBeginFrame() call on the other GPU vendors seems to
succeed without retry.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/2099>
For reverse playback, we are always copying decoded
frame to downstream buffer. So the pool size can be
and need to be large enough.
In case that forward playback, however, we need to restrict
the max pool size for performance reason. Otherwise decoder
will keep copying decoded texture to downstream buffer pool
if decoding is faster than downstream throughput
performance and also there are queue element between them.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/2083>
Decoder might be able to copy decoded texture to the other buffer pool
during playback depending on context. In that case, copied one
has no D3D11_BIND_DECODER bind flag.
If we used ID3D11VideoProcessor previously for decoder texture,
and incoming texture supports ID3D11VideoProcessor as well even if it has no
D3D11_BIND_DECODER flag (having D3D11_BIND_RENDER_TARGET for example),
allow zero-copying instead of using our fallback texture.
Frequent conversion tool change (between ID3D11VideoProcessor and generic shader)
might result in inconsistent image quality.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/2084>
... instead of QueryInterface-ing per elements. Note that
ID3D11VideoDevice and ID3D11VideoContext objects might not be available
if device doesn't support video interface.
So GstD3D11Device object will create those objects only when requested.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/2079>
Direct3D11 objects are COM, and most COM C APIs are verbose
(C++ is a little better). So, by using C++ APIs, we can make code
shorter and more readable.
Moreover, "ComPtr" helper class (which is C++ only) can be
utilized, that is very helpful for avoiding error-prone COM refcounting
issue/leak.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/2077>
... instead of the largest we ever seen.
Note that d3d11h264dec element holds previously configured DPB size
for later decoder object re-open decision.
This is to fix below case:
1) Initial SPS, required DPB size is 6
- decoder object is opened with DPB size 6
- max_dpb_size is now 6
2) SPS update with resolution change, required DPB size is 1
- decoder object is re-opened with DPB size 1
- max_dpb_size should be updated to 1, but it didn't happen (BUG)
3) SPS update without resolution change, only required DPB size is updated to 6
- decoder object should be re-opened but didn't happen
because we didn't update max_dpb_size at 2).
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/2056>
To convert decoded texture into other format, downstream would use
video processor instead of shader. In order for downstream to
be able to use video processor even if we copied decoded texture
into downstream pool, we should set this bind flag. Otherwise,
downstream would keep switching video processor and shader
to convert format which would result in inconsistent image quality.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/2051>
AS-IS:
D3D11Convert class is baseclass of D3D11ColorConvert and D3D11Scale
* GstD3D11Convert
|_ GstD3D11ColorConvert
|_ GstD3D11Scale
TO-BE:
Introducing a new base class for color conversion and/or rescale elements
* GstD3D11BaseConvert
|_ GstD3D11Convert
|_ GstD3D11ColorConvert
|_ GstD3D11Scale
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/2029>
Add a new element d3d11deinterlace to support deinterlacing.
Similar to d3d11videosink and d3d11compositor, this element is
a wrapper bin of set of child elements including helpful
conversion elements (upload/download and color convert)
to make this element configurable between non-d3d11 elements.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/2016>
Since our decoder DPB texture pool cannot be grown once it's
configured, we should pre-allocate sufficient number of textures
for zero-copy playback (but not too many).
The "min buffers" allocation query parameter can be a hint for
the number of required textures in addition to DPB size.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/2017>
Unlike other stateless decoder implementations (e.g., VA),
our DPB pool cannot be grown since we are using
texture array (pre-allocated, fixed-size d3d11 texture pool).
So, if there's no more available texture to use,
there's no way other than copying it to downstream's
d3d11 buffer pool. Otherwise deadlock will happen.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/2003>
* Don't warn for live object, since ID3D11Debug itself seems to be
holding refcount of ID3D11Device at the moment we called
ID3D11Debug::ReportLiveDeviceObjects(). It would report live object
always
* Device might not be able to support some formats (e.g., P010)
especially in case of WARP device. We don't need to warn about that.
* gst_d3d11_device_new() can be used for device enumeration. Don't warn
even if we cannot create D3D11 device with given adapter index therefore.
* Don't warn for HLSL compiler warning. It's just noise and
should not be critical thing at all
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1986>
Add a way to support drawing on application's texture instead of
usual window handle.
To make use of this new feature, application should follow below step.
1) Enable this feature by using "draw-on-shared-texture" property
2) Watch "begin-draw" signal
3) On "begin-draw" signal handler, application can request drawing
by using "draw" signal action. Note that "draw" signal action
should be happen before "begin-draw" signal handler is returned
NOTE 1) For texture sharing, creating a texture with
D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX flag is strongly recommend
if possible because we cannot ensure sync a texture
which was created with D3D11_RESOURCE_MISC_SHARED
and it would cause glitch with ID3D11VideoProcessor use case.
NOTE 2) Direct9Ex doesn't support texture sharing which was
created with D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX. In other words,
D3D11_RESOURCE_MISC_SHARED is the only option for Direct3D11/Direct9Ex interop.
NOTE 3) Because of missing synchronization around ID3D11VideoProcessor,
If shared texture was created with D3D11_RESOURCE_MISC_SHARED,
d3d11videosink might use fallback texture to convert DXVA texture
to normal Direct3D texture. Then converted texture will be
copied to user-provided shared texture.
* Why not use generic appsink approach?
In order for application to be able to store video data
which was produced by GStreamer in application's own texture,
there would be two possible approaches,
one is copying our texture into application's own texture,
and the other is drawing on application's own texture directly.
The former (appsink way) cannot be a zero-copy by nature.
In order to support zero-copy processing, we need to draw on
application's own texture directly.
For example, assume that application wants RGBA texture.
Then we can imagine following case.
"d3d11h264dec ! d3d11convert ! video/x-raw(memory:D3D11Memory),format=RGBA ! appsink"
^
|_ allocate new Direct3D texture for RGBA format
In above case, d3d11convert will allocate new texture(s) for RGBA format
and then application will copy again the our RGBA texutre into
application's own texture. One texture allocation plus per frame GPU copy will hanppen
in that case therefore.
Moreover, in order for application to be able to access
our texture, we need to allocate texture with additional flags for
application's Direct3D11 device to be able to read texture data.
That would be another implementation burden on our side
But with this MR, we can configure pipeline in this way
"d3d11h264dec ! d3d11videosink".
In that way, we can save at least one texture allocation and
per frame texutre copy since d3d11videosink will convert incoming texture
into application's texture format directly without copy.
* What if we expose texture without conversion and application does
conversion by itself?
As mentioned above, for application to be able to access our texture
from application's Direct3D11 device, we need to allocate texture
in a special form. But in some case, that might not be possible.
Also, if a texture belongs to decoder DPB, exposing such texture
to application is unsafe and usual Direct3D11 shader cannot handle
such texture. To convert format, ID3D11VideoProcessor API needs to
be used but that would be a implementation burden for application.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1873>
