Commit graph

70 commits

Author SHA1 Message Date
Seungha Yang
d9c486352c mfvideoenc: Don't ignore previous flow return value
In case of ASYNC MFT (hardware encoder), we were ignoring previous
finish_frame or pad_push return value. so, error wasn't propagated.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/2370>
2021-06-30 18:35:35 +09:00
Seungha Yang
ba26a5aea8 mfvideoenc: Enhance B-frame timestamp handling
When B-frame is enabled, encoder seems to adjust PTS of encoded sample
by using frame duration.

For instance, one observed timestamp pattern by using B-frame enabled
and 30fps stream is:
* Frame-1: MF pts 0:00.033333300 MF dts 0:00.000000000
* Frame-2: MF pts 0:00.133333300 MF dts 0:00.033333300
* Frame-3: MF pts 0:00.066666600 MF dts 0:00.066666600
* Frame-4: MF pts 0:00.099999900 MF dts 0:00.100000000

We can notice that the amount of PTS shift is frame duration and
Frame-4 exhibits PTS < DTS.

To compensate shifted timestamp, we should
calculate the timestamp offset and re-calculate DTS correspondingly.
Otherwise, total timeline of output stream will be shifted, and that
can cause time sync issue.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/2354>
2021-06-23 14:12:22 +00:00
Seungha Yang
569910a5ac mfh264enc, mfh265enc: Set profile string to src caps
Set configured profile to src caps so that downstream can figure
out selected profile.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/2347>
2021-06-22 15:20:05 +00:00
Seungha Yang
08b792d24c mfvideoenc: Fix UWP build
Add missing GST_MF_HAVE_D3D11 define guard

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/2185>
2021-04-22 15:50:15 +09:00
Seungha Yang
cc7bb86aa6 mfvideoenc: Don't pass 0/1 framerate to MFT
Some MFT implementations do not accept 0/1 framerate and it will
result in encoder open failure. If framerate is unknown,
we will use arbitrary 25/1 framerate value.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/2106>
2021-03-25 03:27:24 +09:00
Matthew Waters
640a65bf96 gst: don't use volatile to mean atomic
volatile is not sufficient to provide atomic guarantees and real atomics
should be used instead.  GCC 11 has started warning about using volatile
with atomic operations.

https://gitlab.gnome.org/GNOME/glib/-/merge_requests/1719

Discovered in https://gitlab.freedesktop.org/gstreamer/gst-plugins-good/-/issues/868

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/2098>
2021-03-22 14:34:36 +11:00
Seungha Yang
e22befad2e mfvideoenc: Enable Direct3D multi-thread protection
As documented by MS. See also
https://docs.microsoft.com/en-us/windows/win32/medfound/supporting-direct3d-11-video-decoding-in-media-foundation#open-a-device-handle

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/2095>
2021-03-20 17:54:04 +09:00
Seungha Yang
824b0ce0f4 Revert "d3d11: Enable native multi-thread protection layer and make use of it"
This reverts commit 872b7f503c.

Native multi-thread protection layer seems to be consuming more CPU
resource than application side protection approach in some cases

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/2095>
2021-03-20 17:53:58 +09:00
Seungha Yang
872b7f503c d3d11: Enable native multi-thread protection layer and make use of it
... instead of our own GRecMutex locking. In this way, any other
Direct3D11 client (MediaFoundation for example) can safely call
any Direct3D11 API even when we are sharing our Direct3D11 device
with others.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/2092>
2021-03-18 16:37:37 +00:00
Seungha Yang
1d6cdbdc90 mediafoundation: Fix resource leak
IMFActivate would hold its internal objects unless user call ShutdownObject(),
even if we release the IMFActivate. Here internal objects may
include Direct3D objects, such as texture, device handle for example.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/2087>
2021-03-16 09:47:41 +00:00
Seungha Yang
1f7681c385 mfvideoenc: Add support for P010 d3d11 texture
Add P010 Direct3D11 texture format support

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1970>
2021-01-21 18:53:27 +09:00
Seungha Yang
84db4b68fe mfvideoenc: Add support for Direct3D11 texture
Initial support for d3d11 texture so that encoder can copy
upstream d3d11 texture into encoder's own texture pool without
downloading memory.

This implementation requires MFTEnum2() API for creating
MFT (Media Foundation Transform) object for specific GPU but
the API is Windows 10 desktop only. So UWP is not target
of this change.
See also https://docs.microsoft.com/en-us/windows/win32/api/mfapi/nf-mfapi-mftenum2

Note that, for MF plugin to be able to support old OS versions
without breakage, this commit will load MFTEnum2() symbol
by using g_module_open()

Summary of required system environment:
- Needs Windows 10 (probably at least RS 1 update)
- GPU should support ExtendedNV12SharedTextureSupported feature
- Desktop application only (UWP is not supported yet)

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1903>
2021-01-13 20:15:04 +00:00
Seungha Yang
e82f1f8b0b mfvideoenc: Re-define default GOP size value
The behavior for zero AVEncMPVGOPSize value would be
varying depending on GPU vendor implementation and some
GPU will produce keyframe only once at the beginning of encoding.
That's unlikely expected result for users.

To make this property behave consistently among various GPUs,
this commit will change default value of "gop-size" property to -1
which means "auto". When "gop-size" is unspecified, then
mfvideoenc will calculate GOP size based on framerate
like that of our x264enc implementation.

See also
https://docs.microsoft.com/en-us/windows/win32/directshow/avencmpvgopsize-property

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1911>
2020-12-27 03:58:39 +09:00
Seungha Yang
8df131ab42 mfvideoenc: Fix use of uninitialized value
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1911>
2020-12-27 03:51:16 +09:00
Seungha Yang
4b522dd355 mfvideoenc: Remove duplicated class registration code
Each codec subclass has the same code for class/element registration,
so we can move the code into one helper methodm and that will make
future enhancement simple.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1909>
2020-12-26 14:25:06 +00:00
Seungha Yang
6195fcf857 mfvideoenc: Improve latency performance for hardware encoder
Unlike software MFT (Media Foundation Transform) which is synchronous
in terms of processing input and output data, hardware MFT works
in asynchronous mode. output data might not be available right after
we pushed one input data into MFT.
Note that async MFT will fire two events, one is "METransformNeedInput"
which happens when MFT can accept more input data,
and the other is "METransformHaveOutput", that's for signaling
there's pending data which can be outputted immediately.

To listen the events, we can wait synchronously via
IMFMediaEventGenerator::GetEvent() or make use of IMFAsyncCallback
object which is asynchronous way and the event will be notified
from Media Foundation's internal worker queue thread.

To handle such asynchronous operation, previous working flow was
as follows (IMFMediaEventGenerator::GetEvent() was used for now)
- Check if there is pending output data and push the data toward downstream.
- Pulling events (from streaming thread) until there's at least
  one pending "METransformNeedInput" event
- Then, push one data into MFT from streaming thread
- Check if there is pending "METransformHaveOutput" again.
  If there is, push new output data to downstream
  (unlikely there is pending output data at this moment)

Above flow was processed from upstream streaming thread. That means
even if there's available output data, it could be outputted later
when the next buffer is pushed from upstream streaming thread.
It would introduce at least one frame latency in case of live stream.

To reduce such latency, this commit modifies the flow to be fully
asynchronous like hardware MFT was designed and to be able to
output encoded data whenever it's available. More specifically,
IMFAsyncCallback object will be used for handling
"METransformNeedInput" and "METransformHaveOutput" events from
Media Foundation's internal thread, and new output data will be
also outputted from the Media Foundation's thread.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1520>
2020-12-19 18:56:33 +00:00
Biswapriyo Nath
8af91be222 mediafoundation: Fix redefinition of variables.
Remove duplicate GstMFDevice and GstMFDeviceProvider declaration.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1884>
2020-12-16 18:48:12 +00:00
Seungha Yang
20d9283e3d mfvideosrc: Use only the first video stream per device
Non-first video stream might not be working with current
implementation. It could be non-video (e.g., photo source) and then
ReadSample() might be blocked forever.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1661>
2020-10-08 20:43:58 +00:00
Seungha Yang
9ecdfea7da mfvideosrc: Fix invalid memory access when outputting jpeg
Don't access unknown-dangerous-nonsense address

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1662>
2020-10-08 16:28:16 +00:00
Seungha Yang
47bbc997f8 mfvideosrc: Set timestamp on buffer when it's captured
Capture the timestamp immediately when new frame is arrived,
instead of doing that on ::create() method. There would be
time gap between captured time and outputting time.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1351>
2020-09-10 10:21:00 +00:00
Seungha Yang
d1d2acead1 mfvideoenc: Add support for zero-copy encoding
Add custom IMFMediaBuffer and IMF2DBuffer implementation in order to
keep track of lifecycle of Media Foundation memory object.
By this new implementation, we can pass raw memory of upstream buffer
to Media Foundation without copy.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1518>
2020-09-09 07:25:52 +00:00
Seungha Yang
5d4ab18ced mediafoundation: Correct wrong raw video format mapping
Was a shameful mistake

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1517>
2020-08-17 21:39:13 +09:00
Seungha Yang
5ffd2c64a0 mediafoundation: Call MFShutdown when destroying plugin
MFStartup and MFShutdown should be paired as documented in
https://docs.microsoft.com/en-us/windows/win32/api/mfapi/nf-mfapi-mfstartup#remarks
Otherwise valgrind-like tools would report false positive memory leak.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1512>
2020-08-13 07:05:36 +00:00
Seungha Yang
4986b0dd34 mfvideosrc: Select common formats if both VideoPreview and VideoRecord are available
Some devices (e.g., Surface Book 2, Surface Pro X) will expose
both MediaStreamType_VideoPreview and MediaStreamType_VideoRecord types
for a logical device. And for some reason, MediaStreamType_VideoPreview
seems to be selected between them while initiailzing device.
But I cannot find any documentation for the decision rule.
To be safe, we will select common formats between them.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1478>
2020-08-01 15:17:05 +00:00
Seungha Yang
479a67c1b7 mfvideosrc: Check framerate for target IMediaFrameFormat selection
Not only resolution and format, but framerate needs to be checked
for proper target IMediaFrameFormat selection.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1478>
2020-08-01 15:17:05 +00:00
Seungha Yang
3fb8caf635 mfvideosrc: Handle I420/IYUV subtypes for UWP cases
Microsoft defines two I420 formats, one is I420, and the other is
IYUV (but both are same, just names are different).
Since both will be converted to GST_VIDEO_FORMAT_I420,
we should check both I420 and IYUV subtypes during
GstVideoFormat to Microsoft's format conversion.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1478>
2020-08-01 15:17:05 +00:00
Seungha Yang
2303ad7bf4 mfvideosrc: Add more debug log
It would be useful for finding the error reason.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1478>
2020-08-01 15:17:05 +00:00
Seungha Yang
fc49886c61 wasapi2, mfvideosrc: Update "dispatcher" property to be only writable
Disallow getting dispatcher pointer, since it doesn't seem to be useful
and might not be safe.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1480>
2020-07-31 21:50:16 +09:00
Seungha Yang
6d960781fc mfvideosrc: Only device activation needs to be running on UI thread
... and the other operations does not have the thread constraint.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1466>
2020-07-26 12:52:15 +00:00
Seungha Yang
502aea3969 mfvideosrc: Add a new property for ICoreDispatcher setting
Since the commit c29c71ae9d,
device activation method will be called from an internal thread.

A problem is that, CoreApplication::GetCurrentView()
method will return nullptr if it was called from non-UI thread,
and as a result, currently implemented method for accessing ICoreDispatcher
will not work in any case. There seems to be no robust way for
accessing ICoreDispatcher other then setting it by user.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1466>
2020-07-26 12:52:15 +00:00
Seungha Yang
42de98fd42 mfvideosrc: Suppress more spammy debug messages
The failure on compressed format (e.g., MJPG, H264 subtypes) is expected.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1468>
2020-07-26 05:48:48 +09:00
Seungha Yang
3422868c59 mfutils: Suppress spammy debug print
Remove FIXME debug print. It seems to be spammy.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1452>
2020-07-22 18:11:18 +09:00
Seungha Yang
6c52008413 mfvideosrc: Expose sorted caps
Sort the list of supported caps for downstream to be able to select
the best image in terms of quality (resolution and framerate) by default.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1452>
2020-07-22 18:11:18 +09:00
Seungha Yang
c3ecea0aa4 mfvideosrc: Don't expose unsupported formats
Some UVC cameras support H.264 stream but we don't support it yet.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1452>
2020-07-22 18:10:57 +09:00
Seungha Yang
37aeb91d54 mfvideosrc: Don't error out when if we've found supported format
While retrieving supported formats by device, the last return might
not be S_OK in case that it's not supported one by us (e.g., H264, JPEG or so).
But if we've found at least one supported raw video format,
we can keep going on.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1451>
2020-07-20 16:13:37 +00:00
Seungha Yang
c1093e3481 plugins: Use g_win32_error_message for HRESULT to string conversion
We don't need to duplicate a method for HRESULT error code to string
conversion. This patch is intended to
* Remove duplicated code
* Ensure FormatMessageW (Unicode version) and avoid FormatMessageA
  (ANSI version), as the ANSI format is not portable at all.
  Note that if "UNICODE" is not defined, FormatMessageA will be aliased
  as FormatMessage by default.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1442>
2020-07-18 11:05:52 +09:00
Seungha Yang
a43d6f6cd9 mfvideosrc: Enable WinRT capture impl. for desktop target
... if target OS version was specified as Windows 10.
When enabled, desktop application can select target capture
implementation between WinRT and Win32
via GST_USE_MF_WINRT_CAPTURE environment
(e,g., GST_USE_MF_WINRT_CAPTURE=1 for WinRT impl.).
Default is Win32 implementation in case of desktop target.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1434>
2020-07-14 16:59:06 +00:00
Mathieu Duponchelle
b33f10e7e2 docs: remove gst prefix from plugin titles 2020-07-02 18:10:21 +02:00
Seungha Yang
8d0dc4fdd2 plugins: Update for documentation of Windows plugins
* Add Since marks
* Make use of GST_PARAM_CONDITIONALLY_AVAILABLE flag
2020-07-02 17:21:29 +02:00
Seungha Yang
4997cde699 mediafoundation: Add VP9 encoder element
Some Intel GPUs support hardware accelerated VP9 encoding and
Microsoft provides software VP9 encoding implementation as well.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1295>
2020-06-22 07:58:03 +00:00
Seungha Yang
1a68da54b6 mfvideosrc: Add support for jpeg on Win32 application
Enable reading jpeg data from webcam if it's supported.
Note that this would be enabled only for Win32.
For UWP, we need to research more about how to support jpeg.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1342>
2020-06-17 17:45:32 +09:00
Seungha Yang
f508c8b988 mfvideosrc: Fix wrong casting
Don't cast ISoftwareBitmap to IMFMediaBuffer

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1342>
2020-06-17 17:37:05 +09:00
Seungha Yang
86e3df9493 mfvideosrc: Add support YUY2 format for UWP
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1342>
2020-06-17 17:37:05 +09:00
Seungha Yang
8d7f537782 mediafoundation: Drop IMFCaptureEngine implementation
It was introduced for later use of its enhanced feature over IMFSourceReader
such as taking photo with video preview, audio/video capturing at
the same time, etc. But currently it's not our use case, and it would
be maintenance burden.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1342>
2020-06-17 17:37:05 +09:00
Seungha Yang
52a82f5b84 mftransform: Fix deadlock when MFT requests processing output twice
This sequence of event/data flow might happen

1) Initially we have one pending output event
  1-1) Then, process the pending output data
2) No pending input event, then we should wait new pending input event
  2-1) Wakeup by new pending event (but it's pending output event)

In above case, MFT will not report new pending input event
if pending output is not processed.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1325>
2020-06-08 23:46:43 +09:00
Seungha Yang
309c679c43 mftransform: Add some debug log
Add some trace level log for debugging

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1325>
2020-06-08 23:42:22 +09:00
Seungha Yang
0f74785b8e mfvideoenc: Set PAR to output IMFMediaType
We've set it to input IMFMediaType but not for output.
So, if PAR is not 1:1, the input IMFMediaType will be accepted
by MFT (default is 1:1).
The PAR of input/output IMFMediaType must be identical

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1322>
2020-06-06 21:19:48 +09:00
Seungha Yang
cec6858401 mftransform: Don't try to drain if MFT is not running
Otherwise MFT will be blocked forever as no event can be generated by
IMFMediaEventGenerator.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1322>
2020-06-06 21:03:05 +09:00
Seungha Yang
12532cc8bb mediafoundation: Use core dispatcher of current view instead of main view
Main view might be hidden depending on application's view tree.
In that case, ICoreApplication object doesn't return get_MainView() method

Note that nothing about this behavior was documented by Microsoft
https://docs.microsoft.com/en-us/uwp/api/windows.applicationmodel.core.coreapplication.mainview

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1310>
2020-05-29 23:01:08 +09:00
Seungha Yang
2dc689c2c8 mediafoundation: Fix undeclared identifier error on UWP build
Some symbols are not available in case of UWP

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1306>
2020-05-28 12:04:43 +00:00