Commit graph

53 commits

Author SHA1 Message Date
Seungha Yang
20d9283e3d mfvideosrc: Use only the first video stream per device
Non-first video stream might not be working with current
implementation. It could be non-video (e.g., photo source) and then
ReadSample() might be blocked forever.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1661>
2020-10-08 20:43:58 +00:00
Seungha Yang
9ecdfea7da mfvideosrc: Fix invalid memory access when outputting jpeg
Don't access unknown-dangerous-nonsense address

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1662>
2020-10-08 16:28:16 +00:00
Seungha Yang
47bbc997f8 mfvideosrc: Set timestamp on buffer when it's captured
Capture the timestamp immediately when new frame is arrived,
instead of doing that on ::create() method. There would be
time gap between captured time and outputting time.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1351>
2020-09-10 10:21:00 +00:00
Seungha Yang
d1d2acead1 mfvideoenc: Add support for zero-copy encoding
Add custom IMFMediaBuffer and IMF2DBuffer implementation in order to
keep track of lifecycle of Media Foundation memory object.
By this new implementation, we can pass raw memory of upstream buffer
to Media Foundation without copy.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1518>
2020-09-09 07:25:52 +00:00
Seungha Yang
5d4ab18ced mediafoundation: Correct wrong raw video format mapping
Was a shameful mistake

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1517>
2020-08-17 21:39:13 +09:00
Seungha Yang
5ffd2c64a0 mediafoundation: Call MFShutdown when destroying plugin
MFStartup and MFShutdown should be paired as documented in
https://docs.microsoft.com/en-us/windows/win32/api/mfapi/nf-mfapi-mfstartup#remarks
Otherwise valgrind-like tools would report false positive memory leak.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1512>
2020-08-13 07:05:36 +00:00
Seungha Yang
4986b0dd34 mfvideosrc: Select common formats if both VideoPreview and VideoRecord are available
Some devices (e.g., Surface Book 2, Surface Pro X) will expose
both MediaStreamType_VideoPreview and MediaStreamType_VideoRecord types
for a logical device. And for some reason, MediaStreamType_VideoPreview
seems to be selected between them while initiailzing device.
But I cannot find any documentation for the decision rule.
To be safe, we will select common formats between them.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1478>
2020-08-01 15:17:05 +00:00
Seungha Yang
479a67c1b7 mfvideosrc: Check framerate for target IMediaFrameFormat selection
Not only resolution and format, but framerate needs to be checked
for proper target IMediaFrameFormat selection.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1478>
2020-08-01 15:17:05 +00:00
Seungha Yang
3fb8caf635 mfvideosrc: Handle I420/IYUV subtypes for UWP cases
Microsoft defines two I420 formats, one is I420, and the other is
IYUV (but both are same, just names are different).
Since both will be converted to GST_VIDEO_FORMAT_I420,
we should check both I420 and IYUV subtypes during
GstVideoFormat to Microsoft's format conversion.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1478>
2020-08-01 15:17:05 +00:00
Seungha Yang
2303ad7bf4 mfvideosrc: Add more debug log
It would be useful for finding the error reason.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1478>
2020-08-01 15:17:05 +00:00
Seungha Yang
fc49886c61 wasapi2, mfvideosrc: Update "dispatcher" property to be only writable
Disallow getting dispatcher pointer, since it doesn't seem to be useful
and might not be safe.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1480>
2020-07-31 21:50:16 +09:00
Seungha Yang
6d960781fc mfvideosrc: Only device activation needs to be running on UI thread
... and the other operations does not have the thread constraint.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1466>
2020-07-26 12:52:15 +00:00
Seungha Yang
502aea3969 mfvideosrc: Add a new property for ICoreDispatcher setting
Since the commit c29c71ae9d,
device activation method will be called from an internal thread.

A problem is that, CoreApplication::GetCurrentView()
method will return nullptr if it was called from non-UI thread,
and as a result, currently implemented method for accessing ICoreDispatcher
will not work in any case. There seems to be no robust way for
accessing ICoreDispatcher other then setting it by user.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1466>
2020-07-26 12:52:15 +00:00
Seungha Yang
42de98fd42 mfvideosrc: Suppress more spammy debug messages
The failure on compressed format (e.g., MJPG, H264 subtypes) is expected.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1468>
2020-07-26 05:48:48 +09:00
Seungha Yang
3422868c59 mfutils: Suppress spammy debug print
Remove FIXME debug print. It seems to be spammy.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1452>
2020-07-22 18:11:18 +09:00
Seungha Yang
6c52008413 mfvideosrc: Expose sorted caps
Sort the list of supported caps for downstream to be able to select
the best image in terms of quality (resolution and framerate) by default.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1452>
2020-07-22 18:11:18 +09:00
Seungha Yang
c3ecea0aa4 mfvideosrc: Don't expose unsupported formats
Some UVC cameras support H.264 stream but we don't support it yet.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1452>
2020-07-22 18:10:57 +09:00
Seungha Yang
37aeb91d54 mfvideosrc: Don't error out when if we've found supported format
While retrieving supported formats by device, the last return might
not be S_OK in case that it's not supported one by us (e.g., H264, JPEG or so).
But if we've found at least one supported raw video format,
we can keep going on.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1451>
2020-07-20 16:13:37 +00:00
Seungha Yang
c1093e3481 plugins: Use g_win32_error_message for HRESULT to string conversion
We don't need to duplicate a method for HRESULT error code to string
conversion. This patch is intended to
* Remove duplicated code
* Ensure FormatMessageW (Unicode version) and avoid FormatMessageA
  (ANSI version), as the ANSI format is not portable at all.
  Note that if "UNICODE" is not defined, FormatMessageA will be aliased
  as FormatMessage by default.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1442>
2020-07-18 11:05:52 +09:00
Seungha Yang
a43d6f6cd9 mfvideosrc: Enable WinRT capture impl. for desktop target
... if target OS version was specified as Windows 10.
When enabled, desktop application can select target capture
implementation between WinRT and Win32
via GST_USE_MF_WINRT_CAPTURE environment
(e,g., GST_USE_MF_WINRT_CAPTURE=1 for WinRT impl.).
Default is Win32 implementation in case of desktop target.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1434>
2020-07-14 16:59:06 +00:00
Mathieu Duponchelle
b33f10e7e2 docs: remove gst prefix from plugin titles 2020-07-02 18:10:21 +02:00
Seungha Yang
8d0dc4fdd2 plugins: Update for documentation of Windows plugins
* Add Since marks
* Make use of GST_PARAM_CONDITIONALLY_AVAILABLE flag
2020-07-02 17:21:29 +02:00
Seungha Yang
4997cde699 mediafoundation: Add VP9 encoder element
Some Intel GPUs support hardware accelerated VP9 encoding and
Microsoft provides software VP9 encoding implementation as well.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1295>
2020-06-22 07:58:03 +00:00
Seungha Yang
1a68da54b6 mfvideosrc: Add support for jpeg on Win32 application
Enable reading jpeg data from webcam if it's supported.
Note that this would be enabled only for Win32.
For UWP, we need to research more about how to support jpeg.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1342>
2020-06-17 17:45:32 +09:00
Seungha Yang
f508c8b988 mfvideosrc: Fix wrong casting
Don't cast ISoftwareBitmap to IMFMediaBuffer

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1342>
2020-06-17 17:37:05 +09:00
Seungha Yang
86e3df9493 mfvideosrc: Add support YUY2 format for UWP
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1342>
2020-06-17 17:37:05 +09:00
Seungha Yang
8d7f537782 mediafoundation: Drop IMFCaptureEngine implementation
It was introduced for later use of its enhanced feature over IMFSourceReader
such as taking photo with video preview, audio/video capturing at
the same time, etc. But currently it's not our use case, and it would
be maintenance burden.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1342>
2020-06-17 17:37:05 +09:00
Seungha Yang
52a82f5b84 mftransform: Fix deadlock when MFT requests processing output twice
This sequence of event/data flow might happen

1) Initially we have one pending output event
  1-1) Then, process the pending output data
2) No pending input event, then we should wait new pending input event
  2-1) Wakeup by new pending event (but it's pending output event)

In above case, MFT will not report new pending input event
if pending output is not processed.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1325>
2020-06-08 23:46:43 +09:00
Seungha Yang
309c679c43 mftransform: Add some debug log
Add some trace level log for debugging

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1325>
2020-06-08 23:42:22 +09:00
Seungha Yang
0f74785b8e mfvideoenc: Set PAR to output IMFMediaType
We've set it to input IMFMediaType but not for output.
So, if PAR is not 1:1, the input IMFMediaType will be accepted
by MFT (default is 1:1).
The PAR of input/output IMFMediaType must be identical

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1322>
2020-06-06 21:19:48 +09:00
Seungha Yang
cec6858401 mftransform: Don't try to drain if MFT is not running
Otherwise MFT will be blocked forever as no event can be generated by
IMFMediaEventGenerator.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1322>
2020-06-06 21:03:05 +09:00
Seungha Yang
12532cc8bb mediafoundation: Use core dispatcher of current view instead of main view
Main view might be hidden depending on application's view tree.
In that case, ICoreApplication object doesn't return get_MainView() method

Note that nothing about this behavior was documented by Microsoft
https://docs.microsoft.com/en-us/uwp/api/windows.applicationmodel.core.coreapplication.mainview

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1310>
2020-05-29 23:01:08 +09:00
Seungha Yang
2dc689c2c8 mediafoundation: Fix undeclared identifier error on UWP build
Some symbols are not available in case of UWP

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1306>
2020-05-28 12:04:43 +00:00
Seungha Yang
7b5f99b47a mediafoundation: Add support MP3 audio encoding
Add MediaFoundation MP3 encoder

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1280>
2020-05-27 10:34:47 +00:00
Seungha Yang
cee619486a mediafoundation: Add support for AAC encoding
Add MediaFoundation AAC encoder element.

Before Windows 10, mono and stereo channels were supported audio channels
configuration by AAC encoder MFT. However, on Windows 10,
5.1 channels support was introduced.

To expose correct range of support format by this element
whatever the OS version is, this element will enumerate
all the supported format by the AAC encoder MFT
and then will configure sink/src templates while plugin init.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1280>
2020-05-27 10:34:47 +00:00
Seungha Yang
18f5bdee16 mfutils: Move IMediaType release function to common utility
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1280>
2020-05-27 10:34:47 +00:00
Seungha Yang
c8469644dd mediafoundation: Add util function to dump IMFAttributes values
It would be useful for debugging.

Reference: https://docs.microsoft.com/en-us/windows/win32/medfound/media-type-debugging-code
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1280>
2020-05-27 10:34:47 +00:00
Seungha Yang
8ce4980273 mfsourceobject: Remove useless null check for string
We can pass null for the value of string type property.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1241>
2020-05-25 15:09:20 +00:00
Seungha Yang
50b36ce257 mediafoundation: Use G_BEGIN_DECLS/G_END_DECLS pair everywhere
... instead of extern "c" {} block.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1241>
2020-05-25 15:09:20 +00:00
Seungha Yang
c4cb51c63d mediafoundation: Add support video capture on UWP app
New video capture implementation using WinRT Media APIs for UWP app.
Due to the strict permission policy of UWP, device enumeration and
open should be done via new WinRT APIs and to get permission from users,
it will invoke permission dialog on UI.
Strictly saying, this implementation is not a part of MediaFoundation
but structurally it's very similar to MediaFoundation API.
So we can avoid some code duplication by adding this implementation
into MediaFoundation plugin.

This implementation requires UniversalApiContract version >= 6.0
which is part of Windows 10 version 1803 (Redstone 4)

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1241>
2020-05-25 15:09:20 +00:00
Seungha Yang
e580676747 mfsourceobject: Move device name, path, and index to public struct
... so that subclass can access each value and update them.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1241>
2020-05-25 15:09:20 +00:00
Seungha Yang
6ae478946c mediafoundation: Fix typo in source object impl.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1241>
2020-05-25 15:09:20 +00:00
Seungha Yang
3efc8f5de9 mftransform: Clear unused output IMediaSample
If MFT doesn't produce encoded output, need to free allocated
output sample and buffer.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1297>
2020-05-25 10:25:37 +00:00
Seungha Yang
b93fbdd3cd mfvideoenc: Fix huge memory leak
Subclass must unref passed GstVideoCodecFrame on GstVideoEncoder::handle_frame()

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1294>
2020-05-24 19:12:28 +09:00
Seungha Yang
dac36cb98f mediafoundation: Add device provider implementation
Only static device probing is supported for now

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1236>
2020-05-20 11:06:09 +00:00
Seungha Yang
ba45e3f82b mfsourceobject: Store selected device path, name and index
Update path, name and index with selected device so that checked by
get_property() after constructed.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1236>
2020-05-20 11:06:09 +00:00
Seungha Yang
8afc283b51 mediafoundation: Refactor GstMFSourceObject implementation
* Move CoInitializeEx/CoUninitialize pair into thread function in order to
  ensure MTA COM thread
* Move common code to baseclass

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1269>
2020-05-15 09:03:13 +00:00
Seungha Yang
c29c71ae9d mediafoundation: Remove COM thread constraints from GstMFTransform object
Move CoInitializeEx/CoUninitialize pair into our dedicated thread so that
we can ensure COM thread is MTA. This will remove thread constraints
around plugin init.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1269>
2020-05-15 09:03:13 +00:00
Joshua M. Doe
bfc605a666 mfvideosrc: add GRAY16_LE format support
FOURCC "Y16 " is used by FLIR Boson USB Video Class (UVC) camera

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1222>
2020-04-28 15:14:50 -04:00
Seungha Yang
912b2d7af9 mediafoundation: Disable querying supprted maximum resolution
Initializing MFT for checking supported maximum resolution is too
slow.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/760>
2020-04-28 14:37:31 +00:00