Commit graph

39 commits

Author SHA1 Message Date
Piotr Brzeziński
e9ab880e66 vtenc: Use new release_frame/drop_frame encoder API
Replaces usage of gst_video_codec_frame_unref everywhere.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/7173>
2024-09-04 06:46:07 +00:00
Piotr Brzeziński
00eb9ad62e vtenc: Restart encoding session when certain errors are detected
Sometimes under certain loads, VT can error out with kVTVideoEncoderMalfunctionErr or kVTVideoEncoderNotAvailableNowErr.
These have been reported to happen more often than usual if CopyProperty/SetProperty() is used close to the encode call.
Both can be worked around by restarting the encoding session.

These errors can be returned either directly from VTCompressionSessionEncodeFrame() or later in the encoding callback.
This patch handles both scenarios the same way - a session restart is be attempted on the next encode_frame() call.

If the error is returned immediately by the encode call, it's possible that some correct frames will still be given to
the output callback, but for simplicity (+ because I wasn't able to verify this scenario) let's just discard those.

In addition, this commit also simplifies the beach/drop logic in enqueue_buffer.

Related bug reports in other projects:
http://www.openradar.me/45889262
https://github.com/aws/amazon-chime-sdk-ios/issues/170#issuecomment-741908622

Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/7173>
2024-09-04 06:46:07 +00:00
Nirbheek Chauhan
fa25dd4de5 vtenc: Do not set the CRF/Quality parameter with ProRes
It's not supported with ProRes, setting the property will fail.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/7232>
2024-07-30 12:42:17 +00:00
Nirbheek Chauhan
cbadb0b4ae vtenc: Log warnings when setting a property fails
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/7232>
2024-07-30 12:42:17 +00:00
Nirbheek Chauhan
f8711239e4 vtenc: Add max-frame-delay property
This controls the number of frames allowed in the compression window.
Not all encoders and implementations support it; I've only managed to
successfully use it with ProRes.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/7232>
2024-07-30 12:42:17 +00:00
Nirbheek Chauhan
30d2b8895a vtenc: Support emulating CBR mode with data rate limits
CBR is only supported on Apple Silicon, and this "emulation" works
surprisingly well. We set the window size to a single frame, and don't
set ABR at all.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/7232>
2024-07-30 12:42:17 +00:00
Nirbheek Chauhan
d327105228 vtenc: Add new property for setting data rate limits
This proxies kVTCompressionPropertyKey_DataRateLimits, except it
only supports a single limit for now.
https://developer.apple.com/documentation/videotoolbox/kvtcompressionpropertykey_dataratelimits

Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/7232>
2024-07-30 12:42:17 +00:00
Nirbheek Chauhan
4207b4e281 vtenc: Add support for constant bitrate encoding
Only supported on macOS 13.0+ and iOS 16.0+ on Apple Silicon.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/7232>
2024-07-30 12:42:17 +00:00
Nirbheek Chauhan
2e39aefe22 vtenc: Eliminate some needless complex code
We do not need a helper that takes a lock to fetch the values of these
properties. There is no race being prevented.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/7232>
2024-07-30 12:42:17 +00:00
Nirbheek Chauhan
3621b82c63 vtenc: Fix setting of quality property
gst_vtenc_set_quality() will never actually set the VT compression
property, because it tries to set it on self->session which is not
initialized at this point.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/7232>
2024-07-30 12:42:17 +00:00
Piotr Brzeziński
32d4e5f73c vtenc: Fix redistribute latency spam
Just a quick fix to only report the maximum noticed delay (measured by frames inside the encoder) instead of changing
the reported latency every time the number there changes, which is way too often.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/7094>
2024-06-24 16:08:37 +02:00
Piotr Brzeziński
4612a6795a vtenc: Enable HEVC with alpha encoding
Adds a separate vtenc_h265a element (with a _hw variant as usual) for the HEVCWithAlpha codec type.
Decided to go with a separate element to not break existing uses of the normal HEVC encoder.
The preserve_alpha property is still only used for ProRes, no need for it here because we explicitly say we want alpha
when using the new element.

For now, the HEVCWithAlpha has an issue where it does not throttle the amount of input frames queued internally.
I added a quick workaround where encode_frame() will block until enqueue_frame() callback notifies it that some space
has been freed up in the internal queue. The limit was set to 5, which should be enough I guess? Hopefully this is not
too prone to race conditions.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/6664>
2024-05-28 10:53:25 +00:00
Piotr Brzeziński
2aa1f465e2 vtenc: Add missing vtenc_h265 docs
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/6664>
2024-05-28 10:53:25 +00:00
Sebastian Dröge
0ef396359c gst: Move GstQueueArray as GstVecDeque to core
And change lengths and indices from guint to gsize for a more correct type.

Also deprecate GstQueueArray and implement it in terms of GstVecDeque.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/6779>
2024-05-06 18:25:42 +00:00
Piotr Brzeziński
3f67e70f39 vtenc: Fix PAUSED->READY deadlock when output loop is running
Explicitly calls gst_vtenc_pause_output_loop when going PAUSED->READY to make sure GST_PAD_STREAM_LOCK is not taken.

Before this change, a deadlock would occur if pipeline got stopped right after one output buffer was generated by vtenc.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/5933>
2024-01-18 07:37:58 +00:00
Piotr Brzeziński
9cbe9a52fe vtenc: Fix deadlock after GST_FLOW_ERROR is received on frame push
This was easy to trigger when testing with e.g. vtenc ! vtdec ! glimagesink and closing the sink via window button,
causing GST_FLOW_ERROR to be received by the output loop, stopping it with the queue still full. This made the
enqueue_buffer() callback to lock waiting for space in our queue, while handle_frame() was waiting for the internal
VideoToolbox queue to free up, so that VTCompressionSessionEncodeFrame could finish. As the output loop was not
running, both functions waited forever.

Fixed by 1) immediately emptying our queue when GST_FLOW_ERROR is received (like we already did with _FLUSHING)
and 2) unconditionally setting the flushing flag in finish_encoding() when it sees the output loop stopped because
of GST_FLOW_ERROR, so that enqueue_buffer() will immediately discard any new frames coming out of VideoToolbox.
Both of those make sure we never run into the both-queues-full scenario.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/5303>
2023-09-20 13:10:02 +00:00
Piotr Brzeziński
510e227996 vtenc: Fix cosmetic issues
Removes accidental GST_VIDEO_DECODER macro usage.
Also sets the initial size of the output queue to avoid having to expand when first buffers come through.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/5164>
2023-08-09 07:35:32 +00:00
Piotr Brzeziński
dc494d9edb vtenc: Move output loop to a separate thread
vtenc has an async output queue, which we only iterate over after another frame is enqueued.
At the very least it means we're always a frame behind the fastest possible output.
In edge cases it's also bug-prone - for example if we only have 1 frame, the downstream caps negotiation
will never happen.

This commit adds a separate task running on the source pad, which only iterates over the output queue
and pushes frames out as soon as they're put there. The queue length is limited to ensure we don't encode
too far ahead compared to what downstream can consume. Any failures that occur when pushing data downstream
will be signalled in self->downstream_ret so that other parts of code can act accordingly.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/4967>
2023-07-21 13:05:26 +00:00
Andoni Morales Alastruey
fc6e6fb547 vtenc: remove duplicated framerate and size variables
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/4282>
2023-06-19 08:37:55 +00:00
Andoni Morales Alastruey
23812bbc92 vtenc: apply DTS offset to ensure DTS <= PTS
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/4282>
2023-06-19 08:37:54 +00:00
Tim-Philipp Müller
81716a1097 gst-plugins-bad: re-indent with GNU indent 2.2.12
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/4182>
2023-03-17 03:18:54 +00:00
Nirbheek Chauhan
2d6da02653 applemedia: Consolidate GstVideoFormat <-> CVPixelFormat conversion
In the process we have changed the color value range from video-range
to full-range, which is probably what people want in the first place.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/4135>
2023-03-14 23:35:16 +00:00
Piotr Brzeziński
b3830b08fd vtenc: Fix checking for certain CPU variants when running in VMs
These checks were introduced to prevent exposing ARGB64/RGBA64 in the caps
when running on M1 Pro/Max with macOS <13 because of a bug in VideoToolbox.
Unfortunately, the initial buffer size of 15 is too short when running
in a VM - the CPU brand string there looks like "Apple M1 Pro (Virtual)",
which due to its length causes sysctlbyname to return -1, resulting in
broken formats still showing up in the caps.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/4080>
2023-02-28 16:00:07 +00:00
Piotr Brzeziński
04615ca55a vtenc/vtdec: Fix typo in RGBA64_LE availability define
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/3912>
2023-02-13 17:48:37 +00:00
Piotr Brzeziński
8d81c4827c vtenc: Disable ARGB/RGBA64 caps on M1 Pro/Max with macOS <13
Fixes #1358.
Passing ARGB64/RGBA64 to vtenc caused the encoding to fail
when running on M1 Pro/Max variants with macOS 12.x, so let's
remove these formats from caps when such scenario is detected.
This issue appears to have been fixed OS-side in macOS 13.0.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/3912>
2023-02-13 17:48:37 +00:00
Piotr Brzeziński
ae200a4d62 vtenc: Disable HW acceleration for interlaced ProRes
Due to a bug in the VT API, attempting to encode interlaced content
with ProRes results in an error, halting the pipeline instead of
gracefully falling back to software encoding.
Should be removed in the future if Apple ever fixes this issue.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/3222>
2023-02-02 06:18:02 +00:00
Piotr Brzeziński
6bf15124e7 applemedia: Add HEVC support to vtenc and vtdec
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/2754>
2022-07-16 05:20:56 +00:00
Nirbheek Chauhan
bb32532f60 applemedia: Disable 64RGBALE support on older macOS
The kCVPixelFormatType_64RGBALE enum is only available on macOS Big
Sur (11.3) and newer. We also cannot use that while configuring the
encoder or decoder on older macOS.

Define the symbol unconditionally, but only use it when we're running
on Big Sur with __builtin_available().

Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/1613>
2022-02-01 16:57:04 +00:00
Nirbheek Chauhan
6ae4497c07 applemedia: Remove some unnecessary variables
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/1613>
2022-02-01 16:57:04 +00:00
Nirbheek Chauhan
f1449ac116 docs: Rename "OS X" to "macOS" in some documentation
No one uses the term "Mac OS X" anymore, it's "macOS". "OS X" is even
worse, because people will usually start the search with "mac".

Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/1607>
2022-01-30 19:17:18 +05:30
Nirbheek Chauhan
d05fcadf4e applemedia: Document vtenc / vtdec elements
Also preserve-alpha property should only be exposed on the
vtenc_prores element since h264 does not support transparency.

Fixes https://gitlab.freedesktop.org/gstreamer/gst-docs/-/issues/94

Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/1601>
2022-01-30 02:17:24 +05:30
Nirbheek Chauhan
6b7d819c25 vtenc: Signal ignored alpha component with ProRes
When the image is opaque but the output ProRes format has an alpha
component (4 component, 32 bits per pixel), Apple requires that we
signal that it should be ignored by setting the depth to 24 bits per
pixel. Not doing so causes the encoded files to fail validation.

So we set that in the caps and qtmux sets the depth value in the
container, which will be read by demuxers so that decoders can skip
those bytes entirely. qtdemux does this, but vtdec does not use this
information at present.

The sister change was made in qtmux and qtdemux in:
https://gitlab.freedesktop.org/gstreamer/gst-plugins-good/-/merge_requests/1061

Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/1489>
2022-01-04 18:01:54 +00:00
Nirbheek Chauhan
b8d6fd905e applemedia: Add ARGB64_BE, RGBA64_LE support to vtenc/vtdec
We can add this now that ARGB64_BE videoconvert support was added in:
https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/1247

Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/1214>
2021-11-01 09:24:52 +00:00
Nirbheek Chauhan
fed69c48b6 vtenc: Add FieldDetail properties for interlaced input
Standard interlace handling:
* If we have interlace-mode=interleaved and the field order, we just
  set it when creating the session
* If we have interlace-mode=(interleaved|mixed) and no field order, we
  set the field order on the first buffer

The encoder session does not support changing the FieldDetail after it
has started encoding frames, so we cannot support mixed streams
correctly.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/1214>
2021-11-01 09:24:52 +00:00
Nirbheek Chauhan
29a32703a5 vtenc: Add a property to forcibly ignore alpha values
This PropertyKey is not documented in any headers anywhere, so we need
to define it ourselves.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/1214>
2021-11-01 09:24:52 +00:00
Nirbheek Chauhan
4e37f99975 vtenc: Set colorimetry information
It looks like VideoToolbox doesn't support all our colorimetries.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/1214>
2021-11-01 09:24:52 +00:00
Nirbheek Chauhan
0f0e68080c applemedia: Add ProRes support to vtenc and vtdec
For vtdec, we continue to prefer NV12; else we pick whatever
downstream wants. In the special case where we're decoding 10-bit or
12-bit ProRes formats, we will prefer AYUV64.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/1214>
2021-11-01 09:24:52 +00:00
Nirbheek Chauhan
db638134e7 vtenc: Improve error reporting in chain function
Otherwise it is quite difficult to figure out why the chain function
failed. Also assert not reached for case statements that should not be
hit.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/1214>
2021-11-01 09:24:52 +00:00
Thibault Saunier
019971a3c7 Move files from gst-plugins-bad into the "subprojects/gst-plugins-bad/" subdir 2021-09-24 16:14:36 -03:00
Renamed from sys/applemedia/vtenc.c (Browse further)