In some cases, decodebin3 will send us incomplete caps (not containing
codec_data), and then a GAP event, which will force a negotiation.
This segfaults due to a null pointer deref because self->input_state
is NULL.
The only possible fix is to avoid negotiating when we get incomplete
caps (to avoid re-negotiationg immediately afterwards, which isn't
supported by some muxers), but also set as much input state as
possible so that a renegotiation triggered by a GAP event can complete
successfully.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/7634>
Sometimes under certain loads, VT can error out with kVTVideoEncoderMalfunctionErr or kVTVideoEncoderNotAvailableNowErr.
These have been reported to happen more often than usual if CopyProperty/SetProperty() is used close to the encode call.
Both can be worked around by restarting the encoding session.
These errors can be returned either directly from VTCompressionSessionEncodeFrame() or later in the encoding callback.
This patch handles both scenarios the same way - a session restart is be attempted on the next encode_frame() call.
If the error is returned immediately by the encode call, it's possible that some correct frames will still be given to
the output callback, but for simplicity (+ because I wasn't able to verify this scenario) let's just discard those.
In addition, this commit also simplifies the beach/drop logic in enqueue_buffer.
Related bug reports in other projects:
http://www.openradar.me/45889262https://github.com/aws/amazon-chime-sdk-ios/issues/170#issuecomment-741908622
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/7173>
This needs significant work to use the new Metal→Vulkan integration
extension `VK_EXT_metal_objects`
```
MoltenVK/mvk_deprecated_api.h:132:1: note: 'vkGetMTLDeviceMVK' has been explicitly marked deprecated here
MVK_DEPRECATED_USE_MTL_OBJS
^
MoltenVK/mvk_deprecated_api.h:74:52: note: expanded from macro 'MVK_DEPRECATED_USE_MTL_OBJS'
#define MVK_DEPRECATED_USE_MTL_OBJS VKAPI_ATTR [[deprecated("Use the VK_EXT_metal_objects extension instead.")]]
^
../sys/applemedia/videotexturecache-vulkan.mm:303:20: error: 'vkSetMTLTextureMVK' is deprecated:
Use the VK_EXT_metal_objects extension instead.
VkResult err = vkSetMTLTextureMVK (memory->vulkan_mem.image, texture);
^
MoltenVK/mvk_deprecated_api.h:151:1: note: 'vkSetMTLTextureMVK' has been explicitly marked deprecated here
MVK_DEPRECATED_USE_MTL_OBJS
^
MoltenVK/mvk_deprecated_api.h:74:52: note: expanded from macro 'MVK_DEPRECATED_USE_MTL_OBJS'
#define MVK_DEPRECATED_USE_MTL_OBJS VKAPI_ATTR [[deprecated("Use the VK_EXT_metal_objects extension instead.")]]
^
2 errors generated.
```
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/7091>
When building for iOS in Cerbero, as of MoltenVK SDK 1.3.283, we have
to statically link to libMoltenVK since it no longer ships a dylib.
This requires linking to libc++, so we find the dep with the objc++
compiler to ensure that meson uses the right linker.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/7091>
This was already being used in handle_frame() for errors that happen when queueing a frame for decoding,
let's do the same when a frame is flagged with an error in the output callback.
From quick testing, this makes seeking more reliable (previously, it would sometimes cause a decoding error
and shut the whole decoder down due to GST_FLOW_ERROR).
Also manually sets the max error count to actually stop processing if too many errors occur.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/6446>
ReferenceMissingErr is not critical and the simplest solution is to just ignore it. The frame has
the FrameDropped flag set when it occurs, so we can just drop it as usual.
BadDataErr is also not immediately critical, but in its case let's set the ERROR flag,
so the output loop can use GST_VIDEO_DECODER_ERROR to count and error out if it happens too many times.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/6446>
Adds a separate vtenc_h265a element (with a _hw variant as usual) for the HEVCWithAlpha codec type.
Decided to go with a separate element to not break existing uses of the normal HEVC encoder.
The preserve_alpha property is still only used for ProRes, no need for it here because we explicitly say we want alpha
when using the new element.
For now, the HEVCWithAlpha has an issue where it does not throttle the amount of input frames queued internally.
I added a quick workaround where encode_frame() will block until enqueue_frame() callback notifies it that some space
has been freed up in the internal queue. The limit was set to 5, which should be enough I guess? Hopefully this is not
too prone to race conditions.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/6664>
Calling gst_pad_peer_query_caps() without a filter can give us EMPTY caps, whereas all the code below
assumes that's not the case. Replacing query+intersect with a filtered query ensures we always get a subset
of the template caps back.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/6429>
In an early non-linked scenario, this was causing a ton of criticals about the queue array,
because the output callback would still fire for leftover frames that were still being processed by VT
at the time the output loop stopped. This makes sure they're flushed correctly as well.
Also renames gst_vtdec_loop to gst_vtdec_output_loop for consistency with related functions.
wip
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/6397>
Sometimes a call to negotiate (and thus drain) can happen from the output loop
(via finish_frame()), which will tell VT to output all internal frames, but that won't succeed
if we happen to decide to wait for the queue to empty (because the loop is waiting for draining to finish and
will not make space in the queue!). This commit adds an override for the queue size limit if we're draining/flushing.
This bug could happen for any formats, but was especially obvious for ProRes, which has dpb_size of 0.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/6397>
osxaudio has a few helper methods potentially useful in atdec (or future atenc), like GStreamer -> CoreAudio
channel mapping. Doesn't make sense to duplicate them in applemedia, and atdec is the only audio-oriented
element there anyway.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/6223>
Explicitly calls gst_vtenc_pause_output_loop when going PAUSED->READY to make sure GST_PAD_STREAM_LOCK is not taken.
Before this change, a deadlock would occur if pipeline got stopped right after one output buffer was generated by vtenc.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/5933>
The pool currently defaults to performing a layout transition to
VK_IMAGE_LAYOUT_TRANSFER_DST_OPTIMAL, with some special exceptions for
video usages. This may not be a legal transition depending on the usage.
Provide an API to explicitly control the initial image layout.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/5881>
This was wrongly calling the base class method, which unnecessairly took the stream lock, already taken by
handle_frame(). The drain() call in negotiate() would then wait for the output loop to pause, while that loop
is stuck waiting to take the stream lock, thus causing a deadlock.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/5521>
Moves outputting frames to a task on the source pad, bringing vtdec in line with vtenc.
This brings possible performance improvements thanks to decoupling queueing new frames from outputting processed ones.
The queue length is limited to `2*DBP` to prevent decoding too far ahead compared to what we're pushing downstream.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/5163>
This was easy to trigger when testing with e.g. vtenc ! vtdec ! glimagesink and closing the sink via window button,
causing GST_FLOW_ERROR to be received by the output loop, stopping it with the queue still full. This made the
enqueue_buffer() callback to lock waiting for space in our queue, while handle_frame() was waiting for the internal
VideoToolbox queue to free up, so that VTCompressionSessionEncodeFrame could finish. As the output loop was not
running, both functions waited forever.
Fixed by 1) immediately emptying our queue when GST_FLOW_ERROR is received (like we already did with _FLUSHING)
and 2) unconditionally setting the flushing flag in finish_encoding() when it sees the output loop stopped because
of GST_FLOW_ERROR, so that enqueue_buffer() will immediately discard any new frames coming out of VideoToolbox.
Both of those make sure we never run into the both-queues-full scenario.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/5303>
vtenc has an async output queue, which we only iterate over after another frame is enqueued.
At the very least it means we're always a frame behind the fastest possible output.
In edge cases it's also bug-prone - for example if we only have 1 frame, the downstream caps negotiation
will never happen.
This commit adds a separate task running on the source pad, which only iterates over the output queue
and pushes frames out as soon as they're put there. The queue length is limited to ensure we don't encode
too far ahead compared to what downstream can consume. Any failures that occur when pushing data downstream
will be signalled in self->downstream_ret so that other parts of code can act accordingly.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/4967>
These days you're can use minFrameDuration and maxFrameDuration which
are CMTime with fractional values. That way we don't need to convert
between double and fractions in a really weird way.
This fixes really odd fractional values exposed in caps, like:
2000000/76923, 1000000/37037, 5000000/178571, 10000000/344827, 10000000/333333
Which are actually just 26/1, 27/1, 28/1, 29/1, 30/1
We can also delete a lot of outdated code for iOS versions older than
7.0 by using newer APIs.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/4134>
