This commit introduces IOSGLMemory which is a GLMemory that falls back to
GstAppleCoreVideoMemory for CPU access. This is a temporary solution until
IOSurface gets exposed as a public framework on iOS and so we can use
IOSurfaceMemory on both MacOS and iOS.
https://bugzilla.gnome.org/show_bug.cgi?id=769210
Add systemstream=false to caps, otherwise the decoder
may be picked for MPEG-PS files. Also parsed=true,
as video toolbox expects entire frame in
VTDecompressionSessionDecodeFrame.
https://bugzilla.gnome.org/show_bug.cgi?id=770049
Rather than assuming something. e.g. zerocopy on iOS with GLES3 requires
the use of Luminance/Luminance Alpha formats and does not work with
Red/RG textures.
The hardware decoder can become (temporarily) unavailable across
VTDecompressionSessionCreate/Destroy calls. During negotiation if the currently
configured caps are still accepted by downstream we keep using them so we don't
have to destroy and recreate the decoding session.
This indirectly fixes https://bugzilla.gnome.org/show_bug.cgi?id=767429, by
making vtdec stick to GLMemory.
The URI must already be escaped by the caller, we don't support passing around
invalid (unescaped) URIs via the GstURIHandler interface.
Also it will escape too much of the URI in this case, e.g.
ipod-library://item/item.m4a?id=3143338395173862951
becomes
ipod-library://item/item.m4a%3Fid%3D3143338395173862951
https://bugzilla.gnome.org/show_bug.cgi?id=767492
Move calling gst_vtdec_push_frames_if_needed from ::set_format to ::negotiate so
that we always drain even when renegotiation is triggered by downstream.
vtdec specifies sysmem; GLMemory as template caps. When negotiating, we used to
call gst_pad_peer_query_caps (..., filter) with our template caps as filter. The
query does gst_caps_intersect (filter, peercaps) internally which gives
precedence to the order of the filter caps. While we want to output sysmem by
default, when negotiating with glimagesink which returns GLMemory; sysmem; we
do want to do GL, so we now query using a NULL filter and intersect the result
with our template caps giving precedence to downstream's caps.
tl;dr: make sure we end up negotiating GLMemory with glimagesink
Similar to vtdec_hw, this commit adds a vtenc_h264_hw element that fails
caps negotiation unless a hardware encoder could actually be acquired.
This is useful in situations where a fallback to a software encoder
other than the vtenc_h264 software encoder is desired (e.g. to x264enc).
https://bugzilla.gnome.org/show_bug.cgi?id=767104
When renegotiating mid stream - for example with variable bitrate
streams - and therefore destroying and recreating VTSessions, the
hw decoder might become temporarily unavailable.
To deal with this and avoid erroring out on bitrate changes,
vtdec_hw now falls back to using the software decoder if the hw
one was available at some point but isn't anymore. At
renegotiation/bitrate change time, it will still retry to open
the hardware one.
::negotiate can be called several times before the CAPS event is sent downstream
so use the currently configured output state caps instead of the pad current
caps when deciding whether to recreate the VTSession or not.
This leads to creating/destroying less VTSessions which makes renegotiation more
reliable especially when using hw decoding.
Don't wait until later, we want to know here if the codec can be opened or not
for the requested format. This was removed (accidentially?) by
119e09eac3
Without this decodebin has no way to switch to a different decoder if this one
does not work.
https://bugzilla.gnome.org/show_bug.cgi?id=762613
Leave kCVOpenGLESTextureCacheMaximumTextureAgeKey to the default (1s). We used
to set it to 0 and flush manually, but apparently (looking at the GLES profiler)
0 means "disable the cache entirely".
Rework the GL context code. Now both avfvideosrc and vtdec can create an
internal GL context for pushing textures. Both elements will still try to
use/switch to a local context where available (including after RECONFIGURE
events).
Actually set the configured framerate. Before we only used to set the first
matching framerate range. On iOS where the camera reports ranges [2, 60], we
used to configure the camera to output anything between 2 and 60fps.
Implement a new memory type wrapping CVPixelBuffer.
There are two immediate advantages:
a) Make the GstMemory itself retain the CVPixelBuffer. Previously,
the containing GstBuffer was solely responsible for the lifetime of
the backing CVPixelBuffer.
With this change, we remove the GST_MEMORY_FLAG_NO_SHARE so that
GstMemory objects be referenced by multiple GstBuffers (doing away
with the need to copy.)
b) Delay locking CVPixelBuffer into CPU memory until it's actually
mapped -- possibly never.
The CVPixelBuffer object is shared among references, shares and
(in planar formats) planes, so a wrapper GstAppleCoreVideoPixelBuffer
structure was introduced to manage locking.
https://bugzilla.gnome.org/show_bug.cgi?id=747216
When doing GLMemory avfvideosrc negotiates UYVY. This change allows avfvideosrc
! tee name=t ! ... ! glimagesink t. ! ... ! gldownload ! vtenc_h264 ! ...
to do GLMemory and 0-copy with the encoder (with the CV meta).
Change texture format from BGRA to NV12. This allows a pipeline like avfvideosrc
! tee name=t ! ... ! glimagesink t. ! ... ! gldownload ! vtenc_h264 ! ... to
negotiate GLMemory. This makes the glimagesink branch much faster (obviously)
and triggers the 0-copy path between avfvideosrc and vtenc (using the CV meta).
Combined this results in a huge perf improvement on iOS (25-30% of CPU time in a
pipeline like the one above).
Note that this doesn't introduce a new shader conversion in the sink, since BGRA
textures had to be copied/converted from format=BGRA,texture-target=RECTANGLE to
format=RGBA,texture-target=2D anyway.
