This commit introduces IOSGLMemory which is a GLMemory that falls back to
GstAppleCoreVideoMemory for CPU access. This is a temporary solution until
IOSurface gets exposed as a public framework on iOS and so we can use
IOSurfaceMemory on both MacOS and iOS.
https://bugzilla.gnome.org/show_bug.cgi?id=769210
Rework the GL context code. Now both avfvideosrc and vtdec can create an
internal GL context for pushing textures. Both elements will still try to
use/switch to a local context where available (including after RECONFIGURE
events).
Actually set the configured framerate. Before we only used to set the first
matching framerate range. On iOS where the camera reports ranges [2, 60], we
used to configure the camera to output anything between 2 and 60fps.
Change texture format from BGRA to NV12. This allows a pipeline like avfvideosrc
! tee name=t ! ... ! glimagesink t. ! ... ! gldownload ! vtenc_h264 ! ... to
negotiate GLMemory. This makes the glimagesink branch much faster (obviously)
and triggers the 0-copy path between avfvideosrc and vtenc (using the CV meta).
Combined this results in a huge perf improvement on iOS (25-30% of CPU time in a
pipeline like the one above).
Note that this doesn't introduce a new shader conversion in the sink, since BGRA
textures had to be copied/converted from format=BGRA,texture-target=RECTANGLE to
format=RGBA,texture-target=2D anyway.
Fixate to the highest possible resolution and fps. Otherwise by default we end
up fixating at 2fps and the lowest supported resolution, which is hardly what
someone who bought an overpriced smartphone wants.
Prefer GLMemory over sysmem. Also now when pushing GLMemory we push the
original formats (UYVY in OSX, BGRA in iOS) and leave it to downstream to
convert.
First of a few commits to stop using CVOpenGLTextureCache on OSX and use
IOSurfaces directly instead. CVOpenGLTextureCache hasn't been updated for OpenGL
3 which is why texture sharing is currently disabled on OSX.
When AVFoundation indicates a supported frame rate range, add it to
the caps. This is important for devices such as the iPhone 6, which
indicate a single AVFrameRateRange of 2fps - 60fps.
https://bugzilla.gnome.org/show_bug.cgi?id=751048
Unless stopRequest is set, we should unlock conditionally -- otherwise,
the 'create:' method can wake up to an empty buffer queue
and pull a nil buffer.
https://bugzilla.gnome.org/show_bug.cgi?id=748054
Otherwise we might set bogus values or GST_CLOCK_TIME_NONE.
Also make sure to reset the caps field to NULL after unreffing
the caps to prevent accidential use afterwards, and unref any
old caps before we remember new caps.
Use YUV instead of RGB textures, then convert using the new apple specific
shader in GstGLColorConvert. Also use GLMemory directly instead of using the
GL upload meta, avoiding an extra texture copy we used to have before.
When doing texture sharing we don't need to call CVPixelBufferLockBaseAddress to
map the buffer in CPU. This cuts about 10% relative cpu time from a vtdec !
glimagesink pipeline.
Use AVF provided timings to timestamp output buffers. Use the running time at
the time the first buffer is produced to base timestamps on. Report 1-frame
latency based on the negotiated framerate instead of hardcoding 4ms latency.
We will run into an assertion in set_caps() if we try to change
caps while the source is already running. Don't try to find new
caps in GstBaseSrc::negotiate() to prevent caps changes.
Handle stride alignment through the use of the video meta API. The
code is based on the corevideobuffer implementation.
If the video meta API is not supported and the underlying buffer
contains padding, the core media buffer is copied to a system memory
buffer.
https://bugzilla.gnome.org/show_bug.cgi?id=727885
AVDeviceFormat and AVFrameRateRange are available in iOS since 7.0
so we need a more dynamic approach to support compilation with older
SDK's. We use a NSObject to avoid referencing those types and key-value
coding or preformSelector to access properties.
