Commit graph

193 commits

Author SHA1 Message Date
Matthew Waters 1da12235c3 gl/upload/meta: remove unneeded texture deletion
The textures are managed by the GstGLMemory inside the output buffer.  Freeing
them prematurely may result in a use-after-free.
2017-12-09 19:32:20 +00:00
Matthew Waters 5efc2e8705 gl/egl: replace gsteglimagememory with an EGLImage wrapper
That can be passed to GstGLMemoryEGL.

This also ports the dmabuf uploader to GstEGLImage and GstGLMemoryEGL.
2017-12-09 19:32:19 +00:00
Gwang Yoon Hwang ecfe5b0801 gl: implement GstGLMemoryEGL
Because current GstEGLImageMemory does not inherit GstGLMemory, GLUpload
allocates additional GLMemory and upload the decoded contents from the decoder
which uses EGLImage (e.g. gst-omx in RPi).

This work adds GstGLMemoryEGL to avoid this overhead. Decoders allocate
GstGLMemoryEGL and decode its contents to the EGLImage of GstGLMemoryEGL.  And
GLUpload uses this memory without allocation of additional textures and blit
operations.

[Matthew Waters]: gst-indent the sources and fix a critical retreiving the egl
display from the memory.

https://bugzilla.gnome.org/show_bug.cgi?id=760916
2017-12-09 19:32:19 +00:00
Matthew Waters 2753b4243e glmemory: add wrapped data pointers to setup_buffer
Allows creating wrapped memories with GstGLAllocationParams.

The wrapped pointers will be set in the parameters before being passed
to the memory allocation function.
2017-12-09 19:32:19 +00:00
Matthew Waters 3ee2183038 glupload: keep the parent buffer around when converting EGLImage to 2D textures
e.g. receiving and releasing a buffer from OMX too early will potentially
cause textures to be overwritten while/before they are displayed.
2017-12-09 19:32:19 +00:00
Matthew Waters 356494d967 glupload: deal with the ANY caps feature correctly
When transforming, xplode it out into the necessary caps features both
with and without the passthough features.

Fixes negotiation in the following class of pipelines:

gl ! textoverlay ! glupload ! glimagesinkelement

https://bugzilla.gnome.org/show_bug.cgi?id=763756
2017-12-09 19:32:17 +00:00
Matthew Waters 31b1094744 gl: misc docs fixes/additions 2017-12-09 19:32:17 +00:00
Matthew Waters 86110c4765 glupload: always add texture-target field to GL caps
1. Various elements/base classes only perform a subset check on accept-caps
2. Some GL elements have texture-target in their pad template
3. When checking subsets, only the caps to check are allowed to contain extra
   fields.  If the 'template' caps have extra fields, the subset fails.
Thus without texture-target on the caps, various accept-caps implementations
were failing.

Also, add some convenience functions for setting and retrieving
texture targets to/from GValue.

https://bugzilla.gnome.org/show_bug.cgi?id=759860
2017-12-09 19:32:14 +00:00
Matthew Waters ec5eb678d0 glupload: passthrough composition caps features
Don't unconditionally add it to any and all caps transformations.

https://bugzilla.gnome.org/show_bug.cgi?id=759860
2017-12-09 19:32:14 +00:00
Nicolas Dufresne 8492c045a4 glupload: Add dmabuf upload method.
This upload method detect and optimize uploads of DMABuf memory. This is
done by creating and caching EGLImages wrapper around DMABuf. The
EGLImages are then binded to a texture which get converter using
standard shader.

Example pipeline:

GST_GL_PLATFORM=egl \
gst-launch-1.0 v4l2src device=/dev/video1 io-mode=4 ! \
               video/x-raw,format=NV12 ! glimagesink

https://bugzilla.gnome.org/show_bug.cgi?id=743345
2017-12-09 19:32:13 +00:00
Alessandro Decina 28dbe4fffc glmemory: add gst_gl_memory_allocator_get_default
Add gst_gl_memory_allocator_get_default to get the default allocator based on
the opengl version. Allows us to stop hardcoding the PBO allocator which isn't
supported on gles2.

Fixes GL upload on iOS9 among other things.
2017-12-09 19:32:13 +00:00
Matthew Waters 09c8c23842 gl*memory*: reverse the parameter order of user_data and destroy notify
The convention is to have the destroy notify last after any user data
2017-12-09 19:32:13 +00:00
Matthew Waters 1673a6192f glmemorypbo: remove our own alloc()/wrapped()/etc functions
replaced by equivalent functionality within gst_gl_base_memory_alloc()
2017-12-09 19:32:13 +00:00
Alessandro Decina b365425dc2 glupload: fix allocator name after GLMemory API changes 2017-12-09 19:32:12 +00:00
Matthew Waters 76a38dd66a glmemory: base classify and add the pbo memory on top
The base class is useful for having multiple backing memory types other
than the default.  e.g. IOSurface, EGLImage, dmabuf?

The PBO transfer logic is now inside GstGLMemoryPBO which uses GstGLBuffer
to manage the PBO memory.

This also moves the format utility functions into their own file.
2017-12-09 19:32:12 +00:00
Nicolas Dufresne 21df596b8b glupload: Only offer custom allocator with caps features
To use GLMemory and EGLImage allocators, one need to know the
libgstgl API. This is only expected if the associated caps features
have been negotiated. Generic element that otherwise receive those
allocators may fail, resulting in broken pieline. We don't want to
force all generic element to check if the allocator is a custom
allocator or a normal allocator (which implement the _alloc method).

https://bugzilla.gnome.org/show_bug.cgi?id=758877
2017-12-09 19:32:11 +00:00
Matthew Waters 434bbc9f01 glupload: explicitly configure the texture target on the gl buffer pool
If we don't, then the pool could end up allocating 2D textures when the
caps explictly state they should be rectangle textures.
2017-12-09 19:32:10 +00:00
Matthew Waters 681cb5b16d glmemory: add support for rectangle textures
Add the various tokens/strings for the differnet texture types (2D, rect, oes)

Changes the GLmemory api to include the GstGLTextureTarget in all relevant
functions.

Update the relevant caps/templates for 2D only textures.
2017-12-09 19:32:10 +00:00
Matthew Waters 73bad9e856 glupload: remove useless release_buffer
It's a leftover from when we weren't outputting GstBuffer's and
returning raw texture id's.
2017-12-09 19:32:09 +00:00
Nicolas Dufresne edd3c007ea glupload: Recalculate offset and size in raw upload
As we only expose the mapped portion of the frame into the GL
memory object (and not the original padding) we need to
re-calculate the size and offset.
2017-12-09 19:32:08 +00:00
Nicolas Dufresne a342d95ca6 glsink: Enable sync meta on pools we offer
As the upload is asynchronous, we need to enable the sync meta to
gain correct rendering. The buffer pool receiver don't know about
that.
2017-12-09 19:32:07 +00:00
Sebastian Dröge 260104e485 glupload: Remove debug output from gst_gl_upload_transform_caps()
We can't know if the GstGLUpload type is initialized at this point already,
and thus our debug category might not be initialized yet... and cause an
assertion here.

As we don't print debug output for any of the other transform functions, let's
defer this problem for now.
2017-12-09 19:32:07 +00:00
Nicolas Dufresne f8dbf10e9e glupload: Add fixme about using bufferpool for raw
http://bugzilla.gnome.org/show_bug.cgi?id=752937
2017-12-09 19:32:07 +00:00
Nicolas Dufresne 2b2048d601 glupload: Keep input frame mapped as long as needed
When performing a raw upload, we need to keep the raw data mapped as
long as needed.

https://bugzilla.gnome.org/show_bug.cgi?id=752937
2017-12-09 19:32:07 +00:00
Nicolas Dufresne 37cc53d26f Revert "glupload: memcpy on raw data upload"
This reverts commit 82c0189b28.

https://bugzilla.gnome.org/show_bug.cgi?id=752937
2017-12-09 19:32:07 +00:00
Jan Schmidt 3bacd64bf1 glupload: Check that caps contain desired caps features
Use 'contains' checks instead of equality checks on caps features
to allow for uploading when caps also contain GstVideoOverlayComposition
meta.

https://bugzilla.gnome.org/show_bug.cgi?id=752912
2017-12-09 19:32:06 +00:00
Luis de Bethencourt 1eeeadb24e glupload: fix memory leak
GstCapsFeatures need to be freed with gst_caps_features_free() after use.

CID #1312136, CID #1312136
2017-12-09 19:32:06 +00:00
Lubosz Sarnecki 076d0517c8 glupload: Detect overlay meta buffers correctly
https://bugzilla.gnome.org/show_bug.cgi?id=745107
2017-12-09 19:32:06 +00:00
Lubosz Sarnecki dc47e6d3c0 glimagesinkbin: Add GstVideoOverlayCompositionMeta caps features
https://bugzilla.gnome.org/show_bug.cgi?id=745107
2017-12-09 19:32:06 +00:00
Lubosz Sarnecki eb618aaac7 glupload: Move debug init to top of the file
https://bugzilla.gnome.org/show_bug.cgi?id=745107
2017-12-09 19:32:05 +00:00
Matthew Waters 5043800a1d glupload: memcpy on raw data upload
Anything else requires keeping track of the GstVideoFrame mapping
across possible multiple buffers to ensure correct data pointer
usage.
2017-12-09 19:32:05 +00:00
Nicolas Dufresne b8d3c2d2a5 gl: Don't leak pool if set_config failed 2017-12-09 19:32:05 +00:00
Matthew Waters 2210ba5d46 glmemory: separate pbo transfer from texture transfers
When supported, the potentially longer pbo upload/download can be
initiated before the texture upload/download, potentially increasing
throughput.
2017-12-09 19:32:04 +00:00
Matthew Waters 900bcea9de glmemory: implement on top of glbasebuffer
Provides convenient access to PBO usage.

Currently texture updates are coupled tightly to data transfers.
2017-12-09 19:32:04 +00:00
Jan Schmidt 8a1fc4ab4e gl: Add support for multiple views in upload, colour convert and download
Support multiple attached views on input/output buffers
by processing each one, not just the first.
2017-12-09 19:32:04 +00:00
Jan Schmidt 95b08b75ba glupload: Don't leak all memory uploaded via raw uploads.
Remove an extra ref missed when switching over to not
reusing output textures
2017-12-09 19:32:02 +00:00
Matthew Waters 62c2d98c52 glupload: don't reuse output textures for raw data upload
Causes stale textures to be used further down the chain.
2017-12-09 19:32:01 +00:00
Jan Schmidt 9958df5dd1 glupload: Make upload method static caps non-const.
Retrieving a GstStaticCaps does a one time internal
initialisation and caches it - they can't be stored as
const structures.
2017-12-09 19:32:00 +00:00
Matthew Waters c902a6f525 glupload: provide the sink template caps that could be used
https://bugzilla.gnome.org/show_bug.cgi?id=746399
2017-12-09 19:32:00 +00:00
Matthew Waters 0b5c7e8b49 gl/docs: update some doc comments 2017-12-09 19:31:58 +00:00
Víctor Manuel Jáquez Leal 55dbb25dcc glupload: move meta-data copy into gst-gl library
In some upload implementations the out buffer has more than one references,
turning the buffer not writable, so it won't be possible to modify its
meta-data.

This patch moves the meta-data copy before increasing the reference of the out
buffer.

https://bugzilla.gnome.org/show_bug.cgi?id=746173
2017-12-09 19:31:58 +00:00
Matthew Waters 7b9d70944b glupload: unref the bufferpool 2017-12-09 19:31:57 +00:00
Matthew Waters 1ba38bc10a glupload: implement propose_allocation pool handling for glmemory upload 2017-12-09 19:31:56 +00:00
Sebastian Dröge 86d1280f2c glupload: Simplify caps feature setting code 2017-12-09 19:31:56 +00:00
Nicolas Dufresne de79830b05 glmemory: Add GstAllocationParams and alignment support
This implements support for GstAllocationParams and memory alignments.
The parameters where simply ignored which could lead to crash on
certain platform when used with libav and no luck.

https://bugzilla.gnome.org/show_bug.cgi?id=744246
2017-12-09 19:31:55 +00:00
Matthew Waters 05a4c89f25 glupload: perform egl upload on the gl thread
Fixes EGLImage usage on raspberry pi

https://bugzilla.gnome.org/show_bug.cgi?id=743914
2017-12-09 19:31:55 +00:00
Víctor Manuel Jáquez Leal e3d457ec8e glupload: Set freed method implementation to NULL
When trying to render buffers with meta:GLTextureUpload the glimagesink crashes
with a segmentation fault.

This patch workarounds this crash setting to NULL the method implementation
after free.

https://bugzilla.gnome.org/show_bug.cgi?id=745206
2017-12-09 19:31:55 +00:00
Edward Hervey f56c4c541a gl: Add/Update more debug statements
Where possible, use the _OBJECT variants in order to track better from
which object the debug statement is coming from

Define (and use) GST_CAT_DEFAULT where applicable

Use GST_PTR_FORMAT where applicable
2017-12-09 19:31:54 +00:00
Matthew Waters 608a3ddc7b glupload: restrict EGL and UploadMeta to RGBA
other formats are highly untested.
2017-12-09 19:31:54 +00:00
Matthew Waters f180fe8dc8 glupload: only try the glmemory upload for sysmem/glmemory caps features
Allows selecting/testing the upload path with explicit caps features.
2017-12-09 19:31:54 +00:00
Matthew Waters 29ca0e8a44 glupload: allow NULL buffer for accept vfunc 2017-12-09 19:31:54 +00:00
Matthew Waters 45e6e77ff0 glupload: remove unsed convert field in private struct 2017-12-09 19:31:54 +00:00
Matthew Waters 530ae2cbb0 glupload/download/convert: provide transform_caps functions
Allows finer grain decisions about formats and features at each
stage of the pipeline.
Also provide propose_allocation for glupload besed on the supported
methods.
2017-12-09 19:31:54 +00:00
Sebastian Dröge 39e719fb33 Constify some static arrays everywhere 2017-12-09 19:31:53 +00:00
Matthew Waters 1dfeae2c42 gl: split glcolorconvert usage from glupload
the separation allows the transfer operation to occur in a separate
thread/time which may increase performance in specific circumstances.
2017-12-09 19:31:52 +00:00
Matthew Waters 4288866e55 glupload: relax the GLMemory uploader check for input caps features
As there may be a sysmem caps features with GLMemory filled buffers.
e.g.

videotestsrc ! glimagesink
2017-12-09 19:31:52 +00:00
Nicola Murino 209fbd950f glupload: fix some memory leaks
https://bugzilla.gnome.org/show_bug.cgi?id=741943
2017-12-09 19:31:52 +00:00
Nicolas Dufresne a1e02726ee gl: Add support for GstVideoAlignment
This allow saving a copy with libav video decoders or decoders with
similar padding requirement.

https://bugzilla.gnome.org/show_bug.cgi?id=740900
2017-12-09 19:31:52 +00:00
Sebastian Dröge 3c7095e5ad glupload: Fix valid compiler warning
gstglupload.c:442:32: error: if statement has empty body [-Werror,-Wempty-body]
    if (upload->texture_ids[i]);
                               ^
2017-12-09 19:31:50 +00:00
Matthew Waters 00506bf183 glupload: rearchitecture for non GLMemory inputs/outputs
Allows other memory types to be implemented/returned/used by the caller.
2017-12-09 19:31:50 +00:00
Matthew Waters ccf8f014f0 glupload: fallback to upload if contexts cannot share GL resources
Fixes pipelines of the form

    gleffects ! tee ! glimagesink

as tee does not pass on the allocation query.
2017-12-09 19:31:46 +00:00
Matthew Waters 6491af9ffc glupload: provide the output buffer that is rendered into
Allows callers to properly reference count the buffers used for
rendering.

Fixes a redraw race in glimagesink where the previous buffer
(the one used for redraw operations) is freed as soon as the next
buffer is uploaded.

1. glimagesink uploads in _prepare() to texture n
1.1 glupload holds buffer n
2. glimagesink _render()s texture n
3. glimagesink uploads texture n+1
3.1 glupload free previous buffer which deletes texture n
3.2 glupload holds buffer n+1
4. glwindow resize/expose
5. glimagesink redraws with texture n

The race is that the buffer n (the one used for redrawing) is freed as soon as
the buffer n+1 arrives.  There could be any amount of time and number of
redraws between this event and when buffer n+1 is actually rendered and thus
replaces buffer n as the redraw source.

https://bugzilla.gnome.org/show_bug.cgi?id=736740
2017-12-09 19:31:46 +00:00
Wang Xin-yu (王昕宇) f322a331c8 glupload: don't determine if frame needs upload by pointer compare
https://bugzilla.gnome.org/show_bug.cgi?id=734269
2017-12-09 19:31:45 +00:00
Matthew Waters c216799b20 glmemory: use GstVideoInfo everywhere
Simplifies a lot of the calling code

https://bugzilla.gnome.org/show_bug.cgi?id=733717
2017-12-09 19:31:45 +00:00
Matthew Waters 57a54e9de4 glupload: always release the previous buffer
Fixes the case where _perform_with_buffer() is called without
intervening calls to _release_buffer() as is the case on start up
with glimagesink.

Also release the buffer when reseting the upload.

https://bugzilla.gnome.org/show_bug.cgi?id=731107
2017-12-09 19:31:42 +00:00
Matthew Waters 4a8b31afd6 glfeature: remove GST_GL_API_GLES3
instead check the gl version using gst_gl_context_check_gl_version()
2017-12-09 19:31:42 +00:00
Sebastian Dröge 631278270f glupload: Fix assertion after API changes 2017-12-09 19:31:42 +00:00
Matthew Waters a22cb227a9 gl/upload: re add the upload meta to the private struct
try and avoid being to trigger happy
2017-12-09 19:31:42 +00:00
Matthew Waters 8b50888179 gl/upload: small code cleanup and gst-indent 2017-12-09 19:31:42 +00:00
Matthew Waters a147b2258b gl/upload: use GstObject's lock instead of our own 2017-12-09 19:31:42 +00:00
Matthew Waters 8ac92cd90d gl/colorconvert: allocate output buffers
Allows the nop optimisation by simply reffing the input buffer.
2017-12-09 19:31:41 +00:00
Matthew Waters 32343333aa gl/upload: add GstGLUploadMeta object
That simply deals with the provider aspect of GstVideoGLTextureUploadMeta.
2017-12-09 19:31:41 +00:00
Matthew Waters 625efc1757 gl/colorconvert: operate on buffers instead of memories 2017-12-09 19:31:41 +00:00
Matthew Waters 132a233f7b gl/memory: implement GL_EXT_texture_rg support
Which is used by default over the Luminance formats due to it
being color renderable with fbos (and deprecation/removal with
GL 3.x).

https://bugzilla.gnome.org/show_bug.cgi?id=729750
https://bugzilla.gnome.org/show_bug.cgi?id=704222
https://bugzilla.gnome.org/show_bug.cgi?id=728890
2017-12-09 19:31:41 +00:00
Matthew Waters 313dfbb500 gl/upload: switch the 'master' texture for YUY2, UYVY
From d4bcef3204 on, using a RGBA
texture to hold the data causes the glmemory to have half width
and a scaling of [2, 1].  Using a LA texture solves this problem
however cannot be attached to the framebuffer for copying into
a RGBA texture.  Which will be solved by moving to EXT_texture_rg.

https://bugzilla.gnome.org/show_bug.cgi?id=728890
2017-12-09 19:31:41 +00:00
Nicolas Dufresne 030fc8b7d0 glupload: Ensure we still have a texture after upload_set_format()
gst_gl_upload_set_format() resets the upload, hence the texture.
So we need to ensure we have a texture after this call when
uploading.
2017-12-09 19:31:40 +00:00
Nicolas Dufresne b70edbff15 glupload: Correctly update the video info from video meta
Using gst_video_info_set_format() isn't complete when updating
a video info from video meta.
2017-12-09 19:31:40 +00:00
Nicolas Dufresne 2f705ac9ca glupload: Correctly update video info in upload fallback
When the upload accelerated method fails, we fallback to an upload,
but the video info was not correctly updated.
2017-12-09 19:31:40 +00:00
Julien Isorce c16ab9e66b gl/upload: set out_tex after reset, not before
Fix a regression introduced recently with the lazy init.
It was happening when calling gst_video_gl_texture_upload_meta_upload
from an aplication. So not using gst_gl_upload_perform_with_buffer.
2017-12-09 19:31:40 +00:00
Matthew Waters 5cdfd51172 gl/upload: update the video info on mapping a video frame
The buffer should contain the most specific data on how the data is
formatted.  We should use this information.

https://bugzilla.gnome.org/show_bug.cgi?id=729896
2017-12-09 19:31:40 +00:00
Matthew Waters 59c388abca gl/upload: avoid performing color conversion when there is no need
One such example is when the buffer contains GstGLMemory in the
RGBA format

https://bugzilla.gnome.org/show_bug.cgi?id=729278
2017-12-09 19:31:40 +00:00
Matthew Waters 9fffa2cbc6 gl/upload: update the respective state when we cache textures/data 2017-12-09 19:31:40 +00:00
Matthew Waters 484ae5a8c5 gl/upload: cache the textures that we are using 2017-12-09 19:31:40 +00:00
Matthew Waters a64efacd0e gl/upload: avoid recreating the GLMemory struct for output textures 2017-12-09 19:31:40 +00:00
Nicolas Dufresne 5c8acc21fe gl: Fix some of the error handling 2017-12-09 19:31:40 +00:00
Nicolas Dufresne 94aa3ea228 glupload: Update GstVideoInfo after the buffer is mapped
gst_video_frame_map() will store an updated video info base
on the video meta. In order to have the right stride and offset
we should update that video info accordingly.
2017-12-09 19:31:40 +00:00
Matthew Waters 977c326c59 gl/colorconvert: implement lazy init 2017-12-09 19:31:40 +00:00
Julien Isorce 17ba668c7a gl/upload: set initted to TRUE when _init_upload succeeded 2017-12-09 19:31:40 +00:00
Matthew Waters 83c2b906d1 gl/upload: fix compilation on OS X 2017-12-09 19:31:40 +00:00
Matthew Waters 862df41b2d gl/upload: fail propoerly if init fails 2017-12-09 19:31:39 +00:00
Matthew Waters 857200ff40 gl: make all GObjects inherit from GstObject
Aids memory leak debugging with GST_TRACE=mem-live
2017-12-09 19:31:39 +00:00
Matthew Waters 6c1a4e30c4 gl/upload: implement lazy init
Rename init_format to set_format
2017-12-09 19:31:39 +00:00
Matthew Waters cda89d7ce0 gl/upload: add get_format method
Simply retreives the format set by init_format
2017-12-09 19:31:39 +00:00
Julien Isorce d7c5444447 gl: do not allocate the target texture of an EGLImage
The target texture of an EGLImage is the texture bind just
before calling glEGLImageTargetTexture2D.

As we currently only support a gl texture as the source of an EGLImage
the gl texture is actually already allocated when creating the eglimage.

I also see some cases where it fails to upload an eglimage when
glTexImage2D is called on the target.
2017-12-09 19:31:39 +00:00
Julien Isorce dd1c2b4c4c gl: delete texture used with GstVideoGLTextureUploadMeta 2017-12-09 19:31:39 +00:00
Matthew Waters 33091ff6a5 gl/mem: implement texture copying between formats with strides properly
Previously, we used the width to determine the amount of data to be
copied using pbos.  This, makes it allocate enough data for the
the strides as well.
2017-12-09 19:31:38 +00:00
Julien Isorce e4b832db90 gl: fix declaration-definition mismatch for _init_upload 2017-12-09 19:31:37 +00:00
Matthew Waters 92c243be62 gl/color: pass large value by reference
Coverity 1199700
2017-12-09 19:31:36 +00:00
Matthew Waters 30c6efc432 gl/mem: allocate the memory per plane
This patch provides the basic infrastructure required for this.
Upload and Download has been ported to this.

Has the nice effect of allowing GstGLMemory to be our
refcounted texture object for any texture type (not just RGBA).

Should not lose any features/video formats.
2017-12-09 19:31:36 +00:00
Matthew Waters fba49c3463 gl: add colorconvert object that converts between color spaces/formats
Currently used by both upload and download objects separately.
2017-12-09 19:31:36 +00:00