This fix a very slow rendering rate regression that only
happens when using gst-launch, i.e. in the case where
the main thread does not run any NSApp loop.
Git bisect reported it has been introduced by the commit
e10d2417e2:
"move to CGL and CAOpenGLLayer for rendering".
Then the commit 7d46357627:
"gstglwindow_cocoa: fix slow render rate" attempted to fix
the slow rendering rate problem when using gst-launch.
At least for me it does not work. I tried several
combinations, for example to flush CA transactions in the
custom app loop, as mentioned in the doc, but the only solution
that fixes the slow rendering is by reducing the loop latency.
From what I tested, no need to put less than 60ms, even if the
framerate has an interval much lower (16.6ms for 60 fps).
Anytime else, we have no idea how to match up map and unmaps.
We also don't know exactly how the calling code is using us.
Also fixes the case where we're trying to transfer while someone else
is accessing our data pointer or texture resulting in mismatched video
frames.
https://bugzilla.gnome.org/show_bug.cgi?id=744839
Where possible, use the _OBJECT variants in order to track better from
which object the debug statement is coming from
Define (and use) GST_CAT_DEFAULT where applicable
Use GST_PTR_FORMAT where applicable
In gst_gl_filter_fixate_caps () it can goto done without freeing the memory of
the tmp GstStructure. This makes it go out of scope and leak.
CID #1265765
Allows finer grain decisions about formats and features at each
stage of the pipeline.
Also provide propose_allocation for glupload besed on the supported
methods.
In gst_gl_window_cocoa_draw we used to just call setNeedsDisplay:YES. That was
creating an implicit CA transaction which was getting committed at the next
runloop iteration. Since we don't know how often the main runloop is running,
and when we run it implicitly (from gst_gl_window_cocoa_nsapp_iteration) we only
do so every 200ms, use an explicit CA transaction instead and commit it
immediately. CA transactions nest and debounce automatically so this will never
result in extra work.
Make GstGLMemory hold the texture target (tex_target) the texture it represents
(tex_id) is bound to. Modify gst_gl_memory_wrapped_texture and
gst_gl_download_perform_with_data to take the texture target as an argument.
This change is needed to support wrapping textures created outside libgstgl,
which might be bound to a target other than GL_TEXTURE_2D. For example on OSX
textures coming from VideoToolbox have target GL_TEXTURE_RECTANGLE.
With this change we still keep (and sometimes imply) GL_TEXTURE_2D as the
target of textures created with libgstgl.
API: modify GstGLMemory
API: modify gst_gl_memory_wrapped_texture
API: gst_gl_download_perform_with_data
Don't call glClear && glClearColor at each draw since we're going to draw the
whole viewport anyway. Gets rid of a glFlush triggered by glClear on OSX.
In order to use pbo's efficiently, the transfer operation has to
be separated from the use of the downloaded data which requires some
rearchitecturing around glcolorconvert/gldownload and elements
Depending on the platform, it was only ever implemented to 1) set a
default surface size, 2) resize based on the video frame or 3) nothing.
Instead, provide a set_preferred_size () that elements/applications
can use to request a certain size which may be ignored for
videooverlay/other cases.
Removes the use of NSOpenGL* variety and functions. Any Cocoa
specific functions that took/returned a NSOpenGL* object now
take/return the CGL equivalents.
The hack causes deadlocks and other interesting problems and it really
can only be fixed properly inside GLib. We will include a patch for
GLib in our builds for now that handles this, and hopefully at some
point GLib will also merge a proper solution.
A proper solution would first require to refactor the polling in
GMainContext to only provide a single fd, e.g. via epoll/kqueue
or a thread like the one added by our patch. Then this single
fd could be retrieved from the GMainContext and directly integrated
into a NSRunLoop.
https://bugzilla.gnome.org/show_bug.cgi?id=741450https://bugzilla.gnome.org/show_bug.cgi?id=704374
Soon after setting two variables to 1, the code checks if their values are
different from each other. This would never be true. Removing this.
CID 1226443
Actually we should always recalculate buffer size since our buffer size
even when not-padded is smaller for many sub-sampled formats. This is
because we don't add padding between the planes.
https://bugzilla.gnome.org/show_bug.cgi?id=740900
Problem was that if buffer was mapped READWRITE (state of buffers from
libav right now), mapping it READ/GL will not upload. This is because the
flag is only set when the buffer is unmapped. We can fix this by setting
the flags in map. This result in already mapped buffer that get mapped
to be read in GL will be uploaded. The problem is that if the write
mapper makes modification afterward, the modification will never get
uploaded.
https://bugzilla.gnome.org/show_bug.cgi?id=740900
A context can create a GLsync object that can be waited on in order
to ensure that GL resources created in one context are able to be
used in another shared context without any chance of reading invalid
data.
This meta would be placed on buffers that are known to cross from
one context to another. The receiving element would then wait
on the sync object to ensure that the data to be used is complete.
Floating point numbers are written differently in different
locales, e.g. in many countries 1/2 = 0,5 instead of 0.5, and
strtod will not be able to parse "0.5" correctly in such a
locale.
Otherwise interesting things will happen in Cocoa applications, like
infinite event loops that block the NSApplication loop forever.
This was only needed for GNUStep and thus can safely be removed now.
Until gcc and GNUStep properly support Objective-C blocks and other
"new" features of Objective-C we can't properly support them without
making the code much more ugly.
https://bugzilla.gnome.org/show_bug.cgi?id=739152
It's architecture dependent and should not be placed into the include
directory as the assumption is that all those headers are architecture
independent.
https://bugzilla.gnome.org/show_bug.cgi?id=739767
Otherwise when resizing the window you will also get messages like:
class NSConcreteMapTable autoreleased with no pool in place - just leaking
class NSConcreteValue autoreleased with no pool in place - just leaking
class NSConcreteValue autoreleased with no pool in place - just leaking
class __NSCFDictionary autoreleased with no pool in place - just leaking
Need to set the ':' as the reshape method now takes one parameter.
For the story, the GstGLNSView was previously inheriting from
NSOpenGLView which has a reshape function without any parameter.
Now the GstGLNSView inherits from NSView and we re-use the reshape
function manually.
Use the reshape function after being defined. The other way
would have been to declare the reshape function in the header.
gstglwindow_cocoa.m: In function '-[GstGLNSView drawRect:]':
gstglwindow_cocoa.m:555: warning: 'GstGLNSView' may not respond to '-reshape'
gstglwindow_cocoa.m:555: warning: (Messages without a matching method signature
gstglwindow_cocoa.m:555: warning: will be assumed to return 'id' and accept
gstglwindow_cocoa.m:555: warning: '...' as arguments.)
GTK-Doc uses a special syntax for code documentation. A multiline comment that
starts with an additional '*' marks a documentation block that will be processed
by the GTK-Doc tools. So GTK-Doc is confused if a comment block starts with that
additional '*' but isn't meant to be processed. Removing this additional '*'.
https://bugzilla.gnome.org/show_bug.cgi?id=739444
gstglwindow_cocoa.m: In function '-[GstGLNSView drawRect:]':
gstglwindow_cocoa.m:555: warning: 'GstGLNSView' may not respond to '-reshape'
gstglwindow_cocoa.m:555: warning: (Messages without a matching method signature
gstglwindow_cocoa.m:555: warning: will be assumed to return 'id' and accept
gstglwindow_cocoa.m:555: warning: '...' as arguments.)
Using NSApp directly seems to confuse something, as the compiler
was expecting an id<NSFileManagerDelegate>. Switched to using
[NSApplication sharedApplication], and specified the delegate
protocol on the window class as well.
https://bugzilla.gnome.org/show_bug.cgi?id=738740
- The shader was outputing the wrong values compared with raw
videotestsrc.
- deal with the texture edge properly.
- properly sample the 2x1 rectangle for the u and v values
- don't double sample the y value
The visible rect and bounds might be the same as before, but Cocoa
might've changed our viewport without us nothing. This happens if
you hide the view and show it again.
This is only for non-Cocoa apps but previously caused a 2 second
waiting during startup for Cocoa apps. This is unacceptable.
Instead we now check a bit more extensive if something actually
runs on the GLib default main context, and if not don't even
bother waiting for something to happen from there.
Otherwise we could pass on a RGBA formatted buffer and downstream would
misinterpret that as some other video format.
Fixes pipelines of the form
gleffects ! tee ! xvimagesink
Allows callers to properly reference count the buffers used for
rendering.
Fixes a redraw race in glimagesink where the previous buffer
(the one used for redraw operations) is freed as soon as the next
buffer is uploaded.
1. glimagesink uploads in _prepare() to texture n
1.1 glupload holds buffer n
2. glimagesink _render()s texture n
3. glimagesink uploads texture n+1
3.1 glupload free previous buffer which deletes texture n
3.2 glupload holds buffer n+1
4. glwindow resize/expose
5. glimagesink redraws with texture n
The race is that the buffer n (the one used for redrawing) is freed as soon as
the buffer n+1 arrives. There could be any amount of time and number of
redraws between this event and when buffer n+1 is actually rendered and thus
replaces buffer n as the redraw source.
https://bugzilla.gnome.org/show_bug.cgi?id=736740
This thread dispatches navigation events. It is needed to avoid deadlocks
between window backend threads that emit navigation events (e.g. X11/GMainLoop
thread) and consumers of navigation events such as glimagesink, see
https://bugzilla.gnome.org/show_bug.cgi?id=733661
GstGlWindow_x11 thread is changed to invoke the navigation thread for navigation
dispatching, instead of emiting the event itself. Othe backends beside X11 do
not dispatch navigation events yet, but should use this thread when dispatching
these events in the future.
The navigation thread is currently part of GstGLWindow and not implemented in
separate subclasses / backends. This will be needed in the future.
gst_gl_window_x11_get_surface_dimensions is also changed to use a cached value
of the window's width, height. These values are now retrieved in the X11
thread, function gst_gl_window_x11_handle_event. This change is needed because
otherwise the XGetWindowAttributes gets called from the navigation thread,
leading to xlib aborting due to multithreaded access (if XInitThreads is not
called before, as is the case for gst-launch)
EGL_CONTEXT_FLAGS_KHR and EGL_CONTEXT_OPENGL_DEBUG_BIT_KHR
don't exist in the Android NDK. Wrap their usage in an #ifdef
EGL_KHR_create_context to fix the build there.
The text for EGL_KHR_create_context added the possiblity for ES
contexts to ask for a debug context however that has not been
fully realized by all implementations. Fallback to a non-debug
context when the implementation errors.
If window is resized, GstStructure pointer values have to be rescaled to
original geometry. A get_surface_dimensions GLWindow class method is added for
this purpose and used in the navigation send_event function.
https://bugzilla.gnome.org/show_bug.cgi?id=703486
Certain elements expect that there be a certain number of lines
that they can write into. e.g. for odd heights, I420, YV12, NV12,
NV21 (and others) Y lines are expected to have exactly twice the
number of U/UV lines.
https://bugzilla.gnome.org/show_bug.cgi?id=733717
We cannot do it as the winsys may crash if we initialize too late.
Example, GLX contexts with Intel drivers:
Intel requires the X Display to be the same in order to share GL
resources across GL contexts. These GL contexts are generally
accessed from different threads. Without winsys support it is
nearly impossible to guarentee that concurrent access will not
occur. This concurrent access could result in crashes or abortion
by the winsys (xcb).
https://bugzilla.gnome.org/show_bug.cgi?id=731525
The reshape property was never used.
Replace the draw property with a signal.
Based on patch by Mathieu Duponchelle <mathieu.duponchelle@epitech.eu>
https://bugzilla.gnome.org/show_bug.cgi?id=704507
Fixes a segfault with decodebin ! glmixer where the request pads on
both sides were being requested after the state change to PAUSED.
Also fixes dynamically adding and removing pads while glmixer is
in a state >= PAUSED.
Currently, GstGLWindowWaylandEGL holds the wayland display connection
If we create the EGLDisplay at the GstDisplay creation time, then
libEGL will internally open another connection to the wayland server.
These two display connections are unable to communicate resulting in
no window output/display and hangs inside libEGL.
Eventually we will move the wl_display from GstGLWindow to GstGLDisplay.
Fixes the case where _perform_with_buffer() is called without
intervening calls to _release_buffer() as is the case on start up
with glimagesink.
Also release the buffer when reseting the upload.
https://bugzilla.gnome.org/show_bug.cgi?id=731107
Fixes issues with .so (without numbering) being installed for development
(such as from mesa-dev) but actual driver (with numbering) coming from
some other place (like nvidia drivers)
Before:
GST_GL_PLATFORM=cocoa GST_GL_WINDOW=cocoa
gst-launch-1.0 videotestsrc ! glimagesink
After:
GST_GL_PLATFORM=cgl GST_GL_WINDOW=cocoa
gst-launch-1.0 videotestsrc ! glimagesink
but still pass --enable-cocoa to configure script
because currently it can only be used with cocoa API.
We could later have cgl/gstglcontext_cgl.h that manages
a CGLContextObj directly and cocoa/gstglcontext_cocoa.h
would just wrap it.
So that it could be used with other Apple's window APIs.
https://bugzilla.gnome.org/show_bug.cgi?id=729245
In a pipeline like so:
videotestsrc ! gleffects ! videoconvert ! sink
gleffects was simply passing the videoconvert bufferpool to videotestsrc
and not creating a glbufferpool. videobufferpool would then fail
to allocate from the glallocator.
From d4bcef3204 on, using a RGBA
texture to hold the data causes the glmemory to have half width
and a scaling of [2, 1]. Using a LA texture solves this problem
however cannot be attached to the framebuffer for copying into
a RGBA texture. Which will be solved by moving to EXT_texture_rg.
https://bugzilla.gnome.org/show_bug.cgi?id=728890
Previously it would only work if the alpha value was in the last
component (RGBx, BGRx). Now it works wherever the alpha value may
be (xRGB, xBGR, etc).
This fixes the OSX build and any builds with --disable-egl. That issue
was introduced in "glfilter: rewrite transform_caps to preserve caps fields".
https://bugzilla.gnome.org/show_bug.cgi?id=729861
Setting a scaled factor for X coordinate is not enough as the indexer
will still think stride is shorter and will not fully skip it. Instead,
update width, so the lines are as expected. Combined with the scale, it
will hide the cropped portion.
https://bugzilla.gnome.org/show_bug.cgi?id=729896
Fix a regression introduced recently with the lazy init.
It was happening when calling gst_video_gl_texture_upload_meta_upload
from an aplication. So not using gst_gl_upload_perform_with_buffer.
gst_video_frame_map() will store an updated video info base
on the video meta. In order to have the right stride and offset
we should update that video info accordingly.
The target texture of an EGLImage is the texture bind just
before calling glEGLImageTargetTexture2D.
As we currently only support a gl texture as the source of an EGLImage
the gl texture is actually already allocated when creating the eglimage.
I also see some cases where it fails to upload an eglimage when
glTexImage2D is called on the target.
GST_GL_WINDOW is used for window system specific choices.
An EGLDisplay can be created from window systems with egl support by using
eglGetDisplay().
https://bugzilla.gnome.org/show_bug.cgi?id=729551
This commit makes the loading of the GModules threadsafe, and
always first tries to load the symbol for the GL library that
is selected for the current context. Only then it falls back
to looking into the current module (NULL), and only as a last
resort the context specific function (e.g. eglGetProcAddress())
is called.
Also add configure parameters to select the names of the library
modules instead of using the defaults, and let the defaults be
independent of the G_MODULE_SUFFIX.
https://bugzilla.gnome.org/show_bug.cgi?id=728753
Previously, we used the width to determine the amount of data to be
copied using pbos. This, makes it allocate enough data for the
the strides as well.
Most of our 2D filters use the same simple vertex shader.
Also define the default fragment shader as the identity.
Avoid duplicating the same vertex and fragment shader text.
This should fix QoS problems, where basesink believed it was rendering with
20FPS but actually we were just queueing up X11 Expose events and only once
in a while something was rendered.
I was lucky that:
e1 = vc_dispmanx_element_add ()
eglCreateWindowSurface (e1)
vc_dispmanx_element_remove (e1)
e2 = vc_dispmanx_element_add ()
resulted in having e2 equal to e1. And also having the egl surface
that does not allocate its buffers before the first draw.
It avoids to draw to an invalid buffer.
Withtout this the default frame buffer is undefined:
glBindFramebuffer (GL_FRAMEBUFFER, 0)
Visually you could see some white frames at the beginning
when lunching videotestsrc ! glimagesink
With OpenGL Profiler from XCode you could see some
GL_INVALID_FRAMEBUFFER_OPERATION for the first frames
"(NSApplication *)sharedApplication This method also makes a connection
to the window server and completes other initialization"
The implicit thing which is not mentioned is that it required
to be called in the main thread.
Fix a regression introduces by 82b7c915bb
When using with gst-launch, it was not possible to click on the close
cross of the window anymore which is a bit anoying and also because
it's was possible before.
Prior to this commit the GstGLContextCocoaClass was initialized
in the main thread because gst_gl_context_new was called in the
state change function from going from ready to paused.
From this commit this call is done from the streaming thread.
So that the call to [NSApplication sharedApplication];
was not done in the main thread anymore.
We now ensure that by assuming there is a GMainLoop running.
It's for debugging purpose so that's ok to do that. Also
note we already do this assumtion to run app itereations.
The regression had no consequence on the cocoa/videooverlay example
(that should be moved from gst-plugins-gl to -bad) because the
application is responsible for that necessary call.
This patch provides the basic infrastructure required for this.
Upload and Download has been ported to this.
Has the nice effect of allowing GstGLMemory to be our
refcounted texture object for any texture type (not just RGBA).
Should not lose any features/video formats.
As the relevant variables are initialized to 0/NULL, we can loop
over the full range and make sure we free partial allocations
when an error happens partway through initialization.
Mesa, for example returns valid pointers for glGetIntegerv and
glGetStringi even if the gl version is less than that required for
both those functions to supposedly exist.
https://bugzilla.gnome.org/show_bug.cgi?id=727324
* picked from old libgstegl:
- GstEGLImageMemory
- GstEGLImageAllocator
- last_buffer management from removed GstEGLImageBufferPool
* add-ons:
- GstEGLImageMemory now old a reference on GstGLContext
so that it can delete the EGLImage and its gltexture source
while having the associated gl context being current.
- add EGLImage support for GstVideoGLTextureUploadMeta which
mainly call EGLImageTargetTexture2D
- GstGLBufferPool now supports GstEGLImageAllocator
- glimagesink / glfilters / etc.. now propose GstEGLImageAllocator
to upstream
https://bugzilla.gnome.org/show_bug.cgi?id=703343
Effective for the case where we have a platform that does not
require a native window. We require a mainloop to run the GL
commands which is currently operated by GstGLWindow.
Intel drivers require the display handles be the same for context
sharing to occur. Also solves some cases of use after free of the
display when integrating with gstreamer-vaapi.
See https://bugs.freedesktop.org/show_bug.cgi?id=41736 for the intel bug.
GstGLDisplayX11 holds the display connection and name. Each thread requires
it's own X11 Display connection (initialised from name) due to the fact that
we do not want to call XInitThreads(). Doing so would result in segfaults
when integrating with GUI toolkits Gtk, Qt, etc.
The Display connection is for OpenGL platforms where a constant display is
required in order to share contexts (egl). In the case of a wrapped context
(added later), we do not have GstGLWindow to retreive the display from so a
'master' connection is used instead.
We can transform from any input in our caps to any output.
With the following pipeline snippet:
... ! vaapidecode ! glcolorscale ! xvimagesink
GstVideoGLTextureUploadMeta was being used on both src and sink
pads causing linking to fail. This allows the usage of the meta
on either pad without affecting whether the meta is chosen on the
other pad.
the 16bit data is uploaded as LUMINANCE_ALPHA, then expanded, composed
in shader. value weight is a little complicate, high byte weight is
255*256/65535 (denormalize to [0~255] ,shift to high byte,then normalize
to [0~1]), low byte weight is 255/65535(similar)
https://bugzilla.gnome.org/show_bug.cgi?id=722670
