diff --git a/markdown/additional/design/opengl.md b/markdown/additional/design/opengl.md new file mode 100644 index 0000000000..6a79cfbc77 --- /dev/null +++ b/markdown/additional/design/opengl.md @@ -0,0 +1,241 @@ +# OpenGL + +OpenGL is a venerable, cross-platform 3D graphics API available for use +on Linux, Windows, MacOS, iOS and Android and is usually backed by +specialized hardware (a GPU) to accelarate rendering. There are however, +also CPU-based software implementations available for ensuring correctness and +OpenGL use without a GPU. + +## Limits imposed by OpenGL + +### OpenGL and Threads + +A major design decision of OpenGL has an OpenGL context only being available +(or current) for use in a single thread. This is directly at odds with +GStreamer's multi-threaded design and requires guidance and/or restrictions on +how GStreamer and the application will interact with OpenGL. There are two +main models for using OpenGL from multiple threads. + +The first involves creating an OpenGL context for each thread that will use +OpenGL in such a way that all the OpenGL contexts can share OpenGL resources. +While this does ensure that each thread can execute and use OpenGL resources +concurrently, it has a high resources cost for two main reasons. + + 1. OpenGL contexts are expensive to create and maintain. e.g. a single OpenGL + context can use multiple MB of memory for storing all the required state. + 2. Transfering between different OpenGL contexts requires synchronisation. + This incurs a penalty at runtime performing the required synchronisation + as well as an increase in code complexity. + +The second involves making the OpenGL context current whenever it is +needed. On some platforms this is possible to perform lazily as the context +will be uncurrented in the previous thread automatically. Regrettably this +behaviour is not guaranteed for all platforms and means that transfering an +OpenGL context between threads requires explicitly uncurrenting from the +previous thread. The other issue with this is performance related where on +some platforms, making an OpenGL current could could consume a large amount of +CPU time. + +GStreamer takes a different approach to this by marshalling all the OpenGL +functionality to a dedicated OpenGL thread that is created and destroyed with +the `GstGLContext`. This removes the synchronisation requirements between +multiple GStreamer OpenGL contexts, removes the need for multiple OpenGL +contexts inside GStreamer reducing the memory requirements, and avoids the +possibly expensive currenting and uncurrenting of the OpenGL context multiple +times per frame. The marshalling is performed in the `GstGLWindow` +implementation by `gst_gl_window_send_message()` / +`gst_gl_window_send_message_async()` and is required to be re-entrant by +ensuring that marshalling from the OpenGL thread must execute the marshalled +operation immediately. The only downside of marshalling everything to a +dedicated thread has is that OpenGL in GStreamer requires two +thread switches for calling a block of OpenGL commands from outside the +OpenGL thread. Given the possibly high cost of the other approaches +this is an acceptable trade-off. + +### OpenGL Function Pointers + +Another design choice of OpenGL is that (at least on the windows platform) +OpenGL functions can be OpenGL context specific. This means that retrieving a +function pointer from one OpenGL context has no guarentee of doing the correct +thing with any other OpenGL context. Each `GstGLContext` has a `GstGLFuncs` +structure that contains a list of OpenGL functions known to libgstgl that is +populated by a call to `gst_gl_context_fill_info()`. + +## libgstgl Library + +This library provides the necessary infrastructure for integrating OpenGL usage +in GStreamer between elements in the pipeline and an application that may use +OpenGL. The library can be split into 3 main parts: + + - Platform-specific wrappers (`GstGLDisplay`, `GstGLWindow`, `GstGLContext`) + - GStreamer subclasses + - Element base clases (`GstGLBaseFilter`, `GstGLFilter`, `GstGLBaseMixer`, etc) + - Data/Memory (`GstGLBuffer`, `GstGLMemory`, `GstGLBufferPool`, etc) + - Synchronisation (`GstGLSyncMeta`) + - OpenGL helpers (`GstGLShader`, `GstGLFramebuffer`, etc) + +### Platform Specifics + +Due to the nature of OpenGL, the use of OpenGL requires a connection to a +windowing system (X11, Wayland, Cocoa, GDI+, etc). These windowing systems +are entirely platform specific and are wrapped in the `GstGLDisplay` generic +interface. Specific functionality exposed by a windowing system may be exposed +by platform-specific `GstGLDisplay` objects. e.g. `GstGLDisplayWayland` or +`GstGLDisplayX11`. + +Another platform specific part of the puzzle is the `GstGLWindow` object which +encapsulates the surface that is being rendered into by OpenGL. There are +specific windowing system implementations for each platform that are all exposed +through the generic `GstGLWindow` interface. An application generally does not +need to interact with the window used by an OpenGL context with GStreamer. + +The last platform specific piece is the `GstGLContext` which represents a +OpenGL context. A `GstGLContext` has two main forms, wrapped (created with +`gst_gl_context_new_wrapped()`) vs not. Some `GstGLContext` API behaves +differently with a wrapped `GstGLContext`. + +#### Wrapped `GstGLContext`s + +A wrapped `GstGLContext` is simply a container to an OpenGL context handle and +does not contain any separate thread to execute the OpenGL operations on. +This means that `gst_gl_context_thread_add()` with a wrapped `GstGLContext` +will attempt to execute the function immediately and may fail in some way +(crashes, ignored) if the actual OpenGL context is not active in the current +thread. Even if the OpenGL context is active in the current thread, GStreamer +will not know that and needs to be told explicitly through the necessary calls +to `gst_gl_context_set_active()`. + +### Application Integration + +There are two types of resources required for integration between GStreamer +elements and the application, platform specific and generic OpenGL resources. + +First, the platform specific resources are the `GstGLDisplay` for the +required connection to the windowing system and the `GstGLContext` for the +OpenGL context that the application is using. With these two pieces of +information, it is possible for certain OpenGL resources to be 'shared' +between GStreamer and the application subject to lifetime and synchronisation +requirements. + +#### Sharing X11 Display With GStreamer + +```c +static gboolean +sync_bus_call (GstBus *bus, GstMessage *msg, gpointer data) +{ + switch (GST_MESSAGE_TYPE (msg)) { + case GST_MESSAGE_NEED_CONTEXT: + { + const gchar *context_type; + GstContext *context = NULL; + + gst_message_parse_context_type (msg, &context_type); + g_print("got need context %s\n", context_type); + + if (g_strcmp0 (context_type, GST_GL_DISPLAY_CONTEXT_TYPE) == 0) { + Display *x11_display; /* get this from the application somehow */ + GstGLDisplay *gl_display = GST_GL_DISPLAY (gst_gl_display_x11_new_with_display (x11_display)); + + context = gst_context_new (GST_GL_DISPLAY_CONTEXT_TYPE, TRUE); + gst_context_set_gl_display (context, gl_display); + + gst_element_set_context (GST_ELEMENT (msg->src), context); + } + if (context) + gst_context_unref (context); + break; + } + default: + break; + } + + return FALSE; +} +``` + +#### Sharing OpenGL Context With GStreamer + +```c +static gboolean +sync_bus_call (GstBus *bus, GstMessage *msg, gpointer data) +{ + switch (GST_MESSAGE_TYPE (msg)) { + case GST_MESSAGE_NEED_CONTEXT: + { + const gchar *context_type; + GstContext *context = NULL; + + gst_message_parse_context_type (msg, &context_type); + g_print("got need context %s\n", context_type); + + if (g_strcmp0 (context_type, "gst.gl.app_context") == 0) { + GstGLContext *gl_context; /* get this from the application somehow */ + GstStructure *s; + + context = gst_context_new ("gst.gl.app_context", TRUE); + s = gst_context_writable_structure (context); + gst_structure_set (s, "context", GST_GL_TYPE_CONTEXT, gl_context, NULL); + + gst_element_set_context (GST_ELEMENT (msg->src), context); + } + if (context) + gst_context_unref (context); + break; + } + default: + break; + } + + return FALSE; +} +``` + +Second, other OpenGL resources such as buffers or textures require wrapping +into a GStreamer compatible format. For data-specific OpenGL resources, they +are wrapped up in some form of `GstMemory`. The application can create these +resources for pushing into GStreamer or consume these resources from GStreamer. +See the resource specific sections for more details. + +### `GstGLMemory` + +`GstGLMemory` is a `GstMemory` subclass that holds a single plane of a video +frame in an OpenGL texture. + +### `GstGLBuffer` + +`GstGLMemory` is a `GstMemory` subclass that holds data stored in an OpenGL +buffer object. + +### Automatic Transfers To/From The GPU + +Both `GstGLMemory` and `GstGLBuffer` implement automatic transfers to and from +OpenGL based entirely on the sequence of calls to `gst_memory_map()` / +`gst_memory_unmap()`. Combined with a special `GstMapFlags`, `GST_MAP_GL`, the +OpenGL based `GstMemory` implementations can discern where the most recent +data was written and copy the data on-demand to where it is needed on read. + +Internally this is implemented using two extra `GstMemoryFlags`, +`GST_GL_BASE_MEMORY_TRANSFER_NEED_DOWNLOAD` and +`GST_GL_BASE_MEMORY_TRANSFER_NEED_UPLOAD` that indicate that a transfer is +needed if that specific domain (system memory or OpenGL) is mapped. +The transfer flags are automatically updated and should not be modified by +elements or the application. + +## Elements + +There are a number of elements that make use of the OpenGL API for their +functionality. A non-comprehensive list is provided below. + + - `glimagesinkelement` - Display a video using OpenGL + - `glcolorconvert` - Convert between diferent color spaces + - `glviewconvert` - Convert between different stereo view formats + - `gltransformation` - Perfom transformations in 3D space of the 2D video plane + - `gleffects` - Various OpenGL effects + - `glvideomixerelement` - Mix video using OpenGL (roughly equivalent to compositor) + - `glcolorbalance` - Color balance filtering + - `gltestsrc` - OpenGL equivalent to videotestsrc + - `glshader` - Execute an arbitrary OpenGL shader + - `gloverlay` - Overlay an image onto a video stream + - `glupload` - Upload data into OpenGL + - `gldownload` - Download data from OpenGL + diff --git a/sitemap.txt b/sitemap.txt index 566a2c27f5..fae510f768 100644 --- a/sitemap.txt +++ b/sitemap.txt @@ -173,6 +173,7 @@ additional/design/miniobject.md additional/design/missing-plugins.md additional/design/negotiation.md + additional/design/opengl.md additional/design/orc-integration.md additional/design/overview.md additional/design/playbin.md