gstreamer/markdown/additional/design/opengl.md

OpenGL

OpenGL is a venerable, cross-platform 3D graphics API available for use on Linux, Windows, MacOS, iOS and Android and is usually backed by specialized hardware (a GPU) to accelarate rendering. There are however, also CPU-based software implementations available for ensuring correctness and OpenGL use without a GPU.

Limits imposed by OpenGL

OpenGL and Threads

A major design decision of OpenGL has an OpenGL context only being available (or current) for use in a single thread. This is directly at odds with GStreamer's multi-threaded design and requires guidance and/or restrictions on how GStreamer and the application will interact with OpenGL. There are two main models for using OpenGL from multiple threads.

The first involves creating an OpenGL context for each thread that will use OpenGL in such a way that all the OpenGL contexts can share OpenGL resources. While this does ensure that each thread can execute and use OpenGL resources concurrently, it has a high resources cost for two main reasons.

1. OpenGL contexts are expensive to create and maintain. e.g. a single OpenGL context can use multiple MB of memory for storing all the required state.
2. Transfering between different OpenGL contexts requires synchronisation. This incurs a penalty at runtime performing the required synchronisation as well as an increase in code complexity.

The second involves making the OpenGL context current whenever it is needed. On some platforms this is possible to perform lazily as the context will be uncurrented in the previous thread automatically. Regrettably this behaviour is not guaranteed for all platforms and means that transfering an OpenGL context between threads requires explicitly uncurrenting from the previous thread. The other issue with this is performance related where on some platforms, making an OpenGL current could could consume a large amount of CPU time.

GStreamer takes a different approach to this by marshalling all the OpenGL functionality to a dedicated OpenGL thread that is created and destroyed with the GstGLContext. This removes the synchronisation requirements between multiple GStreamer OpenGL contexts, removes the need for multiple OpenGL contexts inside GStreamer reducing the memory requirements, and avoids the possibly expensive currenting and uncurrenting of the OpenGL context multiple times per frame. The marshalling is performed in the GstGLWindow implementation by gst_gl_window_send_message() / gst_gl_window_send_message_async() and is required to be re-entrant by ensuring that marshalling from the OpenGL thread must execute the marshalled operation immediately. The only downside of marshalling everything to a dedicated thread has is that OpenGL in GStreamer requires two thread switches for calling a block of OpenGL commands from outside the OpenGL thread. Given the possibly high cost of the other approaches this is an acceptable trade-off.

OpenGL Function Pointers

Another design choice of OpenGL is that (at least on the windows platform) OpenGL functions can be OpenGL context specific. This means that retrieving a function pointer from one OpenGL context has no guarentee of doing the correct thing with any other OpenGL context. Each GstGLContext has a GstGLFuncs structure that contains a list of OpenGL functions known to libgstgl that is populated by a call to gst_gl_context_fill_info().

libgstgl Library

This library provides the necessary infrastructure for integrating OpenGL usage in GStreamer between elements in the pipeline and an application that may use OpenGL. The library can be split into 3 main parts:

• Platform-specific wrappers (GstGLDisplay, GstGLWindow, GstGLContext)
• GStreamer subclasses
  • Element base clases (GstGLBaseFilter, GstGLFilter, GstGLBaseMixer, etc)
  • Data/Memory (GstGLBuffer, GstGLMemory, GstGLBufferPool, etc)
  • Synchronisation (GstGLSyncMeta)
• OpenGL helpers (GstGLShader, GstGLFramebuffer, etc)

Platform Specifics

Due to the nature of OpenGL, the use of OpenGL requires a connection to a windowing system (X11, Wayland, Cocoa, GDI+, etc). These windowing systems are entirely platform specific and are wrapped in the GstGLDisplay generic interface. Specific functionality exposed by a windowing system may be exposed by platform-specific GstGLDisplay objects. e.g. GstGLDisplayWayland or GstGLDisplayX11.

Another platform specific part of the puzzle is the GstGLWindow object which encapsulates the surface that is being rendered into by OpenGL. There are specific windowing system implementations for each platform that are all exposed through the generic GstGLWindow interface. An application generally does not need to interact with the window used by an OpenGL context with GStreamer.

The last platform specific piece is the GstGLContext which represents a OpenGL context. A GstGLContext has two main forms, wrapped (created with gst_gl_context_new_wrapped()) vs not. Some GstGLContext API behaves differently with a wrapped GstGLContext.

Wrapped GstGLContexts

A wrapped GstGLContext is simply a container to an OpenGL context handle and does not contain any separate thread to execute the OpenGL operations on. This means that gst_gl_context_thread_add() with a wrapped GstGLContext will attempt to execute the function immediately and may fail in some way (crashes, ignored) if the actual OpenGL context is not active in the current thread. Even if the OpenGL context is active in the current thread, GStreamer will not know that and needs to be told explicitly through the necessary calls to gst_gl_context_set_active().

Application Integration

There are two types of resources required for integration between GStreamer elements and the application, platform specific and generic OpenGL resources.

First, the platform specific resources are the GstGLDisplay for the required connection to the windowing system and the GstGLContext for the OpenGL context that the application is using. With these two pieces of information, it is possible for certain OpenGL resources to be 'shared' between GStreamer and the application subject to lifetime and synchronisation requirements.

Sharing X11 Display With GStreamer

static gboolean
sync_bus_call (GstBus *bus, GstMessage *msg, gpointer data)
{
  switch (GST_MESSAGE_TYPE (msg)) {
    case GST_MESSAGE_NEED_CONTEXT:
    {
      const gchar *context_type;
      GstContext *context = NULL;
     
      gst_message_parse_context_type (msg, &context_type);
      g_print("got need context %s\n", context_type);

      if (g_strcmp0 (context_type, GST_GL_DISPLAY_CONTEXT_TYPE) == 0) {
        Display *x11_display; /* get this from the application somehow */
        GstGLDisplay *gl_display = GST_GL_DISPLAY (gst_gl_display_x11_new_with_display (x11_display));

        context = gst_context_new (GST_GL_DISPLAY_CONTEXT_TYPE, TRUE);
        gst_context_set_gl_display (context, gl_display);

        gst_element_set_context (GST_ELEMENT (msg->src), context);
      }
      if (context)
        gst_context_unref (context);
      break;
    }
    default:
      break;
  }

  return FALSE;
}

Sharing OpenGL Context With GStreamer

static gboolean
sync_bus_call (GstBus *bus, GstMessage *msg, gpointer    data)
{
  switch (GST_MESSAGE_TYPE (msg)) {
    case GST_MESSAGE_NEED_CONTEXT:
    {
      const gchar *context_type;
      GstContext *context = NULL;
     
      gst_message_parse_context_type (msg, &context_type);
      g_print("got need context %s\n", context_type);

      if (g_strcmp0 (context_type, "gst.gl.app_context") == 0) {
        GstGLContext *gl_context; /* get this from the application somehow */
        GstStructure *s;

        context = gst_context_new ("gst.gl.app_context", TRUE);
        s = gst_context_writable_structure (context);
        gst_structure_set (s, "context", GST_GL_TYPE_CONTEXT, gl_context, NULL);

        gst_element_set_context (GST_ELEMENT (msg->src), context);
      }
      if (context)
        gst_context_unref (context);
      break;
    }
    default:
      break;
  }

  return FALSE;
}

Second, other OpenGL resources such as buffers or textures require wrapping into a GStreamer compatible format. For data-specific OpenGL resources, they are wrapped up in some form of GstMemory. The application can create these resources for pushing into GStreamer or consume these resources from GStreamer. See the resource specific sections for more details.

GstGLMemory

GstGLMemory is a GstMemory subclass that holds a single plane of a video frame in an OpenGL texture.

GstGLBuffer

GstGLMemory is a GstMemory subclass that holds data stored in an OpenGL buffer object.

Automatic Transfers To/From The GPU

Both GstGLMemory and GstGLBuffer implement automatic transfers to and from OpenGL based entirely on the sequence of calls to gst_memory_map() / gst_memory_unmap(). Combined with a special GstMapFlags, GST_MAP_GL, the OpenGL based GstMemory implementations can discern where the most recent data was written and copy the data on-demand to where it is needed on read.

Internally this is implemented using two extra GstMemoryFlags, GST_GL_BASE_MEMORY_TRANSFER_NEED_DOWNLOAD and GST_GL_BASE_MEMORY_TRANSFER_NEED_UPLOAD that indicate that a transfer is needed if that specific domain (system memory or OpenGL) is mapped. The transfer flags are automatically updated and should not be modified by elements or the application.

Elements

There are a number of elements that make use of the OpenGL API for their functionality. A non-comprehensive list is provided below.

• glimagesinkelement - Display a video using OpenGL
• glcolorconvert - Convert between diferent color spaces
• glviewconvert - Convert between different stereo view formats
• gltransformation - Perfom transformations in 3D space of the 2D video plane
• gleffects - Various OpenGL effects
• glvideomixerelement - Mix video using OpenGL (roughly equivalent to compositor)
• glcolorbalance - Color balance filtering
• gltestsrc - OpenGL equivalent to videotestsrc
• glshader - Execute an arbitrary OpenGL shader
• gloverlay - Overlay an image onto a video stream
• glupload - Upload data into OpenGL
• gldownload - Download data from OpenGL