gstreamer/docs/design/draft-va.txt

Video Acceleration
------------------

Status:

  DRAFT. Outdated for 1.0, we want to use video/x-raw in order to
  interoperate with all elements that handle video. Optimized handling of
  the VA surfaces can be done with the metadata.
  
Purpose:

  Provide an standardized generic way to introduce Video Acceleration APIs in
  already available elements instead of duplicating those into specialized ones.
  
  Provide a mechanism for a light GstBuffer subclassing in order to be able
  exchange VA related context and surfaces between elements.
  
  Provide some basic primitives to be used in the elements keeping
  the implementation of those in the backends of the helper library.
  
  The motivation of this proposal is: 
    - to avoid multiple duplicty of code
    - to avoid the use of GstBuffer subclassing and be more close to GstBuffer2
    - to avoid the overhead that's introduced with subclassing/GType checks 
    - to permit multiple elements interact having a common standard

Proposal:

   video/x-raw-va
   
   Light subclassing embedding an structure in the data field of a standard
   GstBuffer.
   
   struct {
     context;
     surface;
     flags;
     subpicture, x, y, w, h;
     parent *
     ...
   } VASurface 
        
   GstVA helper library
   --------------------
   
   Common API that it's implemented by a backend for each VA API. 
   
   +-------+      +---------------+
   |       |----> | VDPAU backend |
   | GstVA |      +---------------+
   |       |      +---------------+
   |       |----> | VAAPI backend |
   +-------+      +---------------+
   
   gst_va_init () : Iterate in the backends and try to initialize those, stops
   when a backend is able to be created.
   
   bool gst_va_available() : permit to query if there's VA available in runtime.
   
   GstBuffer * gst_va_buffer_new (vacontext, vasurface, vaflags, parent*, 
     free_func*):
   Create a new GstBuffer with the proper light subclass and store on it
   the provided context, surface, flags...
   - parent* and free_func* to implement a mechanism to return surfaces in 
     the pool driven by parent.
   
   gst_va_buffer_get_context (GstBuffer *)
   gst_va_buffer_get_surface (GstBuffer *)
   gst_va_buffer_get_flags (GstBuffer *)
   ....
   Some public/private? accessors to be able recover fields from a VA GstBuffer
   internal structure.
   
   Primitives:
   
   gst_va_render_display (GstBuffer * va, display, x, y, w, h, ...)
   Put a surface in the screen at the specified position.

   gst_va_render_pixmap (GstBuffer * va, pixmap *, w, h, ...)
   Put a surface in a pixmap, to be used paired with GLX_texture_from_pixmap to
   upload into a OpenGL texture.
   
   gst_va_render_memory (GstBuffer * va, planes *, strides *, w, h, ...) 
   To pull back into main memory a VA surface.
   
   gst_va_buffer_associate_subpicture (GstBuffer * va, GstBuffer * subpicture,
      x, y, w, h,...)
   Take a reference of subpicture and keep it in va internal structure.
  
Use cases:

   Instead of create a new specialized element for each API just change
   a generic element once.
   
   To add support for a new API just have to create a new backend.

   xvimagesink
   -----------
      
   - In _setcaps negotiate accept VA kind of stream when gst_va_init and
     gst_va_available tell us that we have hardware capable of it.
   
   - In _render when the buffers have the VA flag render display it with
     gst_va_render_display()
     
   videomixer, dvdspu, textoverlay, ...
   ------------------------------------

   - In _setcaps negotiate accept VA kind of stream when gst_va_init and
     gst_va_available tell us that we have hardware capable of it.
   
   - Use gst_va_buffer_associate_subpicture () where necessary.