gstreamer/subprojects/gst-docs/markdown/additional/design/dmabuf.md

# DMA buffers

This document describes the GStreamer caps negotiation of DMA buffers on
Linux-like platforms.

The DMA buffer sharing is the efficient way to share the buffer/memory
between different Linux kernel driver, such as codecs/3D/display/cameras.
For example, the decoder may want its output to be directly shared with the
display server for rendering without a copy.

Any device driver which is part of DMA buffer sharing, can do so as either
the *exporter* or *importer* of buffers.

This kind of buffer/memory is usually stored in non-system memory (maybe in
device's local memory or something else not directly accessible by the
CPU), then its memory mapping for CPU access may impose a big overhead and
low performance, or even impossible.

DMA buffers are exposed to user-space as *file descriptors* allowing to pass
them between processes.


# DRM PRIME buffers

PRIME is the cross device buffer sharing framework in DRM kernel
subsystem. These are the ones normally used in GStreamer which might
contain video frames.

PRIME buffers requires some metadata to describe how to interpret them,
such as a set of file descriptors (for example, one per plane), color
definition in fourcc, and DRM-modifiers. If the frame is going to be mapped
onto system's memory, also is needed padding, strides, offsets, etc.


## File descriptor

Each file descriptor represents a chunk of a frame, usually a plane. For
example, when a DMA buffer contains NV12 format data, it might be
composited by 2 planes: one for its Y component and the other for both UV
components. Then, the hardware may use two detached memory chunks, one per
plane, exposed as two file descriptors. Otherwise, if hardware uses only
one continuous memory chunk for all the planes, the DMA buffer should just
have one file descriptor.


## DRM fourcc

Just like fourcc common usage, DRM-fourcc describes the underlying format
of the video frame, such as `DRM_FORMAT_YVU420` or `DRM_FORMAT_NV12`. All
of them with the prefix `DRM_FORMAT_`. Please refer to `drm_fourcc.h` in
the kernel for a full list. This list of fourcc formats maps to GStreamer
video formats, although the GStreamer formats may have a slighly different.
For example, DRM_FORMAT_ARGB8888 corresponds to GST_VIDEO_FORMAT_BGRA.


## DRM modifier

DRM-modifier describes the translation mechanism between pixel to memory
samples and the actual memory storage of the buffer. The most
straightforward modifier is LINEAR, where each pixel has contiguous storage
and pixel location in memory can be easily calculated with the stride. This
is considered the baseline interchange format, and most convenient for CPU
access. Nonetheless, modern hardware employs more sophisticated memory
access mechanisms, such as tiling and possibly compression.  For example,
the TILED modifier describes memory storage where pixels are stored in 4x4
blocks arranged in row-major ordering. For example, the first tile in
memory stores pixels (0,0) to (3,3) inclusive, and the second tile in
memory stores pixels (4,0) to (7,3) inclusive, and so on.

DRM-modifier is a sixteen hexadecimal digits to represent these memory
layouts. For example, `0x0000000000000000` means linear,
`0x0100000000000001` means Intel's X tile mode, etc. Please refer to
`drm_fourcc.h` in kernel for a full list.

Excepting the linear modifier, the first 8 bits represent the vendor ID and
the other 56 bits describe the memory layout, which may be hardware
dependent. Users should be careful when interpreting non-linear memory by
themselves.

Please bear in mind that, even for the linear modifier, as the access to
DMA memory's content is through `map()` / `unmap()` functions, its
read/write performance may be low or even bad, because of its cache type
and coherence assurance. So, most of the times, it's advised to avoid that
code path for upload or download frame data.


## Meta Data

The meta data contains information about how to interpret the memory
holding the video frame, either when the frame mapped and its DRM modifier
is linear, or by other API that imports those DMA buffers.


# DMABufs in GStreamer


## Representation

In GStreamer, a full DMA buffer-based video frame is mapped to a
`GstBuffer`, and each file descriptor used to describe the whole frame is
held by a `GstMemory` mini-object. A derived class of `GstDmaBufAllocator`
would be implemented for every wrapped API *exporting* DMA buffers to
user-space, as memory allocator.


## DRM format caps field

The *GstCapsFeatures* *memory:DMABuf* is usually used to negotiate DMA
buffers. It is recommended to allow DMAbuf to flow without the
*GstCapsFeatures* *memory:DMABuf* if the DRM-modifier is linear.

But also, in order to negotiate *memory:DMABuf* thoroughly, it's required
to match the DRM-modifiers between upstream and downstream. Otherwise video
sinks might end rendering wrong frames assuming linear access.

Because DRM-fourcc and DRM-modifier are both necessary to render frames
DMABuf-backed, we now consider both as a pair and combine them together to
assure uniqueness. In caps, we use a *:* to link them together and write in
the mode of *DRM_FORMAT:DRM_MODIFIER*, which represents a totally new single video
format. For example, `NV12:0x0100000000000002` is a new video format
combined by video format NV12 and the modifier `0x0100000000000002`. It's
not NV12 and it's not its subset either.

*DRM_FORMAT* can be printed by using
`GST_FOURCC_FORMAT` and `GST_FOURCC_ARGS` macros from the
`DRM_FORMAT_*` constants, it is NOT a `GstVideoFormat`, so it would be
different from the content of the `format` field in a non-dmabuf caps.
A modifier must always be present, except if the modifier is linear,
then it should not be included, so `NV12:0x0000000000000000` is
invalid, it must be `drm-format=NV12`. DRM fourcc are used
instead of a `GstVideoFormat` to make it easier for non-GStreamer
developers to understand what the system is trying to achieve.

Please note that this form of video format only appears within
*memory:DMABuf* feature. It must not appear in any other video caps
feature.

Unlike other type of video buffers, DMABuf frames might not be mappable and
its internal format is opaque to the user. Then, unless the modifier is
linear (0x0000000000000000) or some other well known tiled format such as
NV12_4L4, NV12_16L16, NV12_64Z32, NV12_16L32S, etc. (which are defined in
video-format.h), we always use `GST_VIDEO_FORMAT_ENCODED` in
`GstVideoFormat` enum to represent its video format.

In order to not misuse this new format with the common video format, **in**
*memory:DMABuf* feature, *drm-format* field in caps will replace the
traditional *format* field.

So a DMABuf-backed video caps may look like:

```
     video/x-raw(memory:DMABuf), \
                drm-format=(string)NV12:0x0x0100000000000001, \
                width=(int)1920, \
                height=(int)1080, \
                interlace-mode=(string)progressive, \
                multiview-mode=(string)mono, \
                multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono, \
                pixel-aspect-ratio=(fraction)1/1, \
                framerate=(fraction)24/1, \
                colorimetry=(string)bt709"
```

And when we call a video info API such as `gst_video_info_from_caps()` with
this caps, it should return an video format as `GST_VIDEO_FORMAT_ENCODED`,
leaving other fields unchanged as normal video caps.

In addition, a new structure

```
struct GstDrmVideoInfo
{
  GstVideoInfo vinfo;
  guint32 drm_fourcc;
  guint64 drm_modifier;
};
```

is introduced to represent more info of DMA video caps. User should use
this DMABuf related API such as `gst_drm_video_info_from_caps()` to recognize
the video format and parse the DMA info from caps.


## Meta data

Besides the *file descriptors*, there may be a `GstVideoMeta` data attached
to each `GstBuffer` to describe more information such as the width, height,
pitches, strides and plane offsets for that DMA buffer (Please note that
the mandatory width and height information appears both in "caps" and here,
and they should be always equal). This kind of information is only obtained
by each module's API, such as the functions
`VkImageDrmFormatModifierExplicitCreateInfoEXT()` in Vulkan, and
`vaExportSurfaceHandle()` in VA-API. The information should be translated
into `GstVideoMeta`'s fields when the DMA buffer is created and
exported. These meta data is useful when other module wants to import the
DMA buffers.

For example, we may create a `GstBuffer` using `vaExportSurfaceHandle()`
VA-API, and set each field of `GstVideoMeta` with information from
`VADRMPRIMESurfaceDescriptor`. Later, a downstream Vulkan element imports
these DMA buffers with `VkImageDrmFormatModifierExplicitCreateInfoEXT()`,
translating fields form buffer's `GstVideoMeta` into the
`VkSubresourceLayout` parameter.

In short, the `GstVideoMeta` contains the common extra video information
about the DMA buffer, which can be interpreted by each module.

Information in `GstVideoMeta` depends on the hardware context and
setting. Its values, such as stride and pitch, may differ from the standard
video format because of the hardware's requirement. For example, if a DMA
buffer represents a compressed video in memory, its pitch and stride may be
smaller than the standard linear one because of the compression. Please
remind that users should not use this meta data to interpret and access the
DMA buffer, **unless the modifier is linear**.


# Negotiation of DMA buffer

If two elements of different modules (for example, VA-API decoder to
Wayland sink) want to transfer dmabufs, the negotiation should ensure a
common *drm-format* (*DRM_FORMAT:DRM_MODIFIER*).  As we already illustrate how to
represent both of them in caps before, so the negotiation here in fact has
no special operation except finding the intersection.


## Static Template Caps

If an element can list all the DRM fourcc/modifier composition at register
time, `gst-inspect` result should look like:

```
SRC template: 'src'
    Availability: Always
      Capabilities:
        video/x-raw(memory:DMABuf)
          width:  [ 16, 16384 ]
          height: [ 16, 16384 ]
          drm-format: { (string)NV12:0x0100000000000001, \
                        (string)YU12, (string)YV12, \
                        (string)YUYV:0x0100000000000002, \
                        (string)P010:0x0100000000000002, \
                        (string)AR24:0x0100000000000002, \
                        (string)AB24:0x0100000000000002, \
                        (string)AR39:0x0100000000000002, \
                        (string)AYUV:0x0100000000000002 }
```

But because sometimes it is impossible to enumerate and list all
drm_fourcc/modifier composition in static templates (for example, we may
need a runtime context which is not available at register time to detect
the real modifers a HW can support), we can let the *drm-format* field
absent to mean the super set of all formats.


## Renegotiation

Sometimes, a renegotiation may happen if the downstream element is not
pleased with the caps set by the upstream element. For example, some sink
element may not know the preferred DRM fourcc/modifier until the real
render target window is realized. Then, it will send a "reconfigure" event
to upstream element to require a renegotiation. At this round negotiation,
the downstream element will provide a more precise *drm-format* list.


## Example

Consider the pipeline of:

```
vapostproc ! video/x-raw(memory:DMABuf) ! glupload
```

both `vapostproc` and `glupload` work on the same GPU. (DMABuf caps filter
is just for illustration, it doesn't need to be specified, since DMA
negotiation is well supported.)

The VA-API based `vapostproc` element can detect the modifiers at the
element registration time and the src template should be:

```
SRC template: 'src'
    Availability: Always
      Capabilities:
        video/x-raw(memory:DMABuf)
          width:  [ 16, 16384 ]
          height: [ 16, 16384 ]
          drm-format: { (string)NV12:0x0100000000000001, \
                        (string)NV12, (string)I420, (string)YV12, \
                        (string)BGRA:0x0100000000000002 }
```

While `glupload` needs the runtime EGL context to check the DRM fourcc and
modifiers, so it can just leave the *drm-format* field absent in its sink
template:

```
SINK template: 'sink'
    Availability: Always
      Capabilities:
        video/x-raw(memory:DMABuf)
          width:  [ 1, 2147483647 ]
          height: [ 1, 2147483647 ]
```

At runtime, when the `vapostproc` wants to decide its src caps, it first
query the downstream `glupload` element about all possible DMA caps. The
`glupload` should answer that query based on the GL/EGL query result, such
as:

```
drm-format: { (string)NV12:0x0100000000000001, (string)BGRA }
```

So, the intersection with `vapostproc`'s src caps will be
`NV12:0x0100000000000001`. It will be the sent to downstream (`glupload`)
by a CAPS event. The `vapostproc` element may also query the allocation
after that CAPS event, but downstream `glupload` will not provide a DMA
buffer pool because EGL API is mostly for DMAbuf importing. Then
`vapostproc` will create its own DMA pool, the buffers created from that
new pool should conform *drm-format*, described in this document, with
`NV12:0x0100000000000001`. Also, the downstream `glupload` should make sure
that it can import other DMA buffers which are not created in the pool it
provided, as long as they conform with *drm-format*
`NV12:0x0100000000000001`.

Then, when `vapostproc` handles each frame, it creates GPU surfaces with
*drm-format* `NV12:0x0100000000000001`. Each surface is also exported as a
set of file descriptors, each one wrapped in `GstMemory` allocated by a
subclass of `GstDmaBufAllocator`. All the `GstMemory` are appended to a
`GstBuffer`. There may be some extra information about the pitch, stride
and plane offset when we export the surface, we also need to translate them
into `GstVideoMeta` and attached it to the `GstBuffer`.

Later `glupload`, when it receives a `GstBuffer`, it can use those file
descriptors with *drm-format* `NV12:0x0100000000000001` to import an
EGLImage. If the `GstVideoMeta` exists, this extra parameters should also
be provided to the importing API.