mirrors/gstreamer
mirrors/gstreamer
1
1
Fork 0
mirror of https://gitlab.freedesktop.org/gstreamer/gstreamer.git synced 2024-12-28 19:20:35 +00:00
Code Issues 1 Projects Releases Wiki Activity
gstreamer/docs/design/draft-subtitle-overlays.txt
Tim-Philipp Müller 308d4b99ea docs: add draft for subtitle overlays to design docs 
Main purpose is to provide a generic way to make subtitles work on
top of non-raw video (vaapi, vdpau, etc.).
2011-11-02 23:35:15 +00:00

548 lines
25 KiB
Text
Raw Blame History

===============================================================
Subtitle overlays, hardware-accelerated decoding and playbin2
===============================================================
Status: EARLY DRAFT / BRAINSTORMING
The following text will use "playbin" synonymous with "playbin2".
=== 1. Background ===
Subtitles can be muxed in containers or come from an external source.
Subtitles come in many shapes and colours. Usually they are either
text-based (incl. 'pango markup'), or bitmap-based (e.g. DVD subtitles
and the most common form of DVB subs). Bitmap based subtitles are
usually compressed in some way, like some form of run-length encoding.
Subtitles are currently decoded and rendered in subtitle-format-specific
overlay elements. These elements have two sink pads (one for raw video
and one for the subtitle format in question) and one raw video source pad.
They will take care of synchronising the two input streams, and of
decoding and rendering the subtitles on top of the raw video stream.
Digression: one could theoretically have dedicated decoder/render elements
that output an AYUV or ARGB image, and then let a videomixer element do
the actual overlaying, but this is not very efficient, because it requires
us to allocate and blend whole pictures (1920x1080 AYUV = 8MB,
1280x720 AYUV = 3.6MB, 720x576 AYUV = 1.6MB) even if the overlay region
is only a small rectangle at the bottom. This wastes memory and CPU.
We could do something better by introducing a new format that only
encodes the region(s) of interest, but we don't have such a format yet, and
are not necessarily keen to rewrite this part of the logic in playbin2
at this point - and we can't change existing elements' behaviour, so would
need to introduce new elements for this.
Playbin2 supports outputting compressed formats, i.e. it does not
force decoding to a raw format, but is happy to output to a non-raw
format as long as the sink supports that as well.
In case of certain hardware-accelerated decoding APIs, we will make use
of that functionality. However, the decoder will not output a raw video
format then, but some kind of hardware/API-specific format (in the caps)
and the buffers will reference hardware/API-specific objects that
the hardware/API-specific sink will know how to handle.
=== 2. The Problem ===
In the case of such hardware-accelerated decoding, the decoder will not
output raw pixels that can easily be manipulated. Instead, it will
output hardware/API-specific objects that can later be used to render
a frame using the same API.
Even if we could transform such a buffer into raw pixels, we most
likely would want to avoid that, in order to avoid the need to
map the data back into system memory (and then later back to the GPU).
It's much better to upload the much smaller encoded data to the GPU/DSP
and then leave it there until rendered.
Currently playbin2 only supports subtitles on top of raw decoded video.
It will try to find a suitable overlay element from the plugin registry
based on the input subtitle caps and the rank. (It is assumed that we
will be able to convert any raw video format into any format required
by the overlay using a converter such as ffmpegcolorspace.)
It will not render subtitles if the video sent to the sink is not
raw YUV or RGB or if conversions have been disabled by setting the
native-video flag on playbin2.
Subtitle rendering is considered an important feature. Enabling
hardware-accelerated decoding by default should not lead to a major
feature regression in this area.
This means that we need to support subtitle rendering on top of
non-raw video.
=== 3. Possible Solutions ===
The goal is to keep knowledge of the subtitle format within the
format-specific GStreamer plugins, and knowledge of any specific
video acceleration API to the GStreamer plugins implementing
that API. We do not want to make the pango/dvbsuboverlay/dvdspu/kate
plugins link to libva/libvdpau/etc. and we do not want to make
the vaapi/vdpau plugins link to all of libpango/libkate/libass etc.
Multiple possible solutions come to mind:
(a) backend-specific overlay elements
e.g. vaapitextoverlay, vdpautextoverlay, vaapidvdspu, vdpaudvdspu,
vaapidvbsuboverlay, vdpaudvbsuboverlay, etc.
This assumes the overlay can be done directly on the backend-specific
object passed around.
The main drawback with this solution is that it leads to a lot of
code duplication and may also lead to uncertainty about distributing
certain duplicated pieces of code. The code duplication is pretty
much unavoidable, since making textoverlay, dvbsuboverlay, dvdspu,
kate, assrender, etc. available in form of base classes to derive
from is not really an option. Similarly, one would not really want
the vaapi/vdpau plugin to depend on a bunch of other libraries
such as libpango, libkate, libtiger, libass, etc.
One could add some new kind of overlay plugin feature though in
combination with a generic base class of some sort, but in order
to accommodate all the different cases and formats one would end
up with quite convoluted/tricky API.
(Of course there could also be a GstFancyVideoBuffer that provides
an abstraction for such video accelerated objects and that could
provide an API to add overlays to it in a generic way, but in the
end this is just a less generic variant of (c), and it is not clear
that there are real benefits to a specialised solution vs. a more
generic one).
(b) convert backend-specific object to raw pixels and then overlay
Even where possible technically, this is most likely very
inefficient.
(c) attach the overlay data to the backend-specific video frame buffers
in a generic way and do the actual overlaying/blitting later in
backend-specific code such as the video sink (or an accelerated
encoder/transcoder)
In this case, the actual overlay rendering (i.e. the actual text
rendering or decoding DVD/DVB data into pixels) is done in the
subtitle-format-specific GStreamer plugin. All knowledge about
the subtitle format is contained in the overlay plugin then,
and all knowledge about the video backend in the video backend
specific plugin.
The main question then is how to get the overlay pixels (and
we will only deal with pixels here) from the overlay element
to the video sink.
This could be done in multiple ways: One could send custom
events downstream with the overlay data, or one could attach
the overlay data directly to the video buffers in some way.
Sending inline events has the advantage that is is fairly
transparent to any elements between the overlay element and
the video sink: if an effects plugin creates a new video
buffer for the output, nothing special needs to be done to
maintain the subtitle overlay information, since the overlay
data is not attached to the buffer. However, it slightly
complicates things at the sink, since it would also need to
look for the new event in question instead of just processing
everything in its buffer render function.
If one attaches the overlay data to the buffer directly, any
element between overlay and video sink that creates a new
video buffer would need to be aware of the overlay data
attached to it and copy it over to the newly-created buffer.
One would have to do implement a special kind of new query
(e.g. FEATURE query) that is not passed on automatically by
gst_pad_query_default() in order to make sure that all elements
downstream will handle the attached overlay data. (This is only
a problem if we want to also attach overlay data to raw video
pixel buffers; for new non-raw types we can just make it
mandatory and assume support and be done with it; for existing
non-raw types nothing changes anyway if subtitles don't work)
(we need to maintain backwards compatibility for existing raw
video pipelines like e.g.: ..decoder ! suboverlay ! encoder..)
Even though slightly more work, attaching the overlay information
to buffers seems more intuitive than sending it interleaved as
events. And buffers stored or passed around (e.g. via the
"last-buffer" property in the sink when doing screenshots via
playbin2) always contain all the information needed.
(d) create a video/x-raw-*-delta format and use a backend-specific videomixer
This possibility was hinted at already in the digression in
section 1. It would satisfy the goal of keeping subtitle format
knowledge in the subtitle plugins and video backend knowledge
in the video backend plugin. It would also add a concept that
might be generally useful (think ximagesrc capture with xdamage).
However, it would require adding foorender variants of all the
existing overlay elements, and changing playbin2 to that new
design, which is somewhat intrusive. And given the general
nature of such a new format/API, we would need to take a lot
of care to be able to accommodate all possible use cases when
designing the API, which makes it considerably more ambitious.
Lastly, we would need to write videomixer variants for the
various accelerated video backends as well.
Overall (c) appears to be the most promising solution. It is the least
intrusive and should be fairly straight-forward to implement with
reasonable effort, requiring only small changes to existing elements
and requiring no new elements.
Doing the final overlaying in the sink as opposed to a videomixer
or overlay in the middle of the pipeline has other advantages:
- if video frames need to be dropped, e.g. for QoS reasons,
we could also skip the actual subtitle overlaying and
possibly the decoding/rendering as well, if the
implementation and API allows for that to be delayed.
- the sink often knows the actual size of the window/surface/screen
the output video is rendered to. This *may* make it possible to
render the overlay image in a higher resolution than the input
video, solving a long standing issue with pixelated subtitles on
top of low-resolution videos that are then scaled up in the sink.
This would require for the rendering to be delayed of course instead
of just attaching an AYUV/ARGB/RGBA blog of pixels to the video buffer
in the overlay, but that could all be supported.
- if the video backend / sink has support for high-quality text
rendering (clutter?) we could just pass the text or pango markup
to the sink and let it do the rest (this is unlikely to be
supported in the general case - text and glyph rendering is
hard; also, we don't really want to make up our own text markup
system, and pango markup is probably too limited for complex
karaoke stuff).
=== 4. API needed ===
(a) Representation of subtitle overlays to be rendered
We need to pass the overlay pixels from the overlay element to the
sink somehow. Whatever the exact mechanism, let's assume we pass
a refcounted GstVideoOverlayComposition struct or object.
A composition is made up of one or more overlays/rectangles.
In the simplest case an overlay rectangle is just a blob of
RGBA/ABGR [FIXME?] or AYUV pixels with positioning info and other
metadata, and there is only one rectangle to render.
We're keeping the naming generic ("OverlayFoo" rather than
"SubtitleFoo") here, since this might also be handy for
other use cases such as e.g. logo overlays or so. It is not
designed for full-fledged video stream mixing though.
// Note: don't mind the exact implementation details, they'll be hidden
// FIXME: might be confusing in 0.11 though since GstXOverlay was
// renamed to GstVideoOverlay in 0.11, but not much we can do,
// maybe we can rename GstVideoOverlay to something better
struct GstVideoOverlayComposition
{
guint num_rectangles;
GstVideoOverlayRectangle ** rectangles;
/* lowest rectangle sequence number still used by the upstream
* overlay element. This way a renderer maintaining some kind of
* rectangles <-> surface cache can know when to free cached
* surfaces/rectangles. */
guint min_seq_num_used;
/* sequence number for the composition (same series as rectangles) */
guint seq_num;
}
struct GstVideoOverlayRectangle
{
/* Position on video frame and dimension of output rectangle in
* output frame terms (already adjusted for the PAR of the output
* frame). x/y can be negative (overlay will be clipped then) */
gint x, y;
guint render_width, render_height;
/* Dimensions of overlay pixels */
guint width, height, stride;
/* This is the PAR of the overlay pixels */
guint par_n, par_d;
/* Format of pixels, GST_VIDEO_FORMAT_ARGB on big-endian systems,
* and BGRA on little-endian systems (i.e. pixels are treated as
* 32-bit values and alpha is always in the most-significant byte,
* and blue is in the least-significant byte).
*
* FIXME: does anyone actually use AYUV in practice? (we do
* in our utility function to blend on top of raw video)
* What about AYUV and endianness? Do we always have [A][Y][U][V]
* in memory? */
/* FIXME: maybe use our own enum? */
GstVideoFormat format;
/* Refcounted blob of memory, no caps or timestamps */
GstBuffer *pixels;
// FIXME: how to express source like text or pango markup?
// (just add source type enum + source buffer with data)
//
// FOR 0.10: always send pixel blobs, but attach source data in
// addition (reason: if downstream changes, we can't renegotiate
// that properly, if we just do a query of supported formats from
// the start). Sink will just ignore pixels and use pango markup
// from source data if it supports that.
//
// FOR 0.11: overlay should query formats (pango markup, pixels)
// supported by downstream and then only send that. We can
// renegotiate via the reconfigure event.
//
/* sequence number: useful for backends/renderers/sinks that want
* to maintain a cache of rectangles <-> surfaces. The value of
* the min_seq_num_used in the composition tells the renderer which
* rectangles have expired. */
guint seq_num;
/* FIXME: we also need a (private) way to cache converted/scaled
* pixel blobs */
}
(a1) Overlay consumer API:
How would this work in a video sink that supports scaling of textures:
gst_foo_sink_render () {
/* assume only one for now */
if video_buffer has composition:
composition = video_buffer.get_composition()
for each rectangle in composition:
if rectangle.source_data_type == PANGO_MARKUP
actor = text_from_pango_markup (rectangle.get_source_data())
else
pixels = rectangle.get_pixels_unscaled (FORMAT_RGBA, ...)
actor = texture_from_rgba (pixels, ...)
.. position + scale on top of video surface ...
}
(a2) Overlay producer API:
e.g. logo or subpicture overlay: got pixels, stuff into rectangle:
if (logoverlay->cached_composition == NULL) {
comp = composition_new ();
rect = rectangle_new (format, pixels_buf,
width, height, stride, par_n, par_d,
x, y, render_width, render_height);
/* composition adds its own ref for the rectangle */
composition_add_rectangle (comp, rect);
rectangle_unref (rect);
/* buffer adds its own ref for the composition */
video_buffer_attach_composition (comp);
/* we take ownership of the composition and save it for later */
logoverlay->cached_composition = comp;
} else {
video_buffer_attach_composition (logoverlay->cached_composition);
}
FIXME: also add some API to modify render position/dimensions of
a rectangle (probably requires creation of new rectangle, unless
we handle writability like with other mini objects).
(b) Fallback overlay rendering/blitting on top of raw video
Eventually we want to use this overlay mechanism not only for
hardware-accelerated video, but also for plain old raw video,
either at the sink or in the overlay element directly.
Apart from the advantages listed earlier in section 3, this
allows us to consolidate a lot of overlaying/blitting code that
is currently repeated in every single overlay element in one
location. This makes it considerably easier to support a whole
range of raw video formats out of the box, add SIMD-optimised
rendering using ORC, or handle corner cases correctly.
(Note: side-effect of overlaying raw video at the video sink is
that if e.g. a screnshotter gets the last buffer via the last-buffer
property of basesink, it would get an image without the subtitles
on top. This could probably be fixed by re-implementing the
property in GstVideoSink though. Playbin2 could handle this
internally as well).
void
gst_video_overlay_composition_blend (GstVideoOverlayComposition * comp
GstBuffer * video_buf)
{
guint n;
g_return_if_fail (gst_buffer_is_writable (video_buf));
g_return_if_fail (GST_BUFFER_CAPS (video_buf) != NULL);
... parse video_buffer caps into BlendVideoFormatInfo ...
for each rectangle in the composition: {
if (gst_video_format_is_yuv (video_buf_format)) {
overlay_format = FORMAT_AYUV;
} else if (gst_video_format_is_rgb (video_buf_format)) {
overlay_format = FORMAT_ARGB;
} else {
/* FIXME: grayscale? */
return;
}
/* this will scale and convert AYUV<->ARGB if needed */
pixels = rectangle_get_pixels_scaled (rectangle, overlay_format);
... clip output rectangle ...
__do_blend (video_buf_format, video_buf->data,
overlay_format, pixels->data,
x, y, width, height, stride);
gst_buffer_unref (pixels);
}
}
(c) Flatten all rectangles in a composition
We cannot assume that the video backend API can handle any
number of rectangle overlays, it's possible that it only
supports one single overlay, in which case we need to squash
all rectangles into one.
However, we'll just declare this a corner case for now, and
implement it only if someone actually needs it. It's easy
to add later API-wise. Might be a bit tricky if we have
rectangles with different PARs/formats (e.g. subs and a logo),
though we could probably always just use the code from (b)
with a fully transparent video buffer to create a flattened
overlay buffer.
(d) core API: new FEATURE query
For 0.10 we need to add a FEATURE query, so the overlay element
can query whether the sink downstream and all elements between
the overlay element and the sink support the new overlay API.
Elements in between need to support it because the render
positions and dimensions need to be updated if the video is
cropped or rescaled, for example.
In order to ensure that all elements support the new API,
we need to drop the query in the pad default query handler
(so it only succeeds if all elements handle it explicitly).
Might want two variants of the feature query - one where
all elements in the chain need to support it explicitly
and one where it's enough if some element downstream
supports it.
In 0.11 this could probably be handled via GstMeta and
ALLOCATION queries (and/or we could simply require
elements to be aware of this API from the start).
There appears to be no issue with downstream possibly
not being linked yet at the time when an overlay would
want to do such a query.
Other considerations:
- renderers (overlays or sinks) may be able to handle only ARGB or only AYUV
(for most graphics/hw-API it's likely ARGB of some sort, while our
blending utility functions will likely want the same colour space as
the underlying raw video format, which is usually YUV of some sort).
We need to convert where required, and should cache the conversion.
- renderers may or may not be able to scale the overlay. We need to
do the scaling internally if not (simple case: just horizontal scaling
to adjust for PAR differences; complex case: both horizontal and vertical
scaling, e.g. if subs come from a different source than the video or the
video has been rescaled or cropped between overlay element and sink).
- renderers may be able to generate (possibly scaled) pixels on demand
from the original data (e.g. a string or RLE-encoded data). We will
ignore this for now, since this functionality can still be added later
via API additions. The most interesting case would be to pass a pango
markup string, since e.g. clutter can handle that natively.
- renderers may be able to write data directly on top of the video pixels
(instead of creating an intermediary buffer with the overlay which is
then blended on top of the actual video frame), e.g. dvdspu, dvbsuboverlay
However, in the interest of simplicity, we should probably ignore the
fact that some elements can blend their overlays directly on top of the
video (decoding/uncompressing them on the fly), even more so as it's
not obvious that it's actually faster to decode the same overlay
70-90 times (say) (ie. ca. 3 seconds of video frames) and then blend
it 70-90 times instead of decoding it once into a temporary buffer
and then blending it directly from there, possibly SIMD-accelerated.
Also, this is only relevant if the video is raw video and not some
hardware-acceleration backend object.
And ultimately it is the overlay element that decides whether to do
the overlay right there and then or have the sink do it (if supported).
It could decide to keep doing the overlay itself for raw video and
only use our new API for non-raw video.
- renderers may want to make sure they only upload the overlay pixels once
per rectangle if that rectangle recurs in subsequent frames (as part of
the same composition or a different composition), as is likely. This caching
of e.g. surfaces needs to be done renderer-side and can be accomplished
based on the sequence numbers. The composition contains the lowest
sequence number still in use upstream (an overlay element may want to
cache created compositions+rectangles as well after all to re-use them
for multiple frames), based on that the renderer can expire cached
objects. The caching needs to be done renderer-side because attaching
renderer-specific objects to the rectangles won't work well given the
refcounted nature of rectangles and compositions, making it unpredictable
when a rectangle or composition will be freed or from which thread
context it will be freed. The renderer-specific objects are likely bound
to other types of renderer-specific contexts, and need to be managed
in connection with those.
- composition/rectangles should internally provide a certain degree of
thread-safety. Multiple elements (sinks, overlay element) might access
or use the same objects from multiple threads at the same time, and it
is expected that elements will keep a ref to compositions and rectangles
they push downstream for a while, e.g. until the current subtitle
composition expires.
=== 5. Future considerations ===
- alternatives: there may be multiple versions/variants of the same subtitle
stream. On DVDs, there may be a 4:3 version and a 16:9 version of the same
subtitles. We could attach both variants and let the renderer pick the best
one for the situation (currently we just use the 16:9 version). With totem,
it's ultimately totem that adds the 'black bars' at the top/bottom, so totem
also knows if it's got a 4:3 display and can/wants to fit 4:3 subs (which
may render on top of the bars) or not, for example.
=== 6. Misc. FIXMEs ===
TEST: should these look (roughly) alike (note text distortion) - needs fixing in textoverlay
gst-launch-0.10 \
videotestsrc ! video/x-raw-yuv,width=640,height=480,pixel-aspect-ratio=1/1 ! textoverlay text=Hello font-desc=72 ! xvimagesink \
videotestsrc ! video/x-raw-yuv,width=320,height=480,pixel-aspect-ratio=2/1 ! textoverlay text=Hello font-desc=72 ! xvimagesink \
videotestsrc ! video/x-raw-yuv,width=640,height=240,pixel-aspect-ratio=1/2 ! textoverlay text=Hello font-desc=72 ! xvimagesink
~~~ THE END ~~~
Reference in a new issue View git blame Copy permalink