mirrors/gstreamer
mirrors/gstreamer
1
1
Fork 0
mirror of https://gitlab.freedesktop.org/gstreamer/gstreamer.git synced 2024-12-22 16:26:39 +00:00
Code Issues 1 Projects Releases Wiki Activity
gstreamer/subprojects/gstreamer-vaapi/NEWS
Tim-Philipp Müller 31b5243e1d Release 1.19.90
2022-01-28 14:28:42 +00:00

1887 lines
84 KiB
Text
Raw Blame History

This file contains invisible Unicode characters

This file contains invisible Unicode characters that are indistinguishable to humans but may be processed differently by a computer. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

GStreamer 1.20 Release Notes
GStreamer 1.20 has not been released yet. It is scheduled for release in
late January / early February 2022.
1.19.x is the unstable development version that is being developed in
the git main branch and which will eventually result in 1.20, and
1.19.90 is the first release candidate in that series (1.20rc1).
1.20 will be backwards-compatible to the stable 1.18, 1.16, 1.14, 1.12,
1.10, 1.8, 1.6, 1.4, 1.2 and 1.0 release series.
See https://gstreamer.freedesktop.org/releases/1.20/ for the latest
version of this document.
Last updated: Wednesday 26 January 2022, 01:00 UTC (log)
Introduction
The GStreamer team is proud to announce a new major feature release in
the stable 1.x API series of your favourite cross-platform multimedia
framework!
As always, this release is again packed with many new features, bug
fixes and other improvements.
Highlights
- Development in GitLab was switched to a single git repository
containing all the modules
- GstPlay: new high-level playback library, replaces GstPlayer
- WebM Alpha decoding support
- Encoding profiles can now be tweaked with additional
application-specified element properties
- Compositor: multi-threaded video conversion and mixing
- RTP header extensions: unified support in RTP depayloader and
payloader base classes
- SMPTE 2022-1 2-D Forward Error Correction support
- Smart encoding (passthrough) support for VP8, VP9, H.265 in
encodebin and transcodebin
- Runtime compatibility support for libsoup2 and libsoup3 (libsoup3
support experimental)
- Video decoder subframe support
- Video decoder automatic packet-loss, data corruption, and keyframe
request handling for RTP / WebRTC / RTSP
- MP4 and Matroska muxers now support profile/level/resolution changes
for H264/H265 input streams (i.e. codec data changing on the fly)
- MP4 muxing mode that initially creates a fragmented mp4 which is
converted to a regular mp4 on EOS
- Audio support for the WebKit Port for Embedded (WPE) web page source
element
- CUDA based video color space convert and rescale elements and
upload/download elements
- NVIDIA memory:NVMM support for OpenGL glupload and gldownload
elements
- Many WebRTC improvements
- The new VA-API plugin implemention fleshed out with more decoders
and new postproc elements
- AppSink API to retrieve events in addition to buffers and buffer
lists
- AppSrc gained more configuration options for the internal queue
(leakiness, limits in buffers and time, getters to read current
levels)
- Updated Rust bindings and many new Rust plugins
- Improved support for custom minimal GStreamer builds
- Support build against FFmpeg 5.0
- Linux Stateless CODEC support gained MPEG2 and VP9
- Windows Direct3D11/DXVA decoder gained AV1 and MPEG2 support
- Lots of new plugins, features, performance improvements and bug
fixes
Major new features and changes
Noteworthy new features and API
- gst_element_get_request_pad() has been deprecated in favour of the
newly-added gst_element_request_pad_simple() which does the exact
same thing but has a less confusing name that hopefully makes clear
that the function request a new pad rather than just retrieves an
already-existing request pad.
Development in GitLab was switched to a single git repository containing all the modules
The GStreamer multimedia framework is a set of libraries and plugins
split into a number of distinct modules which are released independently
and which have so far been developed in separate git repositories in
freedesktop.org GitLab.
In addition to these separate git repositories there was a gst-build
module that would use the Meson build systems’s subproject feature to
download each individual module and then build everything in one go. It
would also provide an uninstalled development environment that made it
easy to work on GStreamer and use or test versions other than the
system-installed GStreamer version.
All of these modules have now (as of 28 September 2021) been merged into
a single git repository (“Mono repository” or “monorepo”) which should
simplify development workflows and continuous integration, especially
where changes need to be made to multiple modules at once.
This mono repository merge will primarily affect GStreamer developers
and contributors and anyone who has workflows based on the GStreamer git
repositories.
The Rust bindings and Rust plugins modules have not been merged into the
mono repository at this time because they follow a different release
cycle.
The mono repository lives in the existing GStreamer core git repository
in GitLab in the new main branch and all future development will happen
on this branch.
Modules will continue to be released as separate tarballs.
For more details, please see the GStreamer mono repository FAQ.
GstPlay: new high-level playback library replacing GstPlayer
- GstPlay is a new high-level playback library that replaces the older
GstPlayer API. It is basically the same API as GstPlayer but
refactored to use bus messages for application notifications instead
of GObject signals. There is still a signal adapter object for those
who prefer signals. Since the existing GstPlayer API is already in
use in various applications, it didn’t seem like a good idea to
break it entirely. Instead a new API was added, and it is expected
that this new GstPlay API will be moved to gst-plugins-base in
future.
- The existing GstPlayer API is scheduled for deprecation and will be
removed at some point in the future (e.g. in GStreamer 1.24), so
application developers are urged to migrate to the new GstPlay API
at their earliest convenience.
WebM alpha decoding
- Implement WebM alpha decoding (VP8/VP9 with alpha), which required
support and additions in various places. This is supported both with
software decoders and hardware-accelerated decoders.
- VP8/VP9 don’t support alpha components natively in the codec, so the
way this is implemented in WebM is by encoding the alpha plane with
transparency data as a separate VP8/VP9 stream. Inside the WebM
container (a variant of Matroska) this is coded as a single video
track with the “normal” VP8/VP9 video data making up the main video
data and each frame of video having an encoded alpha frame attached
to it as extra data ("BlockAdditional").
- matroskademux has been extended extract this per-frame alpha side
data and attach it in form of a GstVideoCodecAlphaMeta to the
regular video buffers. Note that this new meta is specific to this
VP8/VP9 alpha support and can’t be used to just add alpha support to
other codecs that don’t support it. Lastly, matroskademux also
advertises the fact that the streams contain alpha in the caps.
- The new codecalpha plugin contains various bits of infrastructure to
support autoplugging and debugging:
- codecalphademux splits out the alpha stream from the metas on
the regular VP8/VP9 buffers
- alphacombine takes two decoded raw video streams (one alpha, one
the regular video) and combines it into a video stream with
alpha
- vp8alphadecodebin + vp9alphadecodebin are wrapper bins that use
the regular vp8dec and vp9dec software decoders to decode
regular and alpha streams and combine them again. To decodebin
these look like regular decoders which ju
- The V4L2 CODEC plugin has stateless VP8/VP9 decoders that can
decode both alpha and non-alpha stream with a single decoder
instance
- A new AV12 video format was added which is basically NV12 with an
alpha plane, which is more convenient for many hardware-accelerated
decoders.
- Watch Nicolas Dufresne’s LCA 2022 talk “Bringing WebM Alpha support
to GStreamer” for all the details and a demo.
RTP Header Extensions Base Class and Automatic Header Extension Handling in RTP Payloaders and Depayloaders
- RTP Header Extensions are specified in RFC 5285 and provide a way to
add small pieces of data to RTP packets in between the RTP header
and the RTP payload. This is often used for per-frame metadata,
extended timestamps or other application-specific extra data. There
are several commonly-used extensions specified in various RFCs, but
senders are free to put any kind of data in there, as long as sender
and receiver both know what that data is. Receivers that don’t know
about the header extensions will just skip the extra data without
ever looking at it. These header extensions can often be combined
with any kind of payload format, so may need to be supported by many
RTP payloader and depayloader elements.
- Inserting and extracting RTP header extension data has so far been a
bit inconvenient in GStreamer: There are functions to add and
retrieve RTP header extension data from RTP packets, but nothing
works automatically, even for common extensions. People would have
to do the insertion/extraction either in custom elements
before/after the RTP payloader/depayloader, or inside pad probes,
which isn’t very nice.
- This release adds various pieces of new infrastructure for generic
RTP header extension handling, as well as some implementations for
common extensions:
- GstRTPHeaderExtension is a new helper base class for reading and
writing RTP header extensions. Nominally this subclasses
GstElement, but only so these extensions are stored in the
registry where they can be looked up by URI or name. They don’t
have pads and don’t get added to the pipeline graph as an
element.
- "add-extension" and "clear-extension" action signals on RTP
payloaders and depayloaders for manual extension management
- The "request-extension" signal will be emitted if an extension
is encountered that requires explicit mapping by the application
- new "auto-header-extension" property on RTP payloaders and
depayloaders for automatic handling of known header extensions.
This is enabled by default. The extensions must be signalled via
caps / SDP.
- RTP header extension implementations:
- rtphdrextclientaudiolevel: Client-to-Mixer Audio Level
Indication (RFC 6464) (also see below)
- rtphdrextcolorspace: Color Space extension, extends RTP
packets with color space and high dynamic range (HDR)
information
- rtphdrexttwcc: Transport Wide Congestion Control support
- gst_rtp_buffer_remove_extension_data() is a new helper function to
remove an RTP header extension from an RTP buffer
- The existing gst_rtp_buffer_set_extension_data() now also supports
shrinking the extension data in size
AppSink and AppSrc improvements
- appsink: new API to pull events out of appsink in addition to
buffers and buffer lists.
There was previously no way for users to receive incoming events
from appsink properly serialised with the data flow, even if they
are serialised events. The reason for that is that the only way to
intercept events was via a pad probe on the appsink sink pad, but
there is also internal queuing inside of appsink, so it’s difficult
to ascertain the right order of everything in all cases.
There is now a new "new-serialized-event" signal which will be
emitted when there’s a new event pending (just like the existing
"new-sample" signal). The "emit-signals" property must be set to
TRUE in order to activate this (but it’s also fine to just pull from
the application thread without using the signals).
gst_app_sink_pull_object() and gst_app_sink_try_pull_object() can be
used to pull out either an event or a new sample carrying a buffer
or buffer list, whatever is next in the queue.
EOS events will be filtered and will not be returned. EOS handling
can be done the ususal way, same as with _pull_sample().
- appsrc: allow configuration of internal queue limits in time and
buffers and add leaky mode.
There is internal queuing inside appsrc so the application thread
can push data into the element which will then be picked up by the
source element’s streaming thread and pushed into the pipeline from
that streaming thread. This queue is unlimited by default and until
now it was only possible to set a maximum size limit in bytes. When
that byte limit is reached, the pushing thread (application thread)
would be blocked until more space becomes available.
A limit in bytes is not particularly useful for many use cases, so
now it is possible to also configure limits in time and buffers
using the new "max-time" and "max-buffers" properties. Of course
there are also matching new read-only"current-level-buffers" and
"current-level-time properties" properties to query the current fill
level of the internal queue in time and buffers.
And as if that wasn’t enough the internal queue can also be
configured as leaky using the new "leaky-type" property. That way
when the queue is full the application thread won’t be blocked when
it tries to push in more data, but instead either the new buffer
will be dropped or the oldest data in the queue will be dropped.
Better string serialization of nested GstCaps and GstStructures
- New string serialisation format for structs and caps that can handle
nested structs and caps properly by using brackets to delimit nested
items (e.g. some-struct, some-field=[nested-struct, nested=true]).
Unlike the default format the new variant can also support more than
one level of nesting. For backwards-compatibility reasons the old
format is still output by default when serialising caps and structs
using the existing API. The new functions gst_caps_serialize() and
gst_structure_serialize() can be used to output strings in the new
format.
Convenience API for custom GstMetas
- New convenience API to register and create custom GstMetas:
gst_meta_register_custom() and gst_buffer_add_custom_meta(). Such
custom meta is backed by a GstStructure and does not require that
users of the API expose their GstMeta implementation as public API
for other components to make use of it. In addition, it provides a
simpler interface by ignoring the impl vs. api distinction that the
regular API exposes. This new API is meant to be the meta
counterpart to custom events and messages, and to be more convenient
than the lower-level API when the absolute best performance isn’t a
requirement. The reason it’s less performant than a “proper” meta is
that a proper meta is just a C struct in the end whereas this goes
through the GstStructure API which has a bit more overhead, which
for most scenarios is negligible however. This new API is useful for
experimentation or proprietary metas, but also has some limitations:
it can only be used if there’s a single producer of these metas;
it’s not allowed to register the same custom meta multiple times or
from multiple places.
Additional Element Properties on Encoding Profiles
- GstEncodingProfile: The new "element-properties" and
gst_encoding_profile_set_element_properties() API allows
applications to set additional element properties on encoding
profiles to configure muxers and encoders. So far the encoding
profile template was the only place where this could be specified,
but often what applications want to do is take a ready-made encoding
profile shipped by GStreamer or the application and then tweak the
settings on top of that, which is now possible with this API. Since
applications can’t always know in advance what encoder element will
be used in the end, it’s even possible to specify properties on a
per-element basis.
Encoding Profiles are used in the encodebin, transcodebin and
camerabin elements and APIs to configure output formats (containers
and elementary streams).
Audio Level Indication Meta for RFC 6464
- New GstAudioLevelMeta containing Audio Level Indication as per RFC
6464
- The level element has been updated to add GstAudioLevelMeta on
buffers if the "audio-level-meta" property is set to TRUE. This can
then in turn be picked up by RTP payloaders to signal the audio
level to receivers through RTP header extensions (see above).
- New Client-to-Mixer Audio Level Indication (RFC6464) RTP Header
Extension which should be automatically created and used by RTP
payloaders and depayloaders if their "auto-header-extension"
property is enabled and if the extension is part of the RTP caps.
Automatic packet loss, data corruption and keyframe request handling for video decoders
- The GstVideoDecoder base class has gained various new APIs to
automatically handle packet loss and data corruption better by
default, especially in RTP, RTSP and WebRTC streaming scenarios, and
to give subclasses more control about how they want to handle
missing data:
- Video decoder subclasses can mark output frames as corrupted via
the new GST_VIDEO_CODEC_FRAME_FLAG_CORRUPTED flag
- A new "discard-corrupted-frames" property allows applications to
configure decoders so that corrupted frames are directly
discarded instead of being forwarded inside the pipeline. This
is a replacement for the "output-corrupt" property of the FFmpeg
decoders.
- RTP depayloaders can now signal to decoders that data is missing
when sending GAP events for lost packets. GAP events can be sent
for various reason in a GStreamer pipeline. Often they are just
used to let downstream elements know that there isn’t a buffer
available at the moment, so downstream elements can move on
instead of waiting for one. They are also sent by RTP
depayloaders in the case that packets are missing, however, and
so far a decoder was not able to differentiate the two cases.
This has been remedied now: GAP events can be decorated with
gst_event_set_gap_flags() and GST_GAP_FLAG_MISSING_DATA to let
decoders now what happened, and decoders can then use that in
some cases to handle missing data better.
- The GstVideoDecoder::handle_missing_data vfunc was added to
inform subclasses about packet loss or missing data and let them
handle it in their own way if they like.
- gst_video_decoder_set_needs_sync_point() lets subclasses signal
that they need the stream to start with a sync point. If
enabled, the base class will discard all non-sync point frames
in the beginning and after a flush and does not pass them to the
subclass. Furthermore, if the first frame is not a sync point,
the base class will try and request a sync frame from upstream
by sending a force-key-unit event (see next items).
- New "automatic-request-sync-points" and
"automatic-request-sync-point-flags" properties to automatically
request sync points when needed, e.g. on packet loss or if the
first frame is not a keyframe. Applications may want to enable
this on decoders operating in e.g. RTP/WebRTC/RTSP receiver
pipelines.
- The new "min-force-key-unit-interval" property can be used to
ensure there’s a minimal interval between keyframe requests to
upstream (and/or the sender) and we’re not flooding the sender
with key unit requests.
- gst_video_decoder_request_sync_point() allows subclasses to
request a new sync point (e.g. if they choose to do their own
missing data handling). This will still honour the
"min-force-key-unit-interval" property if set.
Improved support for custom minimal GStreamer builds
- Element registration and registration of other plugin features
inside plugin init functions has been improved in order to
facilitate minimal custom GStreamer builds.
- A number of new macros have been added to declare and create
per-element and per-pluginfeature register functions in all plugins,
and then call those from the per-plugin plugin_init functions:
- GST_ELEMENT_REGISTER_DEFINE,
GST_DEVICE_PROVIDER_REGISTER_DEFINE,
GST_DYNAMIC_TYPE_REGISTER_DEFINE, GST_TYPE_FIND_REGISTER_DEFINE
for the actual registration call with GStreamer
- GST_ELEMENT_REGISTER, GST_DEVICE_PROVIDER_REGISTER,
GST_DYNAMIC_TYPE_REGISTER, GST_PLUGIN_STATIC_REGISTER,
GST_TYPE_FIND_REGISTER to call the registration function defined
by the REGISTER_DEFINE macro
- GST_ELEMENT_REGISTER_DECLARE,
GST_DEVICE_PROVIDER_REGISTER_DECLARE,
GST_DYNAMIC_TYPE_REGISTER_DECLARE,
GST_TYPE_FIND_REGISTER_DECLARE to declare the registration
function defined by the REGISTER_DEFINE macro
- and various variants for advanced use cases.
- This means that applications can call the per-element and
per-pluginfeature registration functions for only the elements they
need instead of registering plugins as a whole with all kinds of
elements that may not be required (e.g. encoder and decoder instead
of just decoder). In case of static linking all unused functions and
their dependencies would be removed in this case by the linker,
which helps minimise binary size for custom builds.
- gst_init() will automatically call a gst_init_static_plugins()
function if one exists.
- See the GStreamer static build documentation and Stéphane’s blog
post Generate a minimal GStreamer build, tailored to your needs for
more details.
New elements
- New aesdec and aesenc elements for AES encryption and decryption in
a custom format.
- New encodebin2 element with dynamic/sometimes source pads in order
to support the option of doing the muxing outside of encodebin,
e.g. in combination with a splitmuxsink.
- New fakeaudiosink and videocodectestsink elements for testing and
debugging (see below for more details)
- rtpisacpay, rtpisacdepay: new RTP payloader and depayloader for iSAC
audio codec
- rtpst2022-1-fecdec, rtpst2022-1-fecenc: new elements providing SMPTE
2022-1 2-D Forward Error Correction. More details in Mathieu’s blog
post.
- isac: new plugin wrapping the Internet Speech Audio Codec reference
encoder and decoder from the WebRTC project.
- asio: plugin for Steinberg ASIO (Audio Streaming Input/Output) API
- gssrc, gssink: add source and sink for Google Cloud Storage
- onnx: new plugin to apply ONNX neural network models to video
- openaptx: aptX and aptX-HD codecs using libopenaptx (v0.2.0)
- qroverlay, debugqroverlay: new elements that allows overlaying data
on top of video in form of a QR code
- cvtracker: new OpenCV-based tracker element
- av1parse, vp9parse: new parsers for AV1 and VP9 video
- va: work on the new VA-API plugin implementation for
hardware-accelerated video decoding and encoding has continued at
pace, with various new decoders and filters having joined the
initial vah264dec:
- vah265dec: VA-API H.265 decoder
- vavp8dec: VA-API VP8 decoder
- vavp9dec: VA-API VP9 decoder
- vaav1dec: VA-API AV1 decoder
- vampeg2dec: VA-API MPEG-2 decoder
- vadeinterlace: : VA-API deinterlace filter
- vapostproc: : VA-API postproc filter (color conversion,
resizing, cropping, color balance, video rotation, skin tone
enhancement, denoise, sharpen)
See Víctor’s blog post “GstVA in GStreamer 1.20” for more details
and what’s coming up next.
- vaapiav1dec: new AV1 decoder element (in gstreamer-vaapi)
- msdkav1dec: hardware-accelerated AV1 decoder using the Intel Media
SDK / oneVPL
- nvcodec plugin for NVIDIA NVCODEC API for hardware-accelerated video
encoding and decoding:
- cudaconvert, cudascale: new CUDA based video color space convert
and rescale elements
- cudaupload, cudadownload: new helper elements for memory
transfer between CUDA and system memory spaces
- nvvp8sldec, nvvp9sldec: new GstCodecs-based VP8/VP9 decoders
- Various new hardware-accelerated elements for Windows:
- d3d11screencapturesrc: new desktop capture element, including a
GstDeviceProvider implementation to enumerate/select target
monitors for capture.
- d3d11av1dec and d3d11mpeg2dec: AV1 and MPEG-2 decoders
- d3d11deinterlace: deinterlacing filter
- d3d11compositor: video composing element
- see Windows section below for more details
- new Rust plugins:
- audiornnoise: Removes noise from an audio stream
- awstranscribeparse: Parses AWS audio transcripts into timed text
buffers
- ccdetect: Detects if valid closed captions are present in a
closed captions stream
- cea608tojson: Converts CEA-608 Closed Captions to a JSON
representation
- cmafmux: CMAF fragmented MP4 muxer
- dashmp4mux: DASH fragmented MP4 muxer
- isofmp4mux: ISO fragmented MP4 muxer
- ebur128level: EBU R128 Loudness Level Measurement
- ffv1dec: FFV1 video decoder
- gtk4paintablesink: GTK4 video sink, which provides a
GdkPaintable that can be rendered in various widgets
- hlssink3: HTTP Live Streaming sink
- hrtfrender: Head-Related Transfer Function (HRTF) renderer
- hsvdetector: HSV colorspace detector
- hsvfilter: HSV colorspace filter
- jsongstenc: Wraps buffers containing any valid top-level JSON
structures into higher level JSON objects, and outputs those as
ndjson
- jsongstparse: Parses ndjson as output by jsongstenc
- jsontovtt: converts JSON to WebVTT subtitles
- regex: Applies regular expression operations on text
- roundedcorners: Adds rounded corners to video
- spotifyaudiosrc: Spotify source
- textahead: Display upcoming text buffers ahead (e.g. for
Karaoke)
- transcriberbin: passthrough bin that transcribes raw audio to
closed captions using awstranscriber and puts the captions as
metas onto the video
- tttojson: Converts timed text to a JSON representation
- uriplaylistbin: Playlist source bin
- webpdec-rs: WebP image decoder with animation support
- New plugin codecalpha with elements to assist with WebM Alpha
decoding
- codecalphademux: Split stream with GstVideoCodecAlphaMeta into
two streams
- alphacombine: Combine two raw video stream (I420 or NV12) as one
stream with alpha channel (A420 or AV12)
- vp8alphadecodebin: A bin to handle software decoding of VP8 with
alpha
- vp9alphadecodebin: A bin to handle software decoding of VP9 with
alpha
- New hardware accelerated elements for Linux:
- v4l2slmpeg2dec: Support for Linux Stateless MPEG2 decoders
- v4l2slvp9dec: Support for Linux Stateless VP9 decoders
- v4l2slvp8alphadecodebin: Support HW accelerated VP8 with alpha
layer decoding
- v4l2slvp9alphadecodebin: Support HW accelerated VP9 with alpha
layer decoding
New element features and additions
- assrender: handle more font mime types; better interaction with
matroskademux for embedded fonts
- audiobuffersplit: Add support for specifying output buffer size in
bytes (not just duration)
- audiolatency: new "samplesperbuffer" property so users can configure
the number of samples per buffer. The default value is 240 samples
which is equivalent to 5ms latency with a sample rate of 48000,
which might be larger than actual buffer size of audio capture
device.
- audiomixer, audiointerleave, GstAudioAggregator: now keep a count of
samples that are dropped or processed as statistic and can be made
to post QoS messages on the bus whenever samples are dropped by
setting the "qos-messages" property on input pads.
- audiomixer, compositor: improved handling of new inputs added at
runtime. New API was added to the GstAggregator base class to allow
subclasses to opt into an aggregation mode where inactive pads are
ignored when processing input buffers
(gst_aggregator_set_ignore_inactive_pads(),
gst_aggregator_pad_is_inactive()). An “inactive pad” in this context
is a pad which, in live mode, hasn’t yet received a first buffer,
but has been waited on at least once. What would happen usually in
this case is that the aggregator would wait for data on this pad
every time, up to the maximum configured latency. This would
inadvertently push mixer elements in live mode to the configured
latency envelope and delay processing when new inputs are added at
runtime until these inputs have actually produced data. This is
usually undesirable. With this new API, new inputs can be added
(requested) and configured and they won’t delay the data processing.
Applications can opt into this new behaviour by setting the
"ignore-inactive-pads" property on compositor, audiomixer or other
GstAudioAggregator-based elements.
- cccombiner: implement “scheduling” of captions. So far cccombiner’s
behaviour was essentially that of a funnel: it strictly looked at
input timestamps to associate together video and caption buffers.
Now it will try to smoothly schedule caption buffers in order to
have exactly one per output video buffer. This might involve
rewriting input captions, for example when the input is CDP then
sequence counters are rewritten, time codes are dropped and
potentially re-injected if the input video frame had a time code
meta. This can also lead to the input drifting from synchronization,
when there isn’t enough padding in the input stream to catch up. In
that case the element will start dropping old caption buffers once
the number of buffers in its internal queue reaches a certain limit
(configurable via the "max-scheduled" property). The new original
funnel-like behaviour can be restored by setting the "scheduling"
property to FALSE.
- ccconverter: new "cdp-mode" property to specify which sections to
include in CDP packets (timecode, CC data, service info). Various
software, including ffmpeg’s Decklink support, fails parsing CDP
packets that contain anything but CC data in the CDP packets.
- clocksync: new "sync-to-first" property for automatic timestamp
offset setup: if set clocksync will set up the "ts-offset" value
based on the first buffer and the pipeline’s running time when the
first buffer arrived. The newly configured "ts-offset" in this case
would be the value that allows outputting the first buffer without
waiting on the clock. This is useful for example to feed a non-live
input into an already-running pipeline.
- compositor:
- multi-threaded input conversion and compositing. Set the
"max-threads" property to activate this.
- new "sizing-policy" property to support display aspect ratio
(DAR)-aware scaling. By default the image is scaled to fill the
configured destination rectangle without padding and without
keeping the aspect ratio. With sizing-policy=keep-aspect-ratio
the input image is scaled to fit the destination rectangle
specified by GstCompositorPad:{xpos, ypos, width, height}
properties preserving the aspect ratio. As a result, the image
will be centered in the destination rectangle with padding if
necessary.
- new "zero-size-is-unscaled" property on input pads. By default
pad width=0 or pad height=0 mean that the stream should not be
scaled in that dimension. But if the "zero-size-is-unscaled"
property is set to FALSE a width or height of 0 is instead
interpreted to mean that the input image on that pad should not
be composited, which is useful when creating animations where an
input image is made smaller and smaller until it disappears.
- improved handling of new inputs at runtime via
"ignore-inactive-pads"property (see above for details)
- allow output format with alpha even if none of the inputs have
alpha (also glvideomixer and other GstVideoAggregator
subclasses)
- dashsink: add h265 codec support and signals for allowing custom
playlist/fragment output
- decodebin3:
- improved decoder selection, especially for hardware decoders
- make input activation “atomic” when adding inputs dynamically
- better interleave handling: take into account decoder latency
for interleave size
- decklink:
- Updated DeckLink SDK to 11.2 to support DeckLink 8K Pro
- decklinkvideosrc:
- More accurate and stable capture timestamps: use the
hardware reference clock time when the frame was finished
being captured instead of a clock time much further down the
road.
- Automatically detect widescreen vs. normal NTSC/PAL
- encodebin:
- add “smart encoding” support for H.265, VP8 and VP9 (i.e. only
re-encode where needed and otherwise pass through encoded video
as-is).
- H264/H265 smart encoding improvements: respect user-specified
stream-format, but if not specified default to avc3/hvc1 with
in-band SPS/PPS/VPS signalling for more flexibility.
- new encodebin2 element with dynamic/sometimes source pads in
order to support the option of doing the muxing outside of
encodebin, e.g. in combination with splitmuxsink.
- add APIs to set element properties on encoding profiles (see
below)
- errorignore: new "ignore-eos" property to also ignore FLOW_EOS from
downstream elements
- giosrc: add support for growing source files: applications can
specify that the underlying file being read is growing by setting
the "is-growing" property. If set, the source won’t EOS when it
reaches the end of the file, but will instead start monitoring it
and will start reading data again whenever a change is detected. The
new "waiting-data" and "done-waiting-data" signals keep the
application informed about the current state.
- gtksink, gtkglsink:
- scroll event support: forwarded as navigation events into the
pipeline
- "video-aspect-ratio-override" property to force a specific
aspect ratio
- "rotate-method" property and support automatic rotation based on
image tags
- identity: new "stats" property allows applications to retrieve the
number of bytes and buffers that have passed through so far.
- interlace: add support for more formats, esp 10-bit, 12-bit and
16-bit ones
- jack: new "low-latency" property for automatic latency-optimized
setting and "port-names" property to select ports explicitly
- jpegdec: support output conversion to RGB using libjpeg-turbo (for
certain input files)
- line21dec:
- "mode" property to control whether and how detected closed
captions should be inserted in the list of existing close
caption metas on the input frame (if any): add, drop, or
replace.
- "ntsc-only" property to only look for captions if video has NTSC
resolution
- line21enc: new "remove-caption-meta" to remove metas from output
buffers after encoding the captions into the video data; support for
CDP closed captions
- matroskademux, matroskamux: Add support for ffv1, a lossless
intra-frame video coding format.
- matroskamux: accept in-band SPS/PPS/VPS for H264 and H265
(i.e. stream-format avc3 and hev1) which allows on-the-fly
profile/level/resolution changes.
- matroskamux: new "cluster-timestamp-offset" property, useful for use
cases where the container timestamps should map to some absolute
wall clock time, for example.
- rtpsrc: add "caps" property to allow explicit setting of the caps
where needed
- mpegts: support SCTE-35 passthrough via new "send-scte35-events"
property on MPEG-TS demuxer tsdemux. When enabled, SCTE 35 sections
(eg ad placement opportunities) are forwarded as events donwstream
where they can be picked up again by mpegtsmux. This required a
semantic change in the SCTE-35 section API: timestamps are now in
running time instead of muxer pts.
- tsdemux: Handle PCR-less MPEG-TS streams; more robust timestamp
handling in certain corner cases and for poorly muxed streams.
- mpegtsmux:
- More conformance improvements to make MPEG-TS analyzers happy:
- PCR timing accuracy: Improvements to the way mpegtsmux
outputs PCR observations in CBR mode, so that a PCR
observation is always inserted when needed, so that we never
miss the configured pcr-interval, as that triggers various
MPEG-TS analyser errors.
- Improved PCR/SI scheduling
- Don’t write PCR until PAT/PMT are output to make sure streams
start cleanly with a PAT/PMT.
- Allow overriding the automatic PMT PID selection via
application-supplied PMT_%d fields in the prog-map
structure/property.
- mp4mux:
- new "first-moov-then-finalise" mode for fragmented output where
the output will start with a self-contained moov atom for the
first fragment, and then produce regular fragments. Then at the
end when the file is finalised, the initial moov is invalidated
and a new moov is written covering the entire file. This way the
file is a “fragmented mp4” file while it is still being written
out, and remains playable at all times, but at the end it is
turned into a regular mp4 file (with former fragment headers
remaining as unused junk data in the file).
- support H.264 avc3 and H.265 hvc1 stream formats as input where
the codec data is signalled in-band inside the bitstream instead
of caps/file headers.
- support profile/level/resolution changes for H264/H265 input
streams (i.e. codec data changing on the fly). Each codec_data
is put into its own SampleTableEntry inside the stsd, unless the
input is in avc3 stream format in which case it’s written
in-band an not in the headers.
- multifilesink: new ""min-keyframe-distance"" property to make
minimum distance between keyframes in next-file=key-frame mode
configurable instead of hard-coding it to 10 seconds.
- mxfdemux has seen a big refactoring to support non-frame wrappings
and more accurate timestamp/seek handling for some formats
- msdk plugin for hardware-accelerated video encoding and decoding
using the Intel Media SDK:
- oneVPL support (Intel oneAPI Video Processing Library)
- AV1 decoding support
- H.264 decoder now supports constrained-high and progressive-high
profiles
- H.264 encoder:
- more configuration options (properties):
"intra-refresh-type", "min-qp" , "max-qp", "p-pyramid",
"dblk-idc"
- H.265 encoder:
- can output main-still-picture profile
- now inserts HDR SEIs (mastering display colour volume and
content light level)
- more configuration options (properties):
"intra-refresh-type", "min-qp" , "max-qp", "p-pyramid",
"b-pyramid", "dblk-idc", "transform-skip"
- support for RGB 10bit format
- External bitrate control in encoders
- Video post proc element msdkvpp gained support for 12-bit pixel
formats P012_LE, Y212_LE and Y412_LE
- nvh264sldec: interlaced stream support
- openh264enc: support main, high, constrained-high and
progressive-high profiles
- openjpeg: support for multithreaded decoding and encoding
- rtspsrc: now supports IPv6 also for tunneled mode (RTSP-over-HTTP);
new "ignore-x-server-reply" property to ignore the
x-server-ip-address server header reply in case of HTTP tunneling,
as it is often broken.
- souphttpsrc: Runtime compatibility support for libsoup2 and
libsoup3. libsoup3 is the latest major version of libsoup, but
libsoup2 and libsoup3 can’t co-exist in the same process because
there is no namespacing or versioning for GObject types. As a
result, it would be awkward if the GStreamer souphttpsrc plugin
linked to a specific version of libsoup, because it would only work
with applications that use the same version of libsoup. To make this
work, the soup plugin now tries to determine the libsoup version
used by the application (and its other dependencies) at runtime on
systems where GStreamer is linked dynamically. libsoup3 support is
still considered somewhat experimental at this point.
- srtsrc, srtsink: add signals for the application to accept/reject
incoming connections
- timeoverlay: new elapsed-running-time time mode which shows the
running time since the first running time (and each flush-stop).
- udpsrc: new timestamping mode to retrieve packet receive timestamps
from the kernel via socket control messages (SO_TIMESTAMPNS) on
supported platforms
- uritranscodebin: new setup-source and element-setup signals for
applications to configure elements used
- v4l2codecs plugin gained support for 4x4 and 32x32 tile formats
enabling some platforms or direct renders. Important memory usage
improvement.
- v4l2slh264dec now implements the final Linux uAPI as shipped on
Linux 5.11 and later.
- valve: add "drop-mode" property and provide two new modes of
operation: in drop-mode=forward-sticky-events sticky events
(stream-start, segment, tags, caps, etc.) are forwarded downstream
even when dropping is enabled; drop-mode=transform-to-gap will in
addition also convert buffers into gap events when dropping is
enabled, which lets downstream elements know that time is advancing
and might allow for preroll in many scenarios. By default all events
and all buffers are dropped when dropping is enabled, which can
cause problems with caps negotiation not progressing or branches not
prerolling when dropping is enabled.
- videocrop: support for many more pixel formats, e.g. planar YUV
formats with > 8bits and GBR* video formats; can now also accept
video not backed by system memory as long as downstream supports the
GstCropMeta
- videotestsrc: new smpte-rp-219 pattern for SMPTE75 RP-219 conformant
color bars
- vp8enc: finish support for temporal scalability: two new properties
("temporal-scalability-layer-flags",
"temporal-scalability-layer-sync-flags") and a unit change on the
"temporal-scalability-target-bitrate" property (now expects bps);
also make temporal scalability details available to RTP payloaders
as buffer metadata.
- vp9enc: new properties to tweak encoder performance:
- "aq-mode" to configure adaptive quantization modes
- "frame-parallel-decoding" to configure whether to create a
bitstream that reduces decoding dependencies between frames
which allows staged parallel processing of more than one video
frames in the decoder. (Defaults to TRUE)
- "row-mt", "tile-columns" and "tile-rows" so multithreading can
be enabled on a per-tile basis, instead of on a per tile-column
basis. In combination with the new "tile-rows" property, this
allows the encoder to make much better use of the available CPU
power.
- vp9dec, vp9enc: add support for 10-bit 4:2:0 and 4:2:2 YUV, as well
as 8-bit 4:4:4
- vp8enc, vp9enc now default to “good quality” for the deadline
property rather then “best quality”. Having the deadline set to best
quality causes the encoder to be absurdly slow, most real-life users
will prefer good-enough quality with better performance instead.
- wpesrc:
- implement audio support: a new sometimes source pad will be
created for each audio stream created by the web engine.
- move wpesrc to wpevideosrc and add a wrapper bin wpesrc to also
support audio
- also handles web:// URIs now (same as cefsrc)
- post messages with the estimated load progress on the bus
- x265enc: add negative DTS support, which means timestamps are now
offset by 1h same as with x264enc
RTP Payloaders and Depayloaders
- rtpisacpay, rtpisacdepay: new RTP payloader and depayloader for iSAC
audio codec
- rtph264depay:
- new "request-keyframe" property to make the depayloader
automatically request a new keyframe from the sender on packet
loss, consistent with the new property on rtpvp8depay.
- new "wait-for-keyframe" property to make depayloader wait for a
new keyframe at the beginning and after packet loss (only
effective if the depayloader outputs AUs), consistent with the
existing property on rtpvp8depay.
- rtpopuspay, rtpopusdepay: support libwebrtc-compatible multichannel
audio in addition to the previously supported multichannel audio
modes
- rtpopuspay: add DTX (Discontinuous Transmission) support
- rtpvp8depay: new "request-keyframe" property to make the depayloader
automatically request a new keyframe from the sender on packet loss.
- rtpvp8pay: temporal scaling support
- rtpvp9depay: Improved SVC handling (aggregate all layers)
RTP Infrastructure
- rtpst2022-1-fecdec, rtpst2022-1-fecenc: new elements providing SMPTE
2022-1 2-D Forward Error Correction. More details in Mathieu’s blog
post.
- rtpreddec: BUNDLE support
- rtpredenc, rtpulpfecenc: add support for Transport-wide Congestion
Control (TWCC)
- rtpsession: new "twcc-feedback-interval" property to allow RTCP TWCC
reports to be scheduled on a timer instead of per marker-bit.
Plugin and library moves
- There were no plugin moves or library moves in this cycle.
Plugin removals
The following elements or plugins have been removed:
- The ofa audio fingerprinting plugin has been removed. The MusicIP
database has been defunct for years so this plugin is likely neither
useful nor used by anyone.
- The mms plugin containing mmssrc has been removed. It seems unlikely
anyone still needs this or that there are even any streams left out
there. The MMS protocol was deprecated in 2003 (in favour of RTSP)
and support for it was dropped with Microsoft Media Services 2008,
and Windows Media Player apparently also does not support it any
more.
Miscellaneous API additions
Core
- gst_buffer_new_memdup() is a convenience function for the
widely-used gst_buffer_new_wrapped(g_memdup(data,size),size)
pattern.
- gst_caps_features_new_single() creates a new single GstCapsFeatures,
avoiding the need to use the vararg function with NULL terminator
for simple cases.
- gst_element_type_set_skip_documentation() can be used by plugins to
signal that certain elements should not be included in the GStreamer
plugin documentation. This is useful for plugins where elements are
registered dynamically based on hardware capabilities and/or where
the available plugins and properties vary from system to system.
This is used in the d3d11 plugin for example to ensure that only the
list of default elements is advertised in the documentation.
- gst_type_find_suggest_empty_simple() is a new convenience function
for typefinders for cases where there’s only a media type and no
other fields.
- New API to create elements and set properties at construction time,
which is not only convenient, but also allows GStreamer elements to
have construct-only properties: gst_element_factory_make_full(),
gst_element_factory_make_valist(),
gst_element_factory_make_with_properties(),
gst_element_factory_create_full(),
gst_element_factory_create_valist(),
gst_element_factory_create_with_properties().
- GstSharedTaskPool: new “shared” task pool subclass with slightly
different default behaviour than the existing GstTaskPool which
would create unlimited number of threads for new tasks. The shared
taskpool creates up to N threads (default: 1) and then distributes
pending tasks to those threads round-robin style, and blocks if no
thread is available. It is possible to join tasks. This can be used
by plugins to implement simple multi-threaded processing and is used
for the new multi-threaded video conversion and compositing done in
GstVideoAggregator, videoconverter and compositor.
Plugins Base Utils library
- GstDiscoverer:
- gst_discoverer_container_info_get_tags() was added to retrieve
global/container tags (vs. per-stream tags). Per-Stream tags can
be retrieved via the existing
gst_discoverer_stream_info_get_tags().
gst_discoverer_info_get_tags(), which for many files returns a
confusing mix of stream and container tags, has been deprecated
in favour of the container/stream-specific functions.
- gst_discoverer_stream_info_get_stream_number() returns a unique
integer identifier for a given stream within the given
GstDiscoverer context. (If this matches the stream number inside
the container bitstream that’s by coincidence and not by
design.)
- gst_pb_utils_get_caps_description_flags() can be used to query
whether certain caps represent a container, audio, video, image,
subtitles, tags, or something else. This only works for formats
known to GStreamer.
- gst_pb_utils_get_file_extension_from_caps() returns a possible file
extension for given caps.
- gst_codec_utils_h264_get_profile_flags_level(): Parses profile,
flags, and level from H264 AvcC codec_data. The format of H264 AVCC
extradata/sequence_header is documented in the ITU-T H.264
specification section 7.3.2.1.1 as well as in ISO/IEC 14496-15
section 5.3.3.1.2.
- gst_codec_utils_caps_get_mime_codec() to convert caps to a RFC 6381
compatible MIME codec string codec. Useful for providing the codecs
field inside the Content-Type HTTP header for containerized formats,
such as mp4 or matroska.
GStreamer OpenGL integration library and plugins
- glcolorconvert: added suppport for converting the video formats
A420, AV12, BGR, BGRA, RGBP and BGRP.
- Added support to GstGLBuffer for persistent buffer mappings where a
Pixel Buffer Object (PBO) can be mapped by both the CPU and the GPU.
This removes a memcpy() when uploading textures or vertices
particularly when software decoders (e.g. libav) are direct
rendering into our memory. Improves transfer performance
significantly. Requires OpenGL 4.4, GL_ARB_buffer_storage or
GL_EXT_buffer_storage
- Added various helper functions for handling 4x4 matrices of affine
transformations as used by GstVideoAffineTransformationMeta.
- Add support to GstGLContext for allowing the application to control
the config (EGLConfig, GLXConfig, etc) used when creating the OpenGL
context. This allows the ability to choose between RGB16 or RGB10A2
or RGBA8 back/front buffer configurations that were previously
hardcoded. GstGLContext also supports retrieving the configuration
it was created with or from an externally provide OpenGL context
handle. This infrastructure is also used to create a compatible
config from an application/externally provided OpenGL context in
order to improve compatibility with other OpenGL frameworks and GUI
toolkits. A new environment variable GST_GL_CONFIG was also added to
be able to request a specific configuration from the command line.
Note: different platforms will have different functionality
available.
- Add support for choosing between EGL and WGL at runtime when running
on Windows. Previously this was a build-time switch. Allows use in
e.g. Gtk applications on Windows that target EGL/ANGLE without
recompiling GStreamer. gst_gl_display_new_with_type() can be used by
applications to choose a specific display type to use.
- Build fixes to explicitly check for Broadcom-specific libraries on
older versions of the Raspberry Pi platform. The Broadcom OpenGL ES
and EGL libraries have different filenames. Using the vc4 Mesa
driver on the Raspberry Pi is not affected.
- Added support to glupload and gldownload for transferring RGBA
buffers using the memory:NVMM available on the Nvidia Tegra family
of embedded devices.
- Added support for choosing libOpenGL and libGLX as used in a GLVND
environment on unix-based platforms. This allows using desktop
OpenGL and EGL without pulling in any GLX symbols as would be
required with libGL.
Video library
- New raw video formats:
- AV12 (NV12 with alpha plane)
- RGBP and BGRP (planar RGB formats)
- ARGB64 variants with specified endianness instead of host
endianness:
- ARGB64_LE, ARGB64_BE
- RGBA64_BE, RGBA64_LE
- BGRA64_BE, BGRA64_LE
- ABGR64_BE, ABGR64_LE
- gst_video_orientation_from_tag() is new convenience API to parse the
image orientation from a GstTagList.
- GstVideoDecoder subframe support (see below)
- GstVideoCodecState now also carries some HDR metadata
- Ancillary video data: implement transform functions for AFD/Bar
metas, so they will be forwarded in more cases
MPEG-TS library
This library only handles section parsing and such, see above for
changes to the actual mpegtsmux and mpegtsdemux elements.
- many additions and improvements to SCTE-35 section parsing
- new API for fetching extended descriptors:
gst_mpegts_find_descriptor_with_extension()
- add support for SIT sections (Selection Information Tables)
- expose event-from-section constructor gst_event_new_mpegts_section()
- parse Audio Preselection Descriptor needed for Dolby AC-4
GstWebRTC library + webrtcbin
- Change the way in which sink pads and transceivers are matched
together to support easier usage. If a pad is created without a
specific index (i.e. using sink_%u as the pad template), then an
available compatible transceiver will be searched for. If a specific
index is requested (i.e. sink_1) then if a transceiver for that
m-line already exists, that transceiver must match the new sink pad
request. If there is no transceiver available in either scenario, a
new transceiver is created. If a mixture of both sink_1 and sink_%u
requests result in an impossible situation, an error will be
produced at pad request time or from create offer/answer.
- webrtcbin now uses regular ICE nomination instead of libnice’s
default of aggressive ICE nomination. Regular ICE nomination is the
default recommended by various relevant standards and improves
connectivity in specific network scenarios.
- Add support for limiting the port range used for RTP with the
addition of the min-rtp-port and max-rtp-port properties on the ICE
object.
- Expose the SCTP transport as a property on webrtcbin to more closely
match the WebRTC specification.
- Added support for taking into account the data channel transport
state when determining the value of the "connection-state" property.
Previous versions of the WebRTC spec did not include the data
channel state when computing this value.
- Add configuration for choosing the size of the underlying sockets
used for transporting media data
- Always advertise support for the transport-cc RTCP feedback protocol
as rtpbin supports it. For full support, the configured caps (input
or through codec-preferences) need to include the relevant RTP
header extension.
- Numerous fixes to caps and media handling to fail-fast when an
incompatible situation is detected.
- Improved support for attaching the required media after a remote
offer has been set.
- Add support for dynamically changing the amount of FEC used for a
particular stream.
- webrtcbin now stops further SDP processing at the first error it
encounters.
- Completed support for either local or the remote closing a data
channel.
- Various fixes when performing BUNDLEing of the media streams in
relation to RTX and FEC usage.
- Add support for writing out QoS DSCP marking on outgoing packets to
improve reliability in some network scenarios.
- Improvements to the statistics returned by the get-stats signal
including the addition of the raw statistics from the internal
RTPSource, the TWCC stats when available.
- The webrtc library does not expose any objects anymore with public
fields. Instead properties have been added to replace that
functionality. If you are accessing such fields in your application,
switch to the corresponding properties.
GstCodecs and Video Parsers
- Support for render delays to improve throughput across all CODECs
(used with NVDEC and V4L2).
- lots of improvements to parsers and the codec parsing decoder base
classes (H264, H265, VP8, VP9, AV1, MPEG-2) used for various
hardware-accelerate decoder APIs.
Bindings support
- gst_allocation_params_new() allocates a GstAllocationParams struct
on the heap. This should only be used by bindings (and freed via
gst_allocation_params_free() then). In C code you would allocate
this on the stack and only init it in place.
- gst_debug_log_literal() can be used to log a string to the debug log
without going through any printf format expansion and associated
overhead. This is mostly useful for bindings such as the Rust
bindings which may have done their own formatting already .
- Provide non-inlined versions of refcounting APIs for various
GStreamer mini objects, so that they can be consumed by bindings
(e.g. gstreamer-sharp): gst_buffer_ref, gst_buffer_unref,
gst_clear_buffer, gst_buffer_copy, gst_buffer_replace,
gst_buffer_list_ref, gst_buffer_list_unref, gst_clear_buffer_list,
gst_buffer_list_copy, gst_buffer_list_replace, gst_buffer_list_take,
gst_caps_ref, gst_caps_unref, gst_clear_caps, gst_caps_replace,
gst_caps_take, gst_context_ref, gst_context_unref, gst_context_copy,
gst_context_replace, gst_event_replace, gst_event_steal,
gst_event_take, gst_event_ref, gst_event_unref, gst_clear_event,
gst_event_copy, gst_memory_ref, gst_memory_unref, gst_message_ref,
gst_message_unref, gst_clear_message, gst_message_copy,
gst_message_replace, gst_message_take, gst_promise_ref,
gst_promise_unref, gst_query_ref, gst_query_unref, gst_clear_query,
gst_query_copy, gst_query_replace, gst_query_take, gst_sample_ref,
gst_sample_unref, gst_sample_copy, gst_tag_list_ref,
gst_tag_list_unref, gst_clear_tag_list, gst_tag_list_replace,
gst_tag_list_take, gst_uri_copy, gst_uri_ref, gst_uri_unref,
gst_clear_uri.
- expose a GType for GstMiniObject
- gst_device_provider_probe() now returns non-floating device object
API Deprecations
- gst_element_get_request_pad() has been deprecated in favour of the
newly-added gst_element_request_pad_simple() which does the exact
same thing but has a less confusing name that hopefully makes clear
that the function request a new pad rather than just retrieves an
already-existing request pad.
- gst_discoverer_info_get_tags(), which for many files returns a
confusing mix of stream and container tags, has been deprecated in
favour of the container-specific and stream-specific functions,
gst_discoverer_container_info_get_tags() and
gst_discoverer_stream_info_get_tags().
- gst_video_sink_center_rect() was deprecated in favour of the more
generic newly-added gst_video_center_rect().
- The GST_MEMORY_FLAG_NO_SHARE flag has been deprecated, as it tends
to cause problems and prevents sub-buffering. If pooling or lifetime
tracking is required, memories should be allocated through a custom
GstAllocator instead of relying on the lifetime of the buffers the
memories were originally attached to, which is fragile anyway.
- The GstPlayer high-level playback library is being replaced with the
new GstPlay library (see above). GstPlayer should be considered
deprecated at this point and will be marked as such in the next
development cycle. Applications should be ported to GstPlay.
- Gstreamer Editing Services: ges_video_transition_set_border(),
ges_video_transition_get_border()
ges_video_transition_set_inverted()
ges_video_transition_is_inverted() have been deprecated, use
ges_timeline_element_set_children_properties() instead.
Miscellaneous performance, latency and memory optimisations
More video conversion fast paths
- v210 ↔ I420, YV12, Y42B, UYVY and YUY2
- A420 → RGB
Less jitter when waiting on the system clock
- Better system clock wait accuracy, less jitter: where available,
clock_nanosleep is used for higher accuracy for waits below 500
usecs, and waits below 2ms will first use the regular waiting system
and then clock_nanosleep for the remainder. The various wait
implementation have a latency ranging from 50 to 500+ microseconds.
While this is not a major issue when dealing with a low number of
waits per second (for ex: video), it does introduce a non-negligible
jitter for synchronisation of higher packet rate systems.
Video decoder subframe support
- The GstVideoDecoder base class gained API to process input at the
sub-frame level. That way video decoders can start decoding slices
before they have received the full input frame in its entirety (to
the extent this is supported by the codec, of course). This helps
with CPU utilisation and reduces latency.
- This functionality is now being used in the OpenJPEG JPEG 2000
decoder, the FFmpeg H.264 decoder (in case of NAL-aligned input) and
the OpenMAX H.264/H.265 decoders (in case of NAL-aligned input).
Miscellaneous other changes and enhancements
- GstDeviceMonitor no longer fails to start just because one of the
device providers failed to start. That could happen for example on
systems where the pulseaudio device provider is installed, but
pulseaudio isn’t actually running but ALSA is used for audio
instead. In the same vein the device monitor now keeps track of
which providers have been started (via the new
gst_device_provider_is_started()) and only stops actually running
device providers when stopping the device monitor.
- On embedded systems it can be useful to create a registry that can
be shared and read by multiple processes running as different users.
It is now possible to set the new GST_REGISTRY_MODE environment
variable to specify the file mode for the registry file, which by
default is set to be only user readable/writable.
- GstNetClientClock will signal lost sync in case the remote time
resets (e.g. because device power cycles), by emitting the “synced”
signal with synced=FALSE parameter, so applications can take action.
- gst_value_deserialize_with_pspec() allows deserialization with a
hint for what the target GType should be. This allows for example
passing arrays of flags through the command line or
gst_util_set_object_arg(), eg: foo="<bar,bar+baz>".
- It’s now allowed to create an empty GstVideoOverlayComposition
without any rectangles by passing a NULL rectangle to
gst_video_overlay_composition_new(). This is useful for bindings and
simplifies application code in some places.
Tracing framework, debugging and testing improvements
- New factories tracer to list loaded elements (and other plugin
features). This can be useful to collect a list of elements needed
for an application, which then in turn can be used to create a
tailored minimal GStreamer build that contains just the elements
needed and nothing else.
- New plugin-feature-loaded tracing hook for use by tracers like the
new factories tracer
- GstHarness: Add gst_harness_set_live() so that harnesses can be set
to non-live and return is-live=false in latency queries if needed.
Default behaviour is to always return is-live=true in latency
queries.
- navseek: new "hold-eos" property. When enabled, the element will
hold back an EOS event until the next keystroke (via navigation
events). This can be used to keep a video sink showing the last
frame of a video pipeline until a key is pressed instead of tearing
it down immediately on EOS.
- New fakeaudiosink element: mimics an audio sink and can be used for
testing and CI pipelines on systems where no audio system is
installed or running. It differs from fakesink in that it only
support audio caps and syncs to the clock by default like a normal
audio sink. It also implements the GstStreamVolume interface like
most audio sinks do.
- New videocodectestsink element for video codec conformance testing:
Calculates MD5 checksums for video frames and skips any padding
whilst doing so. Can optionally also write back the video data with
padding removed into a file for easy byte-by-byte comparison with
reference data.
Tools
gst-inspect-1.0
- Can sort the list of plugins by passing --sort=name as command line
option
gst-launch-1.0
- will now error out on top-level properties that don’t exist and
which were silently ignored before
- On Windows the high-resolution clock is enabled now, which provides
better clock and timer performance on Windows (see Windows section
below for more details).
gst-play-1.0
- New --start-position command line argument to start playback from
the specified position
- Audio can be muted/unmuted in interactive mode by pressing the m
key.
- On Windows the high-resolution clock is enabled now (see Windows
section below for more details)
gst-device-monitor-1.0
- New --include-hidden command line argument to also show “hidden”
device providers
ges-launch-1.0
- New interactive mode that allows seeking and such. Can be disabled
by passing the --no-interactive argument on the command line.
- Option to forward tags
- Allow using an existing clip to determine the rendering format (both
topology and profile) via new --profile-from command line argument.
GStreamer RTSP server
- GstRTSPMediaFactory gained API to disable RTCP
(gst_rtsp_media_factory_set_enable_rtcp(), "enable-rtcp" property).
Previously RTCP was always allowed for all RTSP medias. With this
change it is possible to disable RTCP completely, no matter if the
client wants to do RTCP or not.
- Make a mount point of / work correctly. While not allowed by the
RTSP 2 spec, the RTSP 1 spec is silent on this and it is used in the
wild. It is now possible to use / as a mount path in
gst-rtsp-server, e.g. rtsp://example.com/ would work with this now.
Note that query/fragment parts of the URI are not necessarily
correctly handled, and behaviour will differ between various
client/server implementations; so use it if you must but don’t bug
us if it doesn’t work with third party clients as you’d hoped.
- multithreading fixes (races, refcounting issues, deadlocks)
- ONVIF audio backchannel fixes
- ONVIF trick mode optimisations
- rtspclientsink: new "update-sdp" signal that allows updating the SDP
before sending it to the server via ANNOUNCE. This can be used to
add additional metadata to the SDP, for example. The order and
number of medias must not be changed, however.
GStreamer VAAPI
- new AV1 decoder element (vaapiav1dec)
- H264 decoder: handle stereoscopic 3D video with frame packing
arrangement SEI messages
- H265 encoder: added Screen Content Coding extensions support
- H265 decoder: gained MAIN_444_12 profile support (decoded to
Y412_LE), and 4:2:2 12-bits support (decoded to Y212_LE)
- vaapipostproc: gained BT2020 color standard support
- vaapidecode: now generates caps templates dynamically at runtime in
order to advertise actually supported caps instead of all
theoretically supported caps.
- GST_VAAPI_DRM_DEVICE environment variable to force a specified DRM
device when a DRM display is used. It is ignored when other types of
displays are used. By default /dev/dri/renderD128 is used for DRM
display.
GStreamer OMX
- subframe support in H.264/H.265 decoders
GStreamer Editing Services and NLE
- framepositioner: new "operator" property to access blending modes in
the compositor
- timeline: Implement snapping to markers
- smart-mixer: Add support for d3d11compositor and glvideomixer
- titleclip: add "draw-shadow" child property
- ges:// URI support to define a timeline from a description.
- command-line-formatter
- Add track management to timeline description
- Add keyframe support
- ges-launch-1.0:
- Add an interactive mode where we can seek etc…
- Add option to forward tags
- Allow using an existing clip to determine the rendering format
(both topology and profile) via new --profile-from command line
argument.
- Fix static build
GStreamer validate
- report: Add a way to force backtraces on reports even if not a
critical issue (GST_VALIDATE_ISSUE_FLAGS_FORCE_BACKTRACE)
- Add a flag to gst_validate_replace_variables_in_string() allow
defining how to resolve variables in structs
- Add gst_validate_bin_monitor_get_scenario() to get the bin monitor
scenario, which is useful for applications that use Validate
directly.
- Add an expected-values parameter to wait, message-type=XX allowing
more precise filtering of the message we are waiting for.
- Add config file support: each test can now use a config file for the
given media file used to test.
- Add support to check properties of object properties
- scenario: Add an "action-done" signal to signal when an action is
done
- scenario: Add a "run-command" action type
- scenario: Allow forcing running action on idle from scenario file
- scenario: Allow iterating over arrays in foreach
- scenario: Rename ‘interlaced’ action to ‘non-blocking’
- scenario: Add a non-blocking flag to the wait signal
GStreamer Python Bindings
- Fixes for Python 3.10
- Various build fixes
- at least one known breaking change caused by g-i annotation changes
(see below)
GStreamer C# Bindings
- Fix GstDebugGraphDetails enum
- Updated to latests GtkSharp
- Updated to include GStreamer 1.20 API
GStreamer Rust Bindings and Rust Plugins
- The GStreamer Rust bindings are released separately with a different
release cadence that’s tied to gtk-rs, but the latest release has
already been updated for the upcoming new GStreamer 1.20 API (v1_20
feature).
- gst-plugins-rs, the module containing GStreamer plugins written in
Rust, has also seen lots of activity with many new elements and
plugins. See the New Elements section above for a list of new Rust
elements.
Build and Dependencies
- Meson 0.59 or newer is required to build GStreamer now.
- The GLib requirement has been bumped to GLib 2.56 or newer (from
March 2018).
- The wpe plugin now requires wpe >= 2.28 and wpebackend-fdo >= 1.8
Explicit opt-in required for build of certain plugins with (A)GPL dependencies
Some plugins have GPL- or AGPL-licensed dependencies and those plugins
will no longer be built by default unless you have explicitly opted in
to allow (A)GPL-licensed dependencies by passing -Dgpl=enabled to Meson,
even if the required dependencies are available.
See Building plugins with (A)GPL-licensed dependencies for more details
and a non-exhaustive list of plugins affected.
gst-build: replaced by mono repository
See mono repository section above and the GStreamer mono repository FAQ.
Cerbero
Cerbero is a meta build system used to build GStreamer plus dependencies
on platforms where dependencies are not readily available, such as
Windows, Android, iOS and macOS.
General Cerbero improvements
- Plugin removed: libvisual
- New plugins: rtpmanagerbad and rist
macOS / iOS specific Cerbero improvements
- XCode 12 support
- macOS OS release support is now future-proof, similar to iOS
- macOS Apple Silicon (ARM64) cross-compile support has been added
- macOS Apple Silicon (ARM64) native support is currently experimental
Windows specific Cerbero improvements
- Visual Studio 2022 support has been added
- bootstrap is faster since it requires building fewer build-tools
recipes on Windows
- package is faster due to better scheduling of recipe stages and
elimination of unnecessary autotools regeneration
- The following plugins are no longer built on Windows:
- a52dec (another decoder is still available in libav)
- dvdread
- resindvd
Windows MSI installer
- no major changes
Linux specific Cerbero improvements
- Fedora, Debian OS release support is now more future-proof
- Amazon Linux 2 support has been added
Android specific Cerbero improvements
- no major changes
Platform-specific changes and improvements
Android
- No major changes
macOS and iOS
- applemedia: add ProRes support to vtenc and vtdec
Windows
- On Windows the high-resolution clock is enabled now in the
gst-launch-1.0 and gst-play-1.0 command line tools, which provides
better clock and timer performance on Windows, at the cost of higher
power consumption. By default, without the high-resolution clock
enabled, the timer precision on Windows is system-dependent and may
be as bad as 15ms which is not good enough for many multimedia
applications. Developers may want to do the same in their Windows
applications if they think it’s a good idea for their application
use case, and depending on the Windows version they target. This is
not done automatically by GStreamer because on older Windows
versions (pre-Windows 10) this affects a global Windows setting and
also there’s a power consumption vs. performance trade-off that may
differ from application to application.
- dxgiscreencapsrc now supports resolution changes
- The wasapi2 audio plugin was rewritten and now has a higher rank
than the old wasapi plugin since it has a number of additional
features such as automatic stream routing, and no
known-but-hard-to-fix issues. The plugin is always built if the
Windows 10 SDK is available now.
- The wasapi device providers now detect and notify dynamic device
additions/removals
- d3d11screencapturesrc: new desktop capture element, including
GstDeviceProvider implementation to enumerate/select target monitors
for capture.
- Direct3D11/DXVA decoder now supports AV1 and MPEG2 codecs
(d3d11av1dec, d3d11mpeg2dec)
- VP9 decoding got more reliable and stable thanks to a newly written
codec parser
- Support for decoding interlaced H.264/AVC streams
- Hardware-accelerated video deinterlacing (d3d11deinterlace) and
video mixing (d3d11compositor)
- Video mixing with the Direct3D11 API (d3d11compositor)
- MediaFoundation API based hardware encoders gained the ability to
receive Direct3D11 textures as an input
- Seungha’s blog post “GStreamer ❤ Windows: A primer on the cool stuff
you’ll find in the 1.20 release” describes many of the
Windows-related improvements in more detail
Linux
- bluez: LDAC Bluetooth audio codec support in a2dpsink and avdtpsink,
as well as an LDAC RTP payloader (rtpldacpay) and an LDAC audio
encoder (ldacenc)
- kmssink: gained support for NV24, NV61, RGB16/BGR16 formats;
auto-detect NVIDIA Tegra driver
Documentation improvements
- hardware-accelerated GPU plugins will now no longer always list all
the element variants for all available GPUs, since those are
system-dependent and it’s confusing for users to see those in the
documentation just because the GStreamer developer who generated the
docs had multiple GPUs to play with at the time. Instead just show
the default elements.
Possibly Breaking and Other Noteworthy Behavioural Changes
- gst_parse_launch(), gst_parse_bin_from_description() and friends
will now error out when setting properties that don’t exist on
top-level bins. They were silently ignored before.
- The GstWebRTC library does not expose any objects anymore with
public fields. Instead properties have been added to replace that
functionality. If you are accessing such fields in your application,
switch to the corresponding properties.
- playbin and uridecodebin now emit the source-setup signal before the
element is added to the bin and linked so that the source element is
already configured before any scheduling query comes in, which is
useful for elements such as appsrc or giostreamsrc.
- The source element inside urisourcebin (used inside uridecodebin3
which is used inside playbin3) is no longer called "source". This
shouldn’t affect anyone hopefully, because there’s a "setup-source"
signal to configure the source element and no one should rely on
names of internal elements anyway.
- The vp8enc element now expects bps (bits per second) for the
"temporal-scalability-target-bitrate" property, which is consistent
with the "target-bitrate" property. Since additional configuration
is required with modern libvpx to make temporal scaling work anyway,
chances are that very few people will have been using this property
- vp8enc and vp9enc now default to “good quality” for the "deadline"
property rather then “best quality”. Having the deadline set to best
quality causes the encoder to be absurdly slow, most real-life users
will want the good quality tradeoff instead.
- The experimental GstTranscoder library API in gst-plugins-bad was
changed from a GObject signal-based notification mechanism to a
GstBus/message-based mechanism akin to GstPlayer/GstPlay.
- MPEG-TS SCTE-35 API: semantic change for SCTE-35 splice commands:
timestamps passed by the application should be in running time now,
since users of the API can’t really be expected to predict the local
PTS of the muxer.
- The GstContext used by souphttpsrc to share the session between
multiple element instances has changed. Previously it provided
direct access to the internal SoupSession object, now it only
provides access to an opaque, internal type. This change is
necessary because SoupSession is not thread-safe at all and can’t be
shared safely between arbitrary external code and souphttpsrc.
- Python bindings: GObject-introspection related Annotation fixes have
led to a case of a GstVideo.VideoInfo-related function signature
changing in the Python bindings (possibly one or two other cases
too). This is for a function that should never have been exposed in
the first place though, so the bindings are being updated to throw
an exception in that case, and the correct replacement API has been
added in form of an override.
Known Issues
- nothing in particular at this point (but also see possibly breaking
changes section above)
Contributors
Aaron Boxer, Adam Leppky, Adam Williamson, Alba Mendez, Alejandro
González, Aleksandr Slobodeniuk, Alexander Vandenbulcke, Alex Ashley,
Alicia Boya García, Andika Triwidada, Andoni Morales Alastruey, Andrew
Wesie, Andrey Moiseev, Antonio Ospite, Antonio Rojas, Arthur Crippa
Búrigo, Arun Raghavan, Ashley Brighthope, Axel Kellermann, Baek, Bastien
Nocera, Bastien Reboulet, Benjamin Gaignard, Bing Song, Binh Truong,
Biswapriyo Nath, Brad Hards, Brad Smith, Brady J. Garvin, Branko
Subasic, Camilo Celis Guzman, Chris Bass, ChrisDuncanAnyvision, Chris
White, Corentin Damman, Daniel Almeida, Daniel Knobe, Daniel Stone,
david, David Fernandez, David Keijser, David Phung, Devarsh Thakkar,
Dinesh Manajipet, Dmitry Samoylov, Dmitry Shusharin, Dominique Martinet,
Doug Nazar, Ederson de Souza, Edward Hervey, Emmanuel Gil Peyrot,
Enrique Ocaña González, Ezequiel Garcia, Fabian Orccon, Fabrice
Fontaine, Fernando Jimenez Moreno, Florian Karydes, Francisco Javier
Velázquez-García, François Laignel, Frederich Munch, Fredrik Pålsson,
George Kiagiadakis, Georg Lippitsch, Göran Jönsson, Guido Günther,
Guillaume Desmottes, Guiqin Zou, Haakon Sporsheim, Haelwenn (lanodan)
Monnier, Haihao Xiang, Haihua Hu, Havard Graff, He Junyan, Helmut
Januschka, Henry Wilkes, Hosang Lee, Hou Qi, Ignacio Casal Quinteiro,
Igor Kovalenko, Ilya Kreymer, Imanol Fernandez, Jacek Tomaszewski, Jade
Macho, Jakub Adam, Jakub Janků, Jan Alexander Steffens (heftig), Jan
Schmidt, Jason Carrete, Jason Pereira, Jay Douglass, Jeongki Kim, Jérôme
Laheurte, Jimmi Holst Christensen, Johan Sternerup, John Hassell, John
Lindgren, John-Mark Bell, Jonathan Matthew, Jordan Petridis, Jose
Quaresma, Julian Bouzas, Julien, Kai Uwe Broulik, Kasper Steensig
Jensen, Kellermann Axel, Kevin Song, Khem Raj, Knut Inge Hvidsten, Knut
Saastad, Kristofer Björkström, Lars Lundqvist, Lawrence Troup, Lim Siew
Hoon, Lucas Stach, Ludvig Rappe, Luis Paulo Fernandes de Barros, Luke
Yelavich, Mads Buvik Sandvei, Marc Leeman, Marco Felsch, Marek Vasut,
Marian Cichy, Marijn Suijten, Marius Vlad, Markus Ebner, Mart Raudsepp,
Matej Knopp, Mathieu Duponchelle, Matthew Waters, Matthieu De Beule,
Mengkejiergeli Ba, Michael de Gans, Michael Olbrich, Michael Tretter,
Michal Dzik, Miguel Paris, Mikhail Fludkov, mkba, Nazar Mokrynskyi,
Nicholas Jackson, Nicola Murino, Nicolas Dufresne, Niklas Hambüchen,
Nikolay Sivov, Nirbheek Chauhan, Olivier Blin, Olivier Crete, Olivier
Crête, Paul Goulpié, Per Förlin, Peter Boba, P H, Philippe Normand,
Philipp Zabel, Pieter Willem Jordaan, Piotrek Brzeziński, Rafał
Dzięgiel, Rafostar, raghavendra, Raghavendra, Raju Babannavar, Raleigh
Littles III, Randy Li, Randy Li (ayaka), Ratchanan Srirattanamet, Raul
Tambre, reed.lawrence, Ricky Tang, Robert Rosengren, Robert Swain, Robin
Burchell, Roman Sivriver, R S Nikhil Krishna, Ruben Gonzalez, Ruslan
Khamidullin, Sanchayan Maity, Scott Moreau, Sebastian Dröge, Sergei
Kovalev, Seungha Yang, Sid Sethupathi, sohwan.park, Sonny Piers, Staz M,
Stefan Brüns, Stéphane Cerveau, Stephan Hesse, Stian Selnes, Stirling
Westrup, Théo MAILLART, Thibault Saunier, Tim, Timo Wischer, Tim-Philipp
Müller, Tim Schneider, Tobias Ronge, Tom Schoonjans, Tulio Beloqui,
tyler-aicradle, U. Artie Eoff, Ung, Val Doroshchuk, VaL Doroshchuk,
Víctor Manuel Jáquez Leal, Vivek R, Vivia Nikolaidou, Vivienne
Watermeier, Vladimir Menshakov, Will Miller, Wim Taymans, Xabier
Rodriguez Calvar, Xavier Claessens, Xℹ Ruoyao, Yacine Bandou, Yinhang
Liu, youngh.lee, youngsoo.lee, yychao, Zebediah Figura, Zhang yuankun,
Zhang Yuankun, Zhao, Zhao Zhili, , Aleksandar Topic, Antonio Ospite,
Bastien Nocera, Benjamin Gaignard, Brad Hards, Carlos Falgueras García,
Célestin Marot, Corentin Damman, Corentin Noël, Daniel Almeida, Daniel
Knobe, Danny Smith, Dave Piché, Dmitry Osipenko, Fabrice Fontaine,
fjmax, Florian Zwoch, Guillaume Desmottes, Haihua Hu, Heinrich Kruger,
He Junyan, Jakub Adam, James Cowgill, Jan Alexander Steffens (heftig),
Jean Felder, Jeongki Kim, Jiri Uncovsky, Joe Todd, Jordan Petridis,
Krystian Wojtas, Marc-André Lureau, Marcin Kolny, Marc Leeman, Mark
Nauwelaerts, Martin Reboredo, Mathieu Duponchelle, Matthew Waters,
Mengkejiergeli Ba, Michael Gruner, Nicolas Dufresne, Nirbheek Chauhan,
Olivier Crête, Philippe Normand, Rafał Dzięgiel, Ralf Sippl, Robert
Mader, Sanchayan Maity, Sangchul Lee, Sebastian Dröge, Seungha Yang,
Stéphane Cerveau, Teh Yule Kim, Thibault Saunier, Thomas Klausner, Timo
Wischer, Tim-Philipp Müller, Tobias Reineke, Tomasz Andrzejak, Trung Do,
Tyler Compton, Ung, Víctor Manuel Jáquez Leal, Vivia Nikolaidou, Wim
Taymans, wngecn, Wonchul Lee, wuchang li, Xavier Claessens, Xi Ruoyao,
Yoshiharu Hirose, Zhao,
… and many others who have contributed bug reports, translations, sent
suggestions or helped testing.
Stable 1.20 branch
After the 1.20.0 release there will be several 1.20.x bug-fix releases
which will contain bug fixes which have been deemed suitable for a
stable branch, but no new features or intrusive changes will be added to
a bug-fix release usually. The 1.20.x bug-fix releases will be made from
the git 1.20 branch, which will be a stable branch.
1.20.0
1.20.0 is scheduled to be released around early February 2022.
Schedule for 1.22
Our next major feature release will be 1.22, and 1.21 will be the
unstable development version leading up to the stable 1.22 release. The
development of 1.21/1.22 will happen in the git main branch.
The plan for the 1.22 development cycle is yet to be confirmed. Assuming
no major project-wide reorganisations in the 1.22 cycle we might try and
aim for a release around August 2022.
1.22 will be backwards-compatible to the stable 1.20, 1.18, 1.16, 1.14,
1.12, 1.10, 1.8, 1.6, 1.4, 1.2 and 1.0 release series.
------------------------------------------------------------------------
These release notes have been prepared by Tim-Philipp Müller with
contributions from Matthew Waters, Nicolas Dufresne, Nirbheek Chauhan,
Sebastian Dröge and Seungha Yang.
License: CC BY-SA 4.0
Reference in a new issue View git blame Copy permalink