Even if no new synchronization information is available.
This is necessary because the timestamp offset logic in rtpbin depends
on the base RTP time that is determined by the jitterbuffer, but this
changes all the time (especially in mode=slave) and the timestamp
offsets have to be updated accordingly. Doing so is especially important
if they're only determined by the RTP-Info, which never changes from the
very beginning.
The interval can be configured via the new min-sync-interval property.
Synchronization happens at least that often, but at most as often as the
old sync-interval property allows.
Both intervals are now based on the monotonic system clock.
Additionally, clean up synchronization code a bit, only emit either
inband NTP or RTCP SR synchronization at the same time, based on which
one has the more recent time information, and only emit RTP-Info
synchronization if it wasn't provided previously at the same time as the
NTP-based synchronization information.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/6543>
There is generally no requirement to ignore RTCP SR if the RTP time of
the SR differs a lot from the last received RTP packet. The mapping
between RTP and NTP time stays valid until there was a stream reset, in
which case we wouldn't use that information anyway.
When using rtcp-sync-send-time=false the default of 1s difference can
easily be exceeded, e.g. if encoding of the stream after capture adds
more than 1s of latency.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/6543>
Never is useful for some RTSP servers that report plain garbage both via
RTCP SR and RTP-Info, for example.
NTP is useful if synchronization should only ever happen based on RTCP
SR or NTP-64 RTP header extension.
Also slightly change the behaviour of always/initial to take RTP-Info
based synchronization into account too. It's supposed to give the same
values as the RTCP SR and is available earlier, so will generally cause
fewer synchronization glitches if it's made use of.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/6543>
Instead of switching on the very first stream, require that all streams
have switched before switching to the different synchronization
mechanism.
Without this there will be a noticeable gap during the switch. E.g. when
going from RTP-Info to NTP-based association, first the first stream
only would get an offset, then the first two, ... then all of them.
Depending on the order of streams this will cause a lot of changes in
ts-offset during the transition.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/6543>
Previously these parameters were randomly changed in the body of the
function to avoid having to declare a new variable, which made the code
very hard to follow. By marking them as const this won't be possible
anymore in the future.
Also the RTP clock-base (RTP time from RTSP RTP-Info) is an unsigned
64 bit integer as it's an extended RTP timestamp.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/6543>
Both were entangled previously and very hard to follow what happens
under which conditions. Now as a very first step the code decides which
of the two cases it is going to apply, and then proceeds accordingly.
This also avoids calculating completely invalid values along the way and
even printing them int the debug output.
Also improve debug output in various places.
This shouldn't cause any behaviour changes.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/6543>
This simplifies the code as it's a much simpler case than the normal
inter-stream synchronization, and interleaving it with that only
reduces readability of the code.
Also improve some debug output in this code path.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/6543>
Adds a separate vtenc_h265a element (with a _hw variant as usual) for the HEVCWithAlpha codec type.
Decided to go with a separate element to not break existing uses of the normal HEVC encoder.
The preserve_alpha property is still only used for ProRes, no need for it here because we explicitly say we want alpha
when using the new element.
For now, the HEVCWithAlpha has an issue where it does not throttle the amount of input frames queued internally.
I added a quick workaround where encode_frame() will block until enqueue_frame() callback notifies it that some space
has been freed up in the internal queue. The limit was set to 5, which should be enough I guess? Hopefully this is not
too prone to race conditions.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/6664>
When we are dealing with parsed inputs (i.e. using identity), we need to ensure
that we have a valid stream collection (and therefore DBCollection) before
anything flows dowsntream.
In those cases, we hold onto those events until we get such a collection.
Fixes#3356
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/6774>
This commit separates collection and selections into a new separate structure:
DecodebinCollection.
This provides a much cleaner/saner way of dealing with collections being
updated, gapless playback, etc...
There is now a list of DecodebinCollection in flight, of which two are special:
* input_collection, the currently inputted/merged collection
* output_collection, the currently active collection on the output of multiqueue
Handling GST_EVENT_SELECT_STREAMS is split, by looking for the collection to
which it applies. And the requested streams are stored in it. IIF that
collection is output_collection we can do the switch, else it will be updated
when it becomes active.
Detecting which collection/selection is active is done by looking at the
GST_EVENT_STREAM_START on the output of the multiqueue.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/6774>
* Move the handling of GST_EVENT_STREAM_START on a slot to a separate function
* There was a lot of usage of `gst_stream_get_stream_id()` for the slot
active_stream. Cache that instead of constantly querying it.
* Rename the variables in `handle_stream_switch()` to be clearer
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/6774>
* Centralize associating an output to a slot in one function, including properly
resetting those fields
* Rename functions to be more explicit
* Move code to "reset" an output stream into a dedicated function (will be used
later)
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/6774>
* Rename the function names to be clearer, with prefixes
* Pass the input (or stream) directly where appropriate
* Document usage, inputs, ownership
* Rename variables for clarity where applicable
* Avoid double lock/unlock if callee can handle it directly
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/6774>
Simplify its usage by having it directly create the message if the collection
changed. This is what caller were always doing and avoids releasing selection
locks yet-another-time
Also use it in more places to avoid code repetition
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/6774>
`StdVideoDecodeH265PictureInfo.flags.IsReference` refers to section 3.132 ITU-T
H.265 specification:
reference picture: A picture that is a short-term reference picture or a
long-term reference picture.
`GstH265Picture.ref` doesn't reflect this, but we need to query the NAL type of
the processed slice.
This patch fixes the validation layer error
`VUID-vkCmdBeginVideoCodingKHR-slotIndex-07239` while using the NVIDIA driver.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/6901>
Conceptually identical to the present signal of d3d11videosink.
This signal will be emitted with current render target
(i.e., swapchain backbuffer) and command queue. Signal handler
can record GPU commands for an overlay image or to blend
an image to the render target.
In addition to d3d12 resources, videosink will send
d3d11 and d2d resources depending on "overlay-mode"
property, so that signal handler can render by using
preferred/required DirectX API.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/6838>
The idea of using separate command queue per videosink was that
swapchain is bound to a command queue and we need to flush the
command queue when window size is changed. But the separate
queue does not seem to improve performance a lot.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/6838>
On various 32 bit systems, time_t is actually 64 bits while long is
still only 32 bits. The macro would wrongly trigger its assertion in
this case if a value with more than 68 years worth of seconds is
converted.
Examples are various newer 32 bit platforms and old ones that are
compiled with -D_TIME_BITS=64.
Also statically assert that time_t is either 32 or 64 bits. Other values
might need adjustments in the macro.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/6869>
The prime_fds for multi planes may be the same. For example, on Intel's
platform, the NV12 surface may have the same FD for the plane0 and the
plane1. Then, the DRM_IOCTL_GEM_CLOSE will close the same handle twice
and get an "Invalid argument 22" error the second time.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/6914>
From the spec (chapter 34, v1.3.283):
````
UNORM: the components are unsigned normalized values in the range [0, 1]
SRGB: the R, G and B components are unsigned normalized value that represent
values using sRGB nonlinear encoding, while the A component (if one
exists) is a regular unsigned normalized value
```
The difference is the storage encoding, the first one is aimed for image
transfers, while the second is for shaders, mostly in the swapchain stage in the
pipeline, and it's done automatically if needed [1].
As far as I have checked, other frameworks (FFmpeg, GTK+), when import or export
images from/to Vulkan, use exclusively UNORM formats, while SRGB formats are
ignored.
My conclusion is that Vulkan formats are related on how bits are stored in
memory rather their transfer functions (colorimetry).
This patch does two interrelated changes:
1. It swaps certain color format maps to try first, in both
gst_vulkan_format_from_video_info() and gst_vulkan_format_from_video_info_2(),
the UNORM formats, when comparing its usage, and later check for SRGB.
2. It removes the code that check for colorimetry in
gst_vulkan_format_from_video_info_2(), since it not storage related.
1. https://community.khronos.org/t/noob-difference-between-unorm-and-srgb/106132/7
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/6797>
This mirrors the behaviour in vp8enc / vp9enc and is generally more
useful than using any framerate from the caps as it provides some degree
of accuracy if the stream doesn't have timestamps perfectly according to
the framerate.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/6891>
Turns out AudioConvertHostTimeToNanos and AudioGetCurrentHostTime are macOS-only APIs, which prevents apps using
GStreamer on iOS from being accepted into App Store.
This commit replaces those functions with a manual version of what they do - mach_absolute_time() for the current time,
and data from mach_timebase_info() at the beginning to convert host timestamps to nanoseconds.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/6789>
The `videosink` refernce in main() is a floating one, so it should not
be unref'ed (the playbin practically takes ownership of it).
This prevents a "gst_object_unref: assertion '((GObject *)
object)->ref_count > 0' failed" at runtime.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/6883>
Using g_error() crashes the application, producing a coredump e.g. when
the user closes the video window, which can be confusing (especially on
the very first tutorial).
Let's change this to log the error message without crashing, using
g_printerr(), like subsequent tutorials.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/6883>
To simplify the description, I'm assuming we only have two streams: video and audio.
For the video stream, we have the following events :
- STREAM_START => stream->wait set to true
- NEW_SEGMENT(1) => blocked waiting in gst_stream_synchronizer_wait
- FLUSH_START => unblocked
- FLUSH_STOP => stream->wait reset to false
- NEW_SEGMENT(2) => not waiting, since stream->wait is false
Then for the audio stream, we have the following events :
- STREAM_START => stream->wait set to true
- NEW_SEGMENT(2) => blocked waiting in gst_stream_synchronizer_wait for ever.
Note: The first NEW_SEGMENT event and the FLUSH_START, FLUSH_STOP events of the audio stream
are dropped before being received by the streamsynchronizer element, because the decodebin audio pad src
is not yet linked to the playsink audio pad sink.
To fix this deadlock, we don't reset stream->wait to false in the FLUSH_STOP event when it is not
waiting for the EOS of the other streams.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/6763>
If we have constant duration buffers, set the duration on
outgoing buffers, like rtpmp4adepay does. This fixes
problems with (for example) muxers like mp4mux not writing
the duration of the final sample into the index.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/6878>
_get_osfhandle() expects valid fd and CRT will abort program
if given paramerter is invalid. The fd can be invalidated
in various way, file was deleted by other process after
we open a file. To avoid it, our own exception
handler must be installed so that _get_osfhandle() can return
INVALID_HANDLE_VALUE if fd is invalid.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/6877>
In some cases you want to ensure that a specific element factory is used
while requiring some specific caps but this was not possible. You can
now do `qtmux:video/x-prores,variant=standard|factory-name=avenc_prores_ks`
to ensure that the `avenc_prores_ks` factory is used to produce the
'standard' variant of prores video stream.
This also enhances a bit the documentation
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/6875>
The encoder can specify the a preferred_output_delay value to get better throughput
performance. The higher delay may get better HW performance, but it may increases
the encoder and pipeline latency.
When the output queue length is smaller than preferred_output_delay, the encoder
will not block to waiting for the encoding output. It will continue to prepare and
send more commands to GPU, which may improve the encoder throughput performance.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/4359>
This counter is incremented once for every segment, meaning it would
e.g. overflow after 24 days when using 1ms segments. Once that happens,
completely wrong positions are reported and invalid memory is handed out
for writing/reading the next segments.
As the affected variables are unfortunately part of the public API of
the struct, a second set of variables is added together with accessor
functions and both variables are kept in sync for backwards
compatibility.
All existing users of the two variables are moved to the new ones but
external code might still run into the overflow.
This also slightly breaks API as external code updating the variables
will have no effect anymore but the only known user of this is
pulsesink.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/6740>
If the muxer times out because of the latency deadline it can happen
that some pads have no caps yet. In that case skip creation of streams
for these pads and create updated section tables once the first buffer
arrives later.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/6823>
This makes sure that for sparse streams (KLV, DVB subtitles, ...) the
muxer does not wait until the next buffer is available for them but
times out on the latency deadline and outputs data.
For non-live pipelines it will still be necessary for upstream to
correctly produce gap events for sparse streams.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/6823>
* gst_ffmpegviddec_frame() is the only caller of gst_ffmpegviddec_video_frame()
and has the same signature. Just move the checks into a single function and
use that.
* Make it clear which frames are the input and output ones in
gst_ffmpegviddec_video_frame() to make issues like the one fixed in the previous
commit more obvious.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/6851>
The FORCE_KEYFRAME frame which has GST_VIDEO_CODEC_FRAME_FLAG_FORCE_KEYFRAME
bit set should be the sync point. So we should let it be an IDR frame to begin
a new GOP, rather than just promote it to an I frame.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/6619>
The FORCE_KEYFRAME frame which has GST_VIDEO_CODEC_FRAME_FLAG_FORCE_KEYFRAME
bit set should be the sync point. So we should let it be an IDR frame to begin
a new GOP, rather than just promote it to an I frame.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/6619>
__STDC_NO_ATOMICS doesn't seem to exist. In fact the only compiler
I've found that sets any of those is msvc, but it sets
__STDC_NO_ATOMICS__, not __STDC_NO_ATOMICS.
__STDC_NO_ATOMICS__ is the one documented by C11 standard.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/6848>
This fixes the code regarding dropping "ghost frames", that is to say input
frames which ended up not producing any decoded frame.
The iteration itself makes sense.. but it was stopping at the "input" frame and
not the decoded frame we just got back.
When dealing with I-frame codecs, ffmpeg will decode frames in separate frames,
so there is no guarantee that they are decoding in order.
Fixes playback issues with such codecs
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/6842>
Set the P and B frame qp to I frame value to avoid generating delta
QP between different frame types. For ICQ and QVBR modes, we can
only set the qpi value, so the qpp and qpb values should be set to
the same value as the qpi.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/6841>
Address below message reported by SDK debug layer.
ID3D12Device::CheckFeatureSupport: Unsupported Decode Profile Specified.
Use ID3D12VideoDevice::CheckFeatureSupport with D3D12_FEATURE_VIDEO_DECODE_PROFILES
to retrieve a list of supported profiles
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/6839>
There's nothing requiring <= 64 channels except for getting the reorder
map and creating a channel mixing matrix, but those won't be possible to
call anyway as channel positions can only express up to 64 channels.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/6819>
Certain V4L2 fourccs don't (yet) have DRM counter parts, in which case
we can't create DMA_DRM caps for them. This is usually the case for
specific tilings, which are represented as modifiers for DMA formats.
While using these tilings is generally preferable - because of e.g.
lower memory usage - it can result in additional conversion steps when
interacting with DMA based APIs such as GL, Vulkan or KMS. In such cases
using a DMA compatible format usually ends up being the better option.
Before the addition of DMA_DRM caps, this was what playbin3 ended up
requesting in various cases - e.g. prefering NV12 over NV12_4L4 - but
the addition of DMA_DRM caps seems to confuse the selection logic.
As a simple and quite robust solution, assume that peers supporting
DMA_DRM caps always prefer these and reorder the caps accordingly.
In the future we plan to have a translation layer for cases where
there is a matching fourcc+modifier pair for a V4L2 fourcc, ensuring
optimal results.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/6645>
Doing it in gst_play_new() means that bindings that directly call
g_object_new() with the GType wouldn't end up initializing both.
This affects at least the Python and GJS bindings.
gst_init() is nonetheless only called from gst_play_new() once because
calling it from class_init would likely lead to problems as that's
called from somewhere in the middle of GObject.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/6801>
Don't add an extra ref if non-floating as that ref will never be
unreffed.
gst_bin_add() is transfer floating (alias to transfer none).
Fixes a leak when a non-floating ref was provided as a return value in
the request-aux-sender signal.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/6807>
