When syncing by running time, multiqueue will throttle unlinked streams
based on a global "high-time" and the pending "next_time" of a stream.
The idea is that we don't want unlinked streams to be "behind" the global
running time of linked streams, so that if/when they get linked (like when
switching tracks) decoding/playback can resume from the same position as
the other streams.
The problem is that it assumes elements downstream will have a more or less
equal buffering/latency ... which isn't the case for streams of different
type. Video decoders tend to have higher latency (and therefore consume more
from upstream to output a given decoded frame) compared to audio ones, resulting
in the computed "high_time" being at the position of the video stream,
much further than the audio streams.
This means the unlinked audio streams end up being quite a bit after the linked
audio streams, resulting in gaps when switching streams.
In order to mitigate this issue, this patch adds a new "group-id" pad property
which allows users to "group" streams together. Calculating the high-time will
now be done not only globally, but also per group. This ensures that within
a given group unlinked streams will be throttled by that group's high-time
instead.
This fixes gaps when switching downstream elements (like switching audio tracks).
When synchronizing the output by time, there are some use-cases (like
allowing gapless playback downstream) where we want the unlinked streams
to stay slightly behind the linked streams.
The "unlinked-cache-time" property allows the user to specify by how
much time the unlinked streams should wait before pushing again.
Multiqueue should only be used to cope with:
* decoupling upstream and dowstream threading (i.e. having separate threads
for elementary streams).
* Ensuring individual queues have enough space to cope with upstream interleave
(distance in stream time between co-located samples). This is to guarantee
that we have enough room in each individual queues to provide new data in
each, without being blocked.
* Limit the queue sizes to that interleave distance (and an extra minimal
buffering size). This is to ensure we don't consume too much memory.
Based on that, multiqueue now continuously calculates the input interleave
(per incoming streaming thread). Based on that, it calculates a target
interleave (currently 1.5 x real_interleave + 250ms padding).
If the target interleave is greater than the current max_size.time, it will
update it accordingly (to allow enough margin to not block).
If the target interleave goes down by more than 50%, we re-adjust it once
we know we have gone past a safe distance (2 x current max_size.time).
This mode can only be used for incoming streams that are guaranteed to be
properly timestamped.
Furthermore, we ignore sparse streams when calculating interleave and maximum
size of queues.
For the simplest of use-cases (single stream), multiqueue acts as a single
queue with a time limit of 250ms.
If there are multiple inputs, but each come from a different streaming thread,
the maximum time limit will also end up being 250ms.
On regular files (more than one input stream from the same upstream streaming
thread), it can reduce the total memory used as much as 10x, ending up with
max_size.time around 500ms.
Due to the adaptive nature, it can also cope with changing interleave (which
can happen commonly on some files at startup/pre-roll time)
It might cause deadlocks to post messages while holding the multiqueue
lock. To avoid this a new boolean flag is set whenever a new buffering percent
is found. The message is posted after the lock can be released.
To make sure the buffering messages are posted in the right order, messages
are posted holding another lock. This prevents 2 threads trying to post
messages at the same time.
https://bugzilla.gnome.org/show_bug.cgi?id=736295
There's no code that uses it other than multiqueue, so make it private
to multiqueue for now. That way we can also do optimisations that
require API/ABI breaks. If anyone ever wants to use it, we can still
make it public again.
Add support for buffering mode where we post BUFFERING messages based on the
level of the queues. It currently operates on the first queue that goes over or
under the high/low thresholds.
Original commit message from CVS:
* plugins/elements/gstmultiqueue.c:
* plugins/elements/gstmultiqueue.h:
revert extra-size-buffers stuff, caused some race conditions
and extra-size-buffers is not used anymore. Docs needs some updates
Original commit message from CVS:
* plugins/elements/gstmultiqueue.c: (gst_multi_queue_init),
(gst_single_queue_flush), (apply_segment), (apply_buffer),
(gst_single_queue_push_one), (gst_multi_queue_loop),
(gst_multi_queue_sink_activate_push), (gst_multi_queue_sink_event),
(gst_multi_queue_src_activate_push), (wake_up_next_non_linked),
(compute_high_id), (gst_single_queue_new):
* plugins/elements/gstmultiqueue.h:
Take the multiqueue lock when updating the fill level so we don't get
confused.
After applying a buffer or event on the src pad segment, make sure to
call gst_data_queue_limits_changed() to get the data queue to unblock
and check the filled state again.
Rework the not-linked pad handling so the logic is that not-linked
pads can push as fast as they like, but only so they never get
ahead of any linked pads.
* tests/check/elements/multiqueue.c: (mq_sinkpad_to_srcpad),
(mq_dummypad_getcaps), (mq_dummypad_chain), (mq_dummypad_event),
(run_output_order_test), (GST_START_TEST), (multiqueue_suite):
Add a test to check that not-linked pads always stay behind
linked pads.
Original commit message from CVS:
* plugins/elements/Makefile.am:
* plugins/elements/gstmultiqueue.h:
* plugins/elements/gstqueue.h:
These are not installed headers, no need for padding.
