As we override the GLib item with our own structure, we cannot use any
function from GList or GQueue that would try to free the RTPJitterBufferItem.
In this patch, we move away from g_queue_new() which forces using
g_queue_free(). This this function could use g_slice_free() if there is any items
left in the queue. Passing the wrong size to GSLice may cause data corruption
and crash.
A better approach would be to use a proper intrusive linked list
implementation but that's left as an exercise for the next person
running into crashes caused by this.
Be ware that this regression was introduced 6 years ago in the following
commit [0], the call to flush() looked useless, as there was a g_queue_free()
afterward.
Signed-off-by: Nicolas Dufresne <nicolas.dufresne@collabora.com>
[0] 479c7642fd
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-good/-/merge_requests/573>
This moves the RtpJitterBufferStructure type, alloc, free into
rtpjitterbuffer.c/h implementation. jitterbuffer.c strictly rely on
the fact this structure is compatible with GList, and so it make more
sense to keep encapsulate it. Also, anything that could possibly
reduce the amount of code in the element is a win.
In order to support that move, a function pointer to free the data
was added. This also allow making the free function option when
flushing the jitterbuffer.
In the case of reordered packets, calculating skew would cause
pts values to be off. Only calculate skew when packets come
in as expected. Also, late RTX packets should not trigger
clock skew adjustments.
Fixes#612
If it goes over 2^15 packets, it will think it has rolled over
and start dropping all packets. So make sure the seqnum distance is not too big.
But let's not limit it to a number that is too small to avoid emptying it
needlessly if there is a spurious huge sequence number, let's allow at
least 10k packets in any case.
When set this property will allow the jitterbuffer to start delivering
packets as soon as N most recent packets have consecutive seqnum. A
faststart-min-packets of zero disables this feature. This heuristic is
also used in rtpsource which implements the probation mechanism and a
similar heuristic is used to handle long gaps.
https://bugzilla.gnome.org/show_bug.cgi?id=769536
In function rtp_jitter_buffer_calculate_pts: If gap in incoming RTP
timestamps is more than (3 * jbuf->clock_rate) we call
rtp_jitter_buffer_reset_skew which resets pts to 0. So components down
the pipeline (playes, mixers) just skip frames/samples until pts becomes
equal to pts before gap.
In version 1.10.2 and before this checking was bypassed for packets with
"estimated dts", and gaps were handled correctly.
https://bugzilla.gnome.org/show_bug.cgi?id=778341
The lost-event was using a different time-domain (dts) than the outgoing
buffers (pts). Given certain network-conditions these two would become
sufficiently different and the lost-event contained timestamp/duration
that was really wrong. As an example GstAudioDecoder could produce
a stream that jumps back and forth in time after receiving a lost-event.
The previous behavior calculated the pts (based on the rtptime) inside the
rtp_jitter_buffer_insert function, but now this functionality has been
refactored into a new function rtp_jitter_buffer_calculate_pts that is
called much earlier in the _chain function to make pts available to
various calculations that wrongly used dts previously
(like the lost-event).
There are however two calculations where using dts is the right thing to
do: calculating the receive-jitter and the rtx-round-trip-time, where the
arrival time of the buffer from the network is the right metric
(and is what dts in fact is today).
The patch also adds two tests regarding B-frames or the
“rtptime-going-backwards”-scenario, as there were some concerns that this
patch might break this behavior (which the tests shows it does not).
Add a need-resync state, this is when we need to try to lock on to a
time/RTPtime pair.
Always check the RTP timestamps and if they go backwards, mark ourselves
as need-resync.
Only resync when need-resync is TRUE and we have a valid time. Otherwise
we keep the old values. This avoids locking on to an invalid time and
causing us to timestamp everything with -1.
Fixes https://bugzilla.gnome.org/show_bug.cgi?id=730417
Rework the packet queue so that the most common action (insert a packet
at the tail of the queue) goes very fast.
Report if a packet was inserted at the head instead of the tail so that
we can know when to retry _pop or _peek.
Make a new method to disable the jitterbuffer buffering.
Rework the update_estimated_eos() method. Calculate how much time
there is left to play. If we have less than the delay of the
jitterbuffer, we disabled buffering because we might never be able to
fill the complete jitterbuffer again.
If we receive an EOS event, disable buffering. We will drain the
buffer and eventually push the EOS event out.
When we reach the estimated NPT timeout and we didn't receive an EOS
event, make one and queue it so that it can be pushed.
Fixes https://bugzilla.gnome.org/show_bug.cgi?id=728017
Add a new timestamp mode that assumes the local and remote clock are
synchronized. It takes the first timestamp as a base time and then uses the RTP
timestamps for the output PTS.
If we were not waiting for the missing seqnum when we insert the lost packet
event in the jitterbuffer, we end up not updating the next_seqnum and wait
forever for the lost packets to arrive. Instead, keep track of the amount of
packets contained by the jitterbuffer item and update the next expected
seqnum only after pushing the buffer/event. This makes sure we correctly handle
GAPS in the sequence numbers.
Make the jitterbuffer operate on a structure containing all the packet
information. This avoids mapping the buffer multiple times just to get the RTP
information. It will also make it possible to store other miniobjects such as
events later.
When deactivating jitterbuffers when the buffering starts, keep the current
percent of the jitterbuffer and also set the jitterbuffer in the buffering state
so that we know when it's filled again.
Add property to get the buffering percentage of the jitterbuffer.
Add signal to pause the jitterbuffer. This will be emitted from gstrtpbin when
one of the jitterbuffers is buffering.
Make rtpbin collect the buffering messages and post a new buffering message with
the min value.
Remove the stats callback from jitterbuffer but pass a percent integer to
functions that affect the buffering state of the jitterbuffer. This allows us
then to post buffering messages from outside of the jitterbuffer lock.
Add callback for buffering stats.
Configure the latency in the jitterbuffer instead of passing it with _insert.
Calculate buffering levels when pushing and popping
Post buffering messages.
When we construct a timestamp that would result in a timestamp that is earlier
than when the packet was received, reset the skew calculation as this is
probably a sign that the sender restarted or paused.
Fixes#593354
Original commit message from CVS:
Patch by: Arnout Vandecappelle <arnout at mind dot be>
* gst/rtpmanager/rtpjitterbuffer.c: (rtp_jitter_buffer_reset_skew),
(calculate_skew):
* gst/rtpmanager/rtpjitterbuffer.h:
Keep track of the last outgoing timestamp and of the last sender-side
time. Timestamps can only go forward if they do at the sender
side, can only go back if they do at the sender side, and remain the
same if they remain the same at the sender side. Fixes#565319.
Original commit message from CVS:
* gst/rtpmanager/gstrtpbin.c: (gst_rtp_bin_associate),
(gst_rtp_bin_handle_sync), (create_stream), (free_stream),
(new_ssrc_pad_found):
Remove internal sync pad, use signals instead to get lip-sync
notifications.
* gst/rtpmanager/gstrtpjitterbuffer.c:
(gst_rtp_jitter_buffer_base_init),
(gst_rtp_jitter_buffer_class_init),
(gst_rtp_jitter_buffer_internal_links), (create_rtcp_sink),
(remove_rtcp_sink), (gst_rtp_jitter_buffer_request_new_pad),
(gst_rtp_jitter_buffer_release_pad),
(gst_rtp_jitter_buffer_sink_rtcp_event),
(gst_rtp_jitter_buffer_chain_rtcp),
(gst_rtp_jitter_buffer_get_property):
* gst/rtpmanager/gstrtpjitterbuffer.h:
Make it possible to send SR packets to the jitterbuffer.
Check if the SR timestamps are valid by comparing them to the RTP
timestamps.
Signal the SR packet and the timing information to listeners.
* gst/rtpmanager/gstrtpssrcdemux.c: (create_demux_pad_for_ssrc),
(gst_rtp_ssrc_demux_rtcp_chain), (gst_rtp_ssrc_demux_src_query):
Remove some unused code.
* gst/rtpmanager/rtpjitterbuffer.c: (rtp_jitter_buffer_reset_skew),
(calculate_skew), (rtp_jitter_buffer_get_sync):
* gst/rtpmanager/rtpjitterbuffer.h:
Keep track of the last seen RTP timestamp so that we can filter out
invalid SR packets.
Original commit message from CVS:
* gst/rtpmanager/gstrtpbin.c: (gst_rtp_bin_associate),
(gst_rtp_bin_sync_chain), (create_stream), (new_ssrc_pad_found):
Do not try to keep track of the clock-rate ourselves but simply get the
value from the jitterbuffer.
* gst/rtpmanager/gstrtpjitterbuffer.c:
(gst_jitter_buffer_sink_parse_caps), (gst_rtp_jitter_buffer_chain),
(gst_rtp_jitter_buffer_get_sync):
* gst/rtpmanager/gstrtpjitterbuffer.h:
Add some debug info.
Pass the clock-rate to the jitterbuffer.
Also pass the clock-rate along with the rtp timestamp when getting the
sync parameters.
* gst/rtpmanager/gstrtpptdemux.c: (gst_rtp_pt_demux_chain):
Fix some debug.
* gst/rtpmanager/rtpjitterbuffer.c: (rtp_jitter_buffer_reset_skew),
(calculate_skew), (rtp_jitter_buffer_get_sync):
* gst/rtpmanager/rtpjitterbuffer.h:
Keep track of clock-rate changes and return the clock-rate together with
the rtp timestamps used for sync.
Don't try to construct timestamps when we have no base_time.
* gst/rtpmanager/rtpsource.c: (get_clock_rate):
Request a new clock-rate when the payload type changes.
Reset the jitter calculation when the clock-rate changes.
Original commit message from CVS:
* gst/rtpmanager/gstrtpbin.c: (on_sender_timeout),
(create_session), (gst_rtp_bin_associate),
(gst_rtp_bin_sync_chain), (gst_rtp_bin_class_init),
(gst_rtp_bin_request_new_pad):
* gst/rtpmanager/gstrtpbin.h:
Add signal to notify listeners when a sender becomes a receiver.
Tweak lip-sync code, don't store our own copy of the ts-offset of the
jitterbuffer, don't adjust sync if the change is less than 4msec.
Get the RTP timestamp <-> GStreamer timestamp relation directly from
the jitterbuffer instead of our inaccurate version from the source.
* gst/rtpmanager/gstrtpjitterbuffer.c:
(gst_rtp_jitter_buffer_chain), (gst_rtp_jitter_buffer_loop),
(gst_rtp_jitter_buffer_get_sync):
* gst/rtpmanager/gstrtpjitterbuffer.h:
Add G_LIKELY macros, use global defines for max packet reorder and
dropouts.
Reset the jitterbuffer clock skew detection when packets seqnums are
changed unexpectedly.
* gst/rtpmanager/gstrtpsession.c: (on_sender_timeout),
(gst_rtp_session_class_init), (gst_rtp_session_init):
* gst/rtpmanager/gstrtpsession.h:
Add sender timeout signal.
* gst/rtpmanager/rtpjitterbuffer.c: (rtp_jitter_buffer_reset_skew),
(calculate_skew), (rtp_jitter_buffer_insert),
(rtp_jitter_buffer_get_sync):
* gst/rtpmanager/rtpjitterbuffer.h:
Add some G_LIKELY macros.
Keep track of the extended RTP timestamp so that we can report the RTP
timestamp <-> GStreamer timestamp relation for lip-sync.
Remove server timestamp gap detection code, the server can sometimes
make a huge gap in timestamps (talk spurts,...) see #549774.
Detect timetamp weirdness instead by observing the sender/receiver
timestamp relation and resync if it changes more than 1 second.
Add method to report about the current rtp <-> gst timestamp relation
which is needed for lip-sync.
* gst/rtpmanager/rtpsession.c: (rtp_session_class_init),
(on_sender_timeout), (check_collision), (rtp_session_process_sr),
(session_cleanup):
* gst/rtpmanager/rtpsession.h:
Add sender timeout signal.
Remove inaccurate rtp <-> gst timestamp relation code, the
jitterbuffer can now do an accurate reporting about this.
* gst/rtpmanager/rtpsource.c: (rtp_source_init),
(rtp_source_update_caps), (calculate_jitter),
(rtp_source_process_rtp):
* gst/rtpmanager/rtpsource.h:
Remove inaccurate rtp <-> gst timestamp relation code.
* gst/rtpmanager/rtpstats.h:
Define global max-reorder and max-dropout constants for use in various
subsystems.
Original commit message from CVS:
Patch by: Olivier Crete <tester@tester.ca>
* gst/rtpmanager/gstrtpjitterbuffer.c:
(gst_jitter_buffer_sink_parse_caps), (gst_rtp_jitter_buffer_chain):
* gst/rtpmanager/rtpjitterbuffer.c: (calculate_skew),
(rtp_jitter_buffer_insert):
* gst/rtpmanager/rtpjitterbuffer.h:
Remove the fixed clock-rate from the jitterbuffer and extend it so that
a clock-rate can be provided with each buffer instead. Fixes#511686.
Original commit message from CVS:
* gst/rtpmanager/gstrtpjitterbuffer.c:
(gst_rtp_jitter_buffer_chain), (gst_rtp_jitter_buffer_loop):
Only peek at the tail element instead of popping it off, which allows
us to greatly simplify things when the tail element changes.
* gst/rtpmanager/gstrtpsession.c:
(gst_rtp_session_event_recv_rtp_sink):
* gst/rtpmanager/gstrtpssrcdemux.c:
(gst_rtp_ssrc_demux_sink_event):
Forward FLUSH events instead of leaking them.
* gst/rtpmanager/rtpjitterbuffer.c: (rtp_jitter_buffer_reset_skew),
(calculate_skew), (rtp_jitter_buffer_insert):
* gst/rtpmanager/rtpjitterbuffer.h:
Remove the tail-changed callback in favour of a simple boolean when we
insert a buffer in the queue.
Add method to peek the tail of the buffer.
Original commit message from CVS:
* gst/rtpmanager/gstrtpjitterbuffer.c:
(gst_rtp_jitter_buffer_flush_start),
(gst_rtp_jitter_buffer_flush_stop),
(gst_rtp_jitter_buffer_change_state), (apply_offset),
(gst_rtp_jitter_buffer_loop):
Remove some old unused variables.
Don't add the latency to the skew corrected timestamp, latency is only
used to sync against the clock.
Improve debugging.
* gst/rtpmanager/rtpjitterbuffer.c: (rtp_jitter_buffer_init),
(rtp_jitter_buffer_reset_skew), (calculate_skew):
* gst/rtpmanager/rtpjitterbuffer.h:
Handle case where server timestamp goes backwards or wildly jumps by
temporarily pausing the skew correction.
Improve debugging.
Original commit message from CVS:
* gst/rtpmanager/gstrtpjitterbuffer.c: (apply_latency),
(gst_rtp_jitter_buffer_loop), (gst_rtp_jitter_buffer_query):
Remove jitter correction code, it's now in the lower level object.
Use new -core method for doing a peer query.
* gst/rtpmanager/rtpjitterbuffer.c: (rtp_jitter_buffer_init),
(calculate_skew), (rtp_jitter_buffer_insert):
* gst/rtpmanager/rtpjitterbuffer.h:
Move jitter correction to the lowlevel jitterbuffer.
Increase the max window size.
When filling the window, already start estimating the skew using a
parabolic weighting factor so that we have a much better startup
behaviour that gets more accurate with the more samples we have.
Increase the default weighting factor for the steady state to get
smoother timestamps.
Original commit message from CVS:
* gst/rtpmanager/gstrtpbin.c: (gst_rtp_bin_dispose),
(gst_rtp_bin_finalize):
Fix cleanup crasher.
* gst/rtpmanager/rtpjitterbuffer.c: (rtp_jitter_buffer_init),
(calculate_skew):
* gst/rtpmanager/rtpjitterbuffer.h:
Dynamically adjust the skew calculation window so that we calculate it
over a period of around 2 seconds.
Original commit message from CVS:
* gst/rtpmanager/gstrtpbin.c: (gst_rtp_bin_set_property),
(gst_rtp_bin_get_property):
Use lock to protect variable.
* gst/rtpmanager/gstrtpjitterbuffer.c:
(gst_rtp_jitter_buffer_class_init),
(gst_jitter_buffer_sink_parse_caps), (gst_rtp_jitter_buffer_chain),
(convert_rtptime_to_gsttime), (gst_rtp_jitter_buffer_loop):
Reconstruct GST timestamp from RTP timestamps based on measured clock
skew and sync offset.
* gst/rtpmanager/rtpjitterbuffer.c: (rtp_jitter_buffer_init),
(rtp_jitter_buffer_set_tail_changed),
(rtp_jitter_buffer_set_clock_rate),
(rtp_jitter_buffer_get_clock_rate), (calculate_skew),
(rtp_jitter_buffer_insert), (rtp_jitter_buffer_peek):
* gst/rtpmanager/rtpjitterbuffer.h:
Measure clock skew.
Add callback to be notfied when a new packet was inserted at the tail.
* gst/rtpmanager/rtpsource.c: (rtp_source_init),
(calculate_jitter), (rtp_source_send_rtp):
* gst/rtpmanager/rtpsource.h:
Remove clock skew detection, it's move to the jitterbuffer now.
