gstreamer/gst/mpegtsdemux/TODO

tsdemux/tsparse TODO
--------------------

* clock for live streams
  In order for playback to happen at the same rate as on the producer,
  we need to estimate the remote clock based on capture time and PCR
  values.
  For this estimation to be as accurate as possible, the capture time
  needs to happen on the sources.
    => Ensure live sources actually timestamp their buffers
  Once we have accurate timestamps, we can use an algorithm to
  calculate the PCR/local-clock skew.
    => Use the EPTLA algorithm as used in -good/rtp/rtpmanager/
     gstrtpjitterbuffer

* Seeking
  => Split out in a separate file/object. It is polluting tsdemux for
  code readability/clarity.

* Perfomance : Creation/Destruction of buffers is slow
  * => This is due to g_type_instance_create using a dogslow rwlock
  which take up to 50% of gst_adapter_take_buffer()
  => Bugzilla #585375 (performance and contention problems)

* mpegtspacketizer
  * offset/timestamp of incoming buffers need to be carried on to the
  sub-buffers in order for several demuxer features to work correctly.

* mpegtsparser
  * SERIOUS room for improvement performance-wise (see callgrind)




Synchronization, Scheduling and Timestamping
--------------------------------------------

  A mpeg-ts demuxer can be used in a variety of situations:
  * lives streaming over DVB, UDP, RTP,..
  * play-as-you-download like HTTP Live Streaming or UPNP/DLNA
  * random-access local playback, file, Bluray, ...

  Those use-cases can be categorized in 3 different categories:
  * Push-based scheduling with live sources [0]
  * Push-based scheduling with non-live sources
  * Pull-based scheduling with fast random-access

  Due to the nature of timing within the mpeg-ts format, we need to
pay extra attention to the outgoing NEWSEGMENT event and buffer
timestamps in order to guarantee proper playback and synchronization
of the stream.


 1) Live push-based scheduling

  The NEWSEGMENT event will be in time format and is forwarded as is,
  and the values are cached locally.

  Since the clock is running when the upstream buffers are captured,
  the outgoing buffer timestamps need to correspond to the incoming
  buffer timestamp values.

    => A delta, DTS_delta between incoming buffer timestamp and
       DTS/PTS needs to be computed.

    => The outgoing buffers will be timestamped with their PTS values
       (overflow corrected) offseted by that initial DTS_delta.

  A latency is introduced between the time the buffer containing the
  first bit of a Access Unit is received in the demuxer and the moment
  the demuxer pushed out the buffer corresponding to that Access Unit.

    => That latency needs to be reported. It corresponds to the
       biggest Access Unit spacing, in this case 1/video-framerate.

  According to the ISO/IEC 13818-1:2007 specifications, D.0.1 Timing
  mode, the "coded audio and video that represent sound and pictures
  that are to be presented simultaneously may be separated in time
  within the coded bit stream by ==>as much as one second<=="

    => The demuxer will therefore report an added latency of 1s to
       handle this interleave.


 2) Non-live push-based scheduling

  If the upstream NEWSEGMENT is in time format, the NEWSEGMENT event
  is forwarded as is, and the values are cached locally.

  If upstream does provide a NEWSEGMENT in another format, we need to
  compute one by taking the default values:
    start : 0
    stop  : GST_CLOCK_TIME_NONE
    time  : 0

  Since no prerolling is happening downstream and the incoming buffers
  do not have capture timestamps, we need to ensure the first buffer
  we push out corresponds to the base segment start runing time.

    => A delta between the first DTS to output and the segment start
       position needs to be computed.

    => The outgoing buffers will be timestamped with their PTS values
       (overflow corrected) offseted by that initial delta.

  Latency is reported just as with the live use-case.


 3) Random access pull-mode

  We do not get a NEWSEGMENT event from upstream, we therefore need to
  compute the outgoing values.

  The base stream/running time corresponds to the DTS of the first
  buffer we will output. The DTS_delta becomes that earliest DTS.

  =>  FILLME

 X) General notes

  It is assumed that PTS/DTS rollovers are detected and corrected such
  as the outgoing timestamps never rollover. This can be easily
  handled by correcting the DTS_delta when such rollovers are
  detected. The maximum value of a GstClockTimeDiff is almost 3
  centuries, we therefore have enough margin to handle a decent number
  of rollovers.

  The generic equation for calculating outgoing buffer timestamps
  therefore becomes:

    D   = DTS_delta, with rollover corrections
    PTS = PTS of the buffer we are going to push out
    TS  = Timestamp of the outgoing buffer

    ==>   TS = PTS + D

  If seeking is handled upstream for push-based cases, whether live or
  not, no extra modification is required.

  If seeking is handled by the demuxer in the non-live push-based
  cases (converting from TIME to BYTES), the demuxer will need to
  set the segment start/time values to the requested seek position.
  The DTS_delta will also have to be recomputed to take into account
  the seek position.


[0] When talking about live sources, we mean this in the GStreamer
definition of live sources, which is to say sources where if we miss
the capture, we will miss the data to be captured. Sources which do
internal buffering (like TCP connections or file descriptors) are
*NOT* live sources.