design: add RTP design docs from -good

Merged them all into one.

markdown/design/rtp.md

@@ -0,0 +1,262 @@
+# RTP
+
+These design docs detail some of the lower-level mechanism of certain parts
+of GStreamer's RTP stack. For a higher-level overview see the [RTP and RTSP
+support](rtp.md) section.
+
+# RTP auxiliary stream design
+
+## Auxiliary elements
+
+There are two kind of auxiliary elements, sender and receiver. Let's
+call them rtpauxsend and rtpauxreceive.
+
+rtpauxsend has always one sink pad and can have unlimited requested src
+pads. If only src pad then it works in SSRC-multiplexed mode, if several
+src pads then it works in session multiplexed mode.
+
+rtpauxreceive has always one ssrc pad and can have unlimited requested
+sink pads. If only one sink pad then it works in SSRC-multiplexed mode,
+if several sink pads then it works in session multiplexed mode.
+
+## Rtpbin and auxiliary elements
+
+### Basic mechanism
+
+rtpbin knows for which session ids the given auxiliary element belong
+to. It's done through "set-aux-send", for rtpauxsend kind, and through
+"set-aux-receive" for rtpauxreceive kind. You can call those signals as
+much as needed for each auxiliary element. So for aux elements that work
+in SSRC-multiplexed mode this signal action is called only one time.
+
+The user has to call those action signals before to request the
+differents rtpbin pads. rtpbin is in charge to link those auxiliary
+elements with the sessions, and on receiver side, rtpbin has also to
+handle the link with ssrcdemux.
+
+rtpbin never knows if the given rtpauxsend is actually a rtprtxsend
+element or another aux element. rtpbin never knows if the given
+rtpauxreceive is actually a rtprtxreceive element or another aux
+element. rtpbin has to be kept generic so that more aux elements can be
+added later without changing rtpbin.
+
+It's currently not possible to use rtpbin with auxiliary stream from
+gst-launch. We can discuss about having the ability for rtpbin to
+instanciate itself the special aux elements rtprtxsend and rtprtxreceive
+but they need to be configured ("payload-type" and "payload-types"
+properties) to make retransmission work. So having several rtprtxsend
+and rtprtxreceive in a rtpbin would require a lot of properties to
+manage them form rtpbin. And for each auxiliary elements.
+
+If you want to use rtprtxreceive and rtprtpsend from gst-launch you have
+to use rtpsession, ssrcdemux and rtpjitterbuffer elements yourself. See
+gtk-doc of rtprtxreceive for an example.
+
+### Requesting the rtpbin's pads on the pipeline receiver side
+
+If rtpauxreceive is set for session, i, j, k then it has to call
+rtpbin::"set-aux-receive" 3 times giving those ids and this aux element.
+It has to be done before requesting the recv\_rtp\_sink\_i,
+recv\_rtp\_sink\_j, recv\_rtp\_sink\_k. For a concrete case
+rtprtxreceive, if the user wants it for session i, then it has to call
+rtpbin::"set-aux-receive" one time giving i and this aux element. Then
+the user can request recv\_rtp\_sink\_i pad.
+
+Calling rtpbin::"set-aux-receive" does not create the session. It add
+the given session id and aux element to a hashtable(key:session id,
+value: aux element). Then when the user ask for
+rtpbin.recv\_rtp\_sink\_i, rtpbin lookup if there is an aux element for
+this i session id. If yes it requests a sink pad to this aux element and
+links it with the recv\_rtp\_src pad of the new gstrtpsession. rtpbin
+also checks that this aux element is connected only one time to
+ssrcdemux. Because rtpauxreceive has only one source pad. Each call to
+request rtpbin.recv\_rtp\_sink\_k will also creates
+rtpbin.recv\_rtp\_src\_k\_ssrc\_pt as usual. So that the user have it
+when then it requests rtpbin. (from gst-launch) or using
+on\_rtpbinreceive\_pad\_added callback from an application.
+
+### Requesting the rtpbin's pads on the pipeline sender side
+
+For the sender this is similar but a bit more complicated to implement.
+When the user asks for rtpbin.send\_rtp\_sink\_i, rtpbin will lookup in
+its second map (key:session id, value: aux send element). If there is
+one aux element, then it will set the sink pad of this aux sender
+element to be the ghost pad rtpbin.send\_rtp\_sink\_i that the user
+asked. rtpbin will also request a src pad of this aux element to connect
+it to gstrtpsession\_i. It will automatically create
+rtpbin.send\_rtp\_src\_i the usuall way. Then if the user asks
+rtpbin.send\_rtp\_src\_k, then rtpbin will also lookup in that map and
+request another source pad of the aux element and connect it to the new
+gstrtpsession\_k.
+
+# RTP collision design
+
+## GstRTPCollision
+
+Custon upstream event which contains the ssrc marked as collided.
+
+This event is generated on both pipeline sender and receiver side by the
+gstrtpsession element when it detects a conflict between ssrc. (same
+session id and same ssrc)
+
+It's an upstream event so that means this event is for now only useful
+on pipeline sender side. Because elements generating packets with the
+collided SSRC are placed upstream from the gstrtpsession.
+
+## rtppayloader
+
+When handling a GstRTPCollision event, the rtppayloader has to choose
+another ssrc.
+
+## BYE only the corresponding source, not the whole session.
+
+When a collision happens for the given ssrc, the associated source is
+marked bye. But we make sure that the whole session is not itself set
+bye. Because internally, gstrtpsession can manages several sources and
+all have their own distinct ssrc.
+
+# RTP retransmission design
+
+## GstRTPRetransmissionRequest
+
+Custom upstream event which mainly contains the ssrc and the seqnum of
+the packet which is asked to be retransmisted.
+
+On the pipeline receiver side this event is generated by the
+gstrtpjitterbuffer element. Then it is translated to a NACK to be sent
+over the network.
+
+On the pipeline sender side, this event is generated by the
+gstrtpsession element when it receives a NACK from the network.
+
+## rtprtxsend element
+
+### Basic mechanism
+
+rtprtxsend keeps a history of rtp packets that it has already sent. When
+it receives the event GstRTPRetransmissionRequest from the downstream
+gstrtpsession element, it loopkup the requested seqnum in its stored
+packets. If the packet is present in its history, it will create a RTX
+packet according to RFC 4588. Then this rtx packet is pushed to its src
+pad as other packets.
+
+rtprtxsend works in SSRC-multiplexed mode, so it has one always sink and
+src pad.
+
+### Building retransmission packet fron original packet
+
+A rtx packet is mostly the same as an orignal packet, except it has its
+own ssrc and its own seqnum. That's why rtprtxsend works in
+SSRC-multiplexed mode. It also means that the same session is used.
+Another difference between rtx packet and its original is that it
+inserts the original seqnum (OSN: 2 bytes) at the beginning of the
+payload. Also rtprtxsend builds rtx packet without padding, to let other
+elements do that. The last difference is the payload type. For now the
+user has to set it through the rtx-payload-type property. Later it will
+be automatically retreive this information from SDP. See fmtp field as
+specifies in the RPC4588 (a=fmtp:99 apt=98) fmtp is the payload type of
+the retransmission stream and apt the payload type of its associated
+master stream.
+
+### Retransmission ssrc and seqnum
+
+To choose rtx\_ssrc it randomly selects a number between 0 and 2^32-1
+until it is different than master\_ssrc. rtx\_seqnum is randomly
+selected between 0 and 2^16-1
+
+### Deeper in the stored buffer history
+
+For the history it uses a GSequence with 2^15-1 as its maximum size.
+Which is resonable as the default value is 100. It contains the packets
+in reverse order they have been sent (head:newest, tail:oldest)
+GSequence allows to add and remove an element in constant time (like a
+queue). Also GSequence allows to do a binary search when rtprtxsend
+lookup in its history. It's important if it receives a lot of requests
+or if the history is large.
+
+### Pending rtx packets
+
+When looking up in its history, if seqnum is found then it pushes the
+buffer into a GQueue to its tail. Before to send the current master
+stream packet, rtprtxsend sends all the buffers which are in this
+GQueue. Taking care of converting them to rtx packets. This way, rtx
+packets are sent in the same order they have been requested.
+(g\_list\_foreach traverse the queue from head to tail) The GQueue is
+cleared between sending 2 master stream packets. So for this GQueue to
+contain more than one element, it means that rtprtxsend receives more
+than one rtx request between sending 2 master packets.
+
+### Collision
+
+When handling a GstRTPCollision event, if the ssrc is its rtx ssrc then
+rtprtxsend clear its history and its pending retransmission queue. Then
+it chooses a rtx\_ssrc until it's different than master ssrc. If the
+GstRTPCollision event does not contain its rtx ssrc, for example its
+master ssrc or other, then it just forwards the event to upstream. So
+that it can be handled by the rtppayloader.
+
+## Rtprtxreceive element
+
+### Basic mechanism
+
+The same rtprtxreceive instance can receive several master streams and
+several retransmission streams. So it will try to dynamically associate
+a rtx ssrc with its master ssrc. So that it can reconstruct the original
+from the proper rtx packet.
+
+The algorithm is based on the fact that seqnums of different streams
+(considering all master and all rtx streams) evolve at a different rate.
+It means that the initial seqnum is random for each one and the offset
+could also be different. So that they are statistically all different at
+a given time. If bad luck then the association is delayed to the next
+rtx request.
+
+The algorithm also needs to know if a given packet is a rtx packet or
+not. To know this information there is the rtx-payload-types property.
+For now the user as to configure it but later it will be automatically
+retreive this information from SDP. It needs to know if the current
+packet is rtx or not in order to know if it can extract the OSN from the
+payload. Otherwise it would extract the OSN even on master streams which
+means nothing and so it could do bad things. In theory maybe it could
+work but we have this information in SDP so why not using it to avoid
+bad associations.
+
+Note that it also means that several master streams can have the same
+payload type. And also several rtx streams can have the same payload
+type. So the information from SDP which gives us which rtx payload type
+belong to a give master payload type is not enough to do the association
+between rtx ssrc and master ssrc.
+
+rtprtxreceive works in SSRC-multiplexed mode, so it has one always sink
+and src pad.
+
+### Deeper in the association algorithm
+
+When it receives a GstRTPRetransmissionRequest event it will remember
+the ssrc and the seqnum from this request.
+
+On incoming packets, if the packet has its ssrc already associated then
+it knows if the ssrc is an rtx ssrc or a master stream ssrc. If this is
+a rtx packet then it recontructs the original and pushs the result to
+src pad as if it was a master packet.
+
+If the ssrc is not yet associated rtprtxreceive checks the payload type.
+if the packet has its payload type marked as rtx then it will extract
+the OSN (original seqnum number) and lookup in its stored requests if a
+seqnum matchs. If found, then it associates the current ssrc to the
+master ssrc marked in the request. If not found it just drops the
+packet. Then it removes the request from the stored requests.
+
+If there are 2 requests with the same seqnum and different ssrc, then
+the couple seqnum,ssrc is removed from the stored requests. A stored
+request actually means that actually the couple seqnum,ssrc is stored.
+If it's happens the request is droped but it avoids to do bad
+associations. In this case the association is just delayed to the next
+request.
+
+### Building original packet from rtx packet
+
+Header, extensions, payload and padding are mostly the same. Except that
+the OSN is removed from the payload. Then ssrc, seqnum, and original
+payload type are correctly set. Original payload type is actually also
+stored when the rtx request is handled.

sitemap.txt

@@ -189,6 +189,7 @@ index.md
 		design/qos.md
 		design/query.md
 		design/relations.md
+		design/rtp.md
 		design/scheduling.md
 		design/seeking.md
 		design/segments.md