This commit introduces the AVTP Clock Reference Format (CRF) Checker
element. This element re-uses the GstAvtpCrfBase class introduced along
with the CRF Synchronizer element.
This element will typically be used along with the avtpsrc element to
ensure that the AVTP timestamp (and H264 timestamp in case of CVF-H264
packets) is "aligned" with the incoming CRF stream. Here, "aligned" means
that the timestamp value should be within 25% of the period of the media
clock recovered from the CRF stream.
The user can also set an option (drop-invalid) in order to drop any packet
whose timestamp is not within the thresholds of the incoming CRF stream.
This commit introduces the AVTP Clock Reference Format (CRF) Synchronizer
element. This element implements the AVTP CRF Listener as described in IEEE
1722-2016 Section 10.
CRF is useful in synchronizing events within different systems by
distributing a common clock. This is useful in a scenario where there are
multiple talkers who are sending data to a single listener which is
processing that data. E.g. CCTV cameras on a network sending AVTP video
streams to a base station to display on the same screen.
It is assumed that all the systems are already time-synchronized with each
other. So, the AVTP Talker essentially adjusts the AVTP Presentation Time
so it's phase-locked with the reference clock provided by the CRF stream.
There are 2 different roles of systems which participate in CRF data
exchange. A system can either be a CRF Talker, which samples it's own
clock and generates a stream of timestamps to transmit over the network, or
a CRF Listener, the system which receives the generated timestamps and
recovers the media clock from the timestamps. It then adjusts it's own
clock to align with recovered media clock. The timestamps generated by the
talker may not be continuous and the listener might have to interpolate
some timestamps to recover the media clock. The number of timestamps to
interpolate is mentioned in the CRF stream AVTPDU (Refer IEEE 1722-2016
Section 10.4 for AVTPDU structure). Only CRF Listener has been implemented
in this commit.
The CRF Sync element will create a separate thread to listen for the CRF
stream. This thread will calculate and store the average period of the
recovered media clock. The pipeline thread will use this stored period
along with the first timestamp of the latest CRF AVTPDU received to
calculate adjustment for timestamps in the audio/video streams. In case of
CRF AVTPDUs with single timestamp, two consecutive CRF AVTPDUs will be used
to figure out the average period of the recovered media clock.
In case of H264 streams, both AVTP timestamp and H264 timestamp will be
adjusted.
In the future commits, another "CRF Checker" element will be introduced
which will validate the timestamps on the AVTP Listener side. Which is why
a lot of code has been implemented as part of the gstcrfbase class.
The avtpsink element is expected to transmit AVTPDUs at specific times,
according to GstBuffer timestamps. Currently, the transmission time is
controlled in software via the rendering synchronization mechanism
provided by GstBaseSink class. However, that mechanism may not cope with
some AVB use-cases such as Class A streams, where AVTPDUs are expected
to be transmitted at every 125 us. Thus, this patch introduces avtpsink
own mechanism which leverages the socket transmission scheduling
infrastructure introduced in Linux kernel 4.19. When supported by the
NIC, the transmission scheduling is offloaded to the hardware, improving
transmission time accuracy considerably.
To illustrate that, a before-after experiment was carried out. The
experimental setup consisted in 2 PCs with Intel i210 card connected
back-to-back running an up-to-date Archlinux with kernel 5.3.1. In one
host gst-launch-1.0 was used to generate a 2-minute Class A stream while
the other host captured the packets. The metric under evaluation is the
transmission interval and it is measured by checking the 'time_delta'
information from ethernet frames captured at the receiving side.
The table below shows the outcome for a 48 kHz, 16-bit sample, stereo
audio stream. The unit is nanoseconds.
| Mean | Stdev | Min | Max | Range |
-------+--------+---------+---------+---------+---------+
Before | 125000 │ 2401 │ 110056 │ 288432 │ 178376 |
After | 125000 │ 18 │ 124943 │ 125055 │ 112 |
Before this patch, the transmission interval mean is equal to the
optimal value (Class A stream -> 125 us interval), and it is kept the
same after the patch. The dispersion measurements, however, had
improved considerably, meaning the system is now consistently
transmitting AVTPDUs at the correct time.
Finally, the socket transmission scheduling infrastructure requires the
system clock to be synchronized with PTP clock so this patches modifies
the AVTP plugin documentation to cover how to achieve that.
This patch introduces the AVTP Compressed Video Format (CVF) depayloader
specified in IEEE 1722-2016 section 8. Currently, this depayloader only
supports H.264 encapsulation described in section 8.5.
Is also worth noting that only single NAL units are handled: aggregated
and fragmented payloads are not handled.
As stated in AVTP CVF payloader patch, AVTP timestamp is used to define
outgoing buffer DTS, while the H264_TIMESTAMP defines outgoing buffer
PTS.
When an AVTP packet is received, the extracted H.264 NAL unit is added to
a "stash" (the out_buffer) of H.264 NAL units. This "stash" is pushed
downstream as single buffer (with NAL units aggregated according to format
used on GStreamer, based on ISO/IEC 14496-15) as soon as we get the AVTP
packet with M bit set.
This patch groups NAL units using a fixed NAL size lenght, sent downstream
on the `codec_data` capability.
The "stash" of NAL units can be prematurely sent downstream if a
discontinuity (a missing SEQNUM) happens.
This patch reuses the infra provided by gstavtpbasedepayload.c.
This patch introduces the AVTP Compressed Video Format (CVF) payloader
specified in IEEE 1722-2016 section 8. Currently, this payload only
supports H.264 encapsulation described in section 8.5.
Is also worth noting that only single NAL units are encapsulated: no
aggregation or fragmentation is performed by the payloader.
An interesting characteristic of CVF H.264 spec is that it defines an
H264_TIMESTAMP, in addition to the AVTP timestamp. The later is
translated to the GST_BUFFER_DTS while the former is translated to the
GST_BUFFER_PTS. From AVTP CVF H.264 spec, it is clear that the AVTP
timestamp is related to the decoding order, while the H264_TIMESTAMP is
an ancillary information to the H.264 decoder.
Upon receiving a buffer containing a group of NAL units, the avtpcvfpay
element will extract each NAL unit and payload them into individual AVTP
packets. The last AVTP packet generated for a group of NAL units will
have the M bit set, so the depayloader is able to properly regroup them.
The exact format of the buffer of NAL units is described on the
'codec_data' capability, which is parsed by the avtpcvfpay, in the same
way done in rtph264pay.
This patch reuses the infra provided by gstavtpbasepayload.c.
This patch introduces the avtpsrc element which implements a typical
network source. The avtpsrc element receives AVTPDUs encapsulated into
Ethernet frames and push them downstream in the GStreamer pipeline.
Implementation if pretty straightforward since the burden is implemented
by GstPushSrc class.
Likewise the avtpsink element, applications that utilize this element
must have CAP_NET_RAW capability since it is required by Linux to open
sockets from AF_PACKET domain.
This patch introduces the avtpsink elements which implements a typical
network sink. Implementation is pretty straightforward since the burden
is implemented by GstBaseSink class.
The avtpsink element defines three new properties: 1) network interface
from where AVTPDU should be transmitted, 2) destination MAC address
(usually a multicast address), and 3) socket priority (SO_PRIORITY).
Socket setup and teardown are done in start/stop virtual methods while
AVTPDU transmission is carried out by render(). AVTPDUs are encapsulated
into Ethernet frames and transmitted to the network via AF_PACKET socket
domain. Linux requires CAP_NET_RAW capability in order to open an
AF_PACKET socket so the application that utilize this element must have
it. For further info about AF_PACKET socket domain see packet(7).
Finally, AVTPDUs are expected to be transmitted at specific times -
according to the GstBuffer presentation timestamp - so the 'sync'
property from GstBaseSink is set to TRUE by default.
This patch introduces the AAF depayloader element, the counterpart from
the AAF payloader. As expected, this element inputs AVTPDUs and outputs
audio raw data and supports AAF PCM encapsulation only.
The AAF depayloader srcpad produces a fixed format that is encoded
within the AVTPDU. Once the first AVTPDU is received by the element, the
audio features e.g. sample format, rate, number of channels, are decoded
and the srcpad caps are set accordingly. Also, at this point, the
element pushes a SEGMENT event downstream defining the segment according
to the AVTP presentation time.
All AVTP depayloaders will share some common code. For that reason, this
patch introduces the GstAvtpBaseDepayload abstract class that implements
common depayloader functionalities. AAF-specific functionalities are
implemented in the derived class GstAvtpAafDepay.
This patch introduces the AVTP Audio Format (AAF) payloader element from
the AVTP plugin. The element inputs audio raw data and outputs AVTP
packets (aka AVTPDUs), implementing a typical protocol payloader element
from GStreamer.
AAF is one of the available formats to transport audio data in an AVTP
system. AAF is specified in IEEE 1722-2016 section 7 and provides two
encapsulation mode: PCM and AES3. This patch implements PCM
encapsulation mode only.
The AAF payloader working mechanism consists of building the AAF header,
prepending it to the GstBuffer received on the sink pad, and pushing the
buffer downstream. Payloader parameters such as stream ID, maximum
transit time, time uncertainty, and timestamping mode are passed via
element properties. AAF doesn't support all possible sample format and
sampling rate values so the sink pad caps template from the payloader is
a subset of audio/x-raw. Additionally, this patch implements only
"normal" timestamping mode from AAF. "Sparse" mode should be implemented
in future.
Upcoming patches will introduce other AVTP payloader elements that will
have some common code. For that reason, this patch introduces the
GstAvtpBasePayload abstract class that implements common payloader
functionalities, and the GstAvtpAafPay class that extends the
GstAvtpBasePayload class, implementing AAF-specific functionalities.
The AAF payloader element is most likely to be used with the AVTP sink
element (to be introduced by a later patch) but it could also be used
with UDP sink element to implement AVTP over UDP as described in IEEE
1722-2016 Annex J.
This element was inspired by RTP payloader elements.
This patch introduces the bootstrap code from the AVTP plugin (plugin
definition and init) as well as the build system files. Upcoming patches
will introduce payloaders, source and sink elements provided by the AVTP
plugin. These elements can be utilized by a GStreamer pipeline to
implement TSN audio/video applications.
Regarding the plugin build system files, both autotools and meson files
are introduced. The AVTP plugin is landed in ext/ since it has an
external dependency on libavtp, an opensource AVTP packetization
library. For further information about libavtp check [1].
[1] https://github.com/AVnu/libavtp
