In some situations, a translated alternate audio stream for a content
might be available.
Instead of going through transcription and translation of the original
audio stream, it may be preferrable for accuracy purposes to simply
transcribe the secondary audio stream.
This MR adds support for doing just that:
* Secondary audio sink pads can be requested as "sink_audio_%u"
* Sometimes audio source pads are added at that point to pass through
the audio, as "src_audio_%u"
* The main transcription bin now contains per-input stream transcription
bins. Those can be individually controlled through properties on the
sink pads, for instance translation-languages can be dynamically set
per audio stream
* Some properties that originally existed on the main element still
remain, but are now simply mapped to the always audio sink pad
* Releasing of secondary sink pads is nominally implemented, but not
tested in states other than NULL
An example launch line for this would be:
```
$ gst-launch-1.0 transcriberbin name=transcriberbin latency=8000 accumulate-time=0 \
cc-caps="closedcaption/x-cea-708, format=cc_data" sink_audio_0::language-code="es-US" \
sink_audio_0::translation-languages="languages, transcript=cc3"
uridecodebin uri=file:///home/meh/Music/chaplin.mkv name=d
d. ! videoconvert ! transcriberbin.sink_video
d. ! clocksync ! audioconvert ! transcriberbin.sink_audio
transcriberbin.src_video ! cea608overlay field=1 ! videoconvert ! autovideosink \
transcriberbin.src_audio ! audioconvert ! fakesink \
uridecodebin uri=file:///home/meh/Music/chaplin-spanish.webm name=d2 \
d2. ! audioconvert ! transcriberbin.sink_audio_0 \
transcriberbin.src_audio_0 ! fakesink
```
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1546>
If a user constrained the supported CAPS, for instance using `video-caps`:
```shell
gst-launch-1.0 videotestsrc ! video/x-raw,format=I420 ! x264enc \
! webrtcsink video-caps=video/x-vp8
```
... a panic would occur which was internally caught without the user being
informed except for the following message which was written to stderr:
> thread 'tokio-runtime-worker' panicked at net/webrtc/src/webrtcsink/imp.rs:3533:22:
> expected audio or video raw caps: video/x-h264, [...] <br>
> note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
The pipeline kept running.
This commit converts the panic into an `Error` which bubbles up as an element
`StreamError::CodecNotFound` which can be handled by the application.
With the above `gst-launch`, this terminates the pipeline with:
> [...] ERROR webrtcsink net/webrtc/src/webrtcsink/imp.rs:3771:gstrswebrtc::
> webrtcsink:👿:BaseWebRTCSink::start_stream_discovery_if_needed::{{closure}}:<webrtcsink0>
> Error running discovery: Unsupported caps: video/x-h264, [...] <br>
> ERROR: from element /GstPipeline:pipeline0/GstWebRTCSink:webrtcsink0:
> There is no codec present that can handle the stream's type. <br>
> Additional debug info: <br>
> net/webrtc/src/webrtcsink/imp.rs(3772): gstrswebrtc::webrtcsink:👿:BaseWebRTCSink::
> start_stream_discovery_if_needed::{{closure}} (): /GstPipeline:pipeline0/GstWebRTCSink:webrtcsink0:
> Failed to look up output caps: Unsupported caps: video/x-h264, [...] <br>
> Execution ended after 0:00:00.055716661 <br>
> Setting pipeline to NULL ...
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1540>
Clippy caught the missing feature `signal` which is used by the WebRTC precise
synchronization examples. When running `cargo` `check`, `build` or `clippy`
without `no-default-dependencies`, this feature was already present due to
dependents crates.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1541>
When swapping between several development branches, compilation times can be
frustrating. This commit proposes adding features to control which signaller
to include when building the webrtc plugin. By default, all signallers are
included, just like before.
Compiling the `webrtc-precise-sync` examples with `--no-default-features`
reduces compilation to 267 crates instead of 429 when all signallers are
compiled in.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1539>
This commit implements [RFC 7273] (NTP & PTP clock signalling & synchronization)
for `webrtcsink` by adding the "ts-refclk" & "mediaclk" SDP media attributes to
identify the clock. These attributes are handled by `rtpjitterbuffer` on the
consumer side. They MUST be part of the SDP offer.
When used with an NTP or PTP clock, "mediaclk" indicates the RTP offset at the
clock's origin. Because the payloaders are not instantiated when the offer is
sent to the consumer, the RTP offset is set to 0 and the payloader
`timstamp-offset`s are set accordingly when they are created.
The `webrtc-precise-sync` examples were updated to be able to start with an NTP
(default), a PTP or the system clock (on the receiver only). The rtp jitter
buffer will synchronize with the clock signalled in the SDP offer provided the
sender is started with `--do-clock-signalling` & the receiver with
`--expect-clock-signalling`.
[RFC 7273]: https://datatracker.ietf.org/doc/html/rfc7273
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1500>
When an encoder was not supported by the `VideoEncoder` `bitrate` accessors, an
`unimplemented` panic would occur which would poison `state` & `settings`
`Mutex`s resulting in other threads panicking, notably entering `end_session()`,
which lead to many failures in `BinImplExt::parent_remove_element()` until a
segmentation fault ended the process. This was observed using `vaapivp9enc`.
This commit logs a warning if an encoder isn't supported by the `bitrate`
accessors and silently by-passes `bitrate`-related operations when unsupported.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1534>
Otherwise the clock id will simply be overridden instead of unscheduling
it, and if the streaming thread of the source pad currently waits on it
then it will wait potentially for a very long time and deactivating the
pad would wait for that to happen.
Also unschedule the clock id on `Drop` of the state to be one the safe
side and not simply forget about it.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1526>
Add `self.increasing_duration` and `self.increasing_counter`
to logs to provide more details of why `overuse_filter()`
determines overuse of network.
To get access to the latest values of those fields we need
to move down the log call. But that is fine, since no other
logged data is modified between the old and new location of
`gst::log!()`.
We do not bother logging `self.last_overuse_estimate` since
that is simply the previously logged value of `estimate`. We
must put the log call before we write the latest value to it
though, in case we want to log it in the future.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1522>
