Commit graph

296 commits

Author SHA1 Message Date
Arun Raghavan
e72db57179 webrtc: Fix whipclientsink name in README
The element name was changed, but the documentation wasn't updated to
match.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1764>
2024-09-03 16:44:19 -04:00
Mathieu Duponchelle
2f9bb62b6b gstwebrtc-api: create control data channel when offering
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1755>
2024-08-27 07:52:12 +02:00
Mathieu Duponchelle
4cf93ccbdb net/webrtc: Add missing npm command to README
Fixes https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/issues/589

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1746>
2024-08-22 15:46:28 +02:00
Jerome Colle
dee0e32dde webrtcsink: add nvv4l2av1enc support
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1735>
2024-08-22 06:41:52 +00:00
Mathieu Duponchelle
8ad882bed5 gstwebrtc-api: address issues raised by mix matrix support
1c48d7065d was mistakenly merged too
early, and there were concerns about the implementation and API design:

The fact that the frontend had to expose a text area specifically for
sending over a mix matrix, and had to manually edit in floats into the
stringified JSON was suboptimal.

Said text area was always present even when remote control was not
enabled.

The sendControlRequest API was made more complex than needed by
accepting an optional stringifier callback.

This patch addresses all those concerns:

The deserialization code in webrtcsink is now made more clever and
robust by first having it pick a numerical type to coerce to when
deserializing arrays with numbers, then making sure it doesn't allow
mixed types in arrays (or arrays of arrays as those too must share
the same inner value type).

The frontend side simply sends over strings wrapped with a request
message envelope to the backend.

The request text area is only shown when remote control is enabled.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1725>
2024-08-22 05:54:46 +00:00
Mathieu Duponchelle
5dc2d56c0e webrtcsink: store mids per-session instead of globally
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1730>
2024-08-21 21:20:40 +00:00
Mathieu Duponchelle
16ee51621e webrtcsink: fix segment format mismatch with remote offer
webrtcsink was starting the negotiation process on Ready and concurrently
moving the consumer pipeline to Playing, but when answering the remote
description was set so fast that input streams were connected (and the time
format set on appsrc) before the state change to Paused had completed.

This meant gst_base_src_start was happening after that and setting the format
back to bytes, the time segment that was next coming in then caused:

basesrc gstbasesrc.c:4255:gst_base_src_push_segment:<video_0> segment format mismatched, ignore

And the consumer pipeline errored out.

The same issue existed in theory when webrtcsink was creating the offer,
but was much harder to trigger as it required that the remote answer
came in before the state change to Paused had completed.

This commit fixes the issue by simply waiting for the state to have
changed to Paused before negotiating.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1730>
2024-08-21 21:20:40 +00:00
Mathieu Duponchelle
1c48d7065d gstwebrtc-api example: add support for requesting mix matrix
This is one example of how a consumer might send over custom upstream
event requests to the producer.

As webrtcsink will deserialize numbers in priority as integers, we need
a custom stringifying function to ensure members of the matrix array are
indeed serialized with the floating point.

An optional stringifier parameter is thus added to the
sendControlRequest API.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1711>
2024-08-15 15:42:04 +00:00
Mathieu Duponchelle
01e28ddfe2 webrtcsink: implement generic data channel control mechanism ..
.. and deprecate data channel navigation in favor of it.

A new property, "enable-data-channel-control" is exposed, when set to
TRUE a control data channel is offered, over which can be sent typed
upstream events.

This means further upstream events will be usable, for now only
navigation and custom upstream events are handled.

In addition, send response messages to notify the consumer of whether
its requests have been handled.

In the future this can also be extended to allow the consumer to send
queries, or seek events ..

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1711>
2024-08-15 15:42:04 +00:00
Mathieu Duponchelle
0a6963f7ce gstwebrtc-api: example: use http by default
That way the webpage connects with ws:/ to the signaller.

Fixes: https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/issues/589
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1704>
2024-08-14 14:10:04 +00:00
Sebastian Dröge
bc930122ba webrtcsrc: Make sure to always call end_session() without the state lock
This was already done in another place for the same reason: preventing a
deadlock. It's probably not correct as hinted by the FIXME comment but
better than deadlocking at least.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1701>
2024-08-13 06:04:09 +00:00
Mathieu Duponchelle
0da1c8e9c9 webrtcsink: fix assertions when finalizing
Dumping the pipeline on state changes from an async bus handler was
triggering criticals.

Instead, dump from the sync handler.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1706>
2024-08-12 09:13:06 +02:00
Mathieu Duponchelle
9080c90120 net/webrtc: add support for answering to webrtcsink
Support was added to the base class when the AWS KVS signaller was
implemented, but the default signaller still only supported the case
where the producer was creating the offer.

Also extend the javascript API

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1702>
2024-08-09 14:02:48 +02:00
Mathieu Duponchelle
a9ff9615ff net/webrtc: correct signaller debug category
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1702>
2024-08-08 18:28:43 +02:00
Mathieu Duponchelle
64f0b76f71 webrtc: update README with section on embedded signalling / web services
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1671>
2024-08-08 16:40:46 +02:00
Mathieu Duponchelle
9455e09d9f webrtcsink: expose properties for running web server
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1671>
2024-08-08 16:40:46 +02:00
Mathieu Duponchelle
b709c56478 webrtcsink: expose properties for running signalling server
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1671>
2024-08-07 19:55:00 +02:00
Sebastian Dröge
6c04b59454 webrtcsrc: Don't hold the state lock while removing sessions
Removing a session can drop its bin and during release of the bin its
pads are removed, but the pad-removed handler is also taking the state
lock.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1695>
2024-08-07 09:35:15 +00:00
Sebastian Dröge
b83b6031e5 Update etherparse and async-tungstenite dependencies
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1691>
2024-08-06 09:00:32 +03:00
Dave Lucia
3a949db720 net/webrtc: Fix turn-servers nick: user -> use
Noticed this typo

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1690>
2024-08-05 12:38:51 -04:00
Sebastian Dröge
fa060b9fa0 Fix various 1.80 clippy warnings
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1688>
2024-08-05 14:14:17 +03:00
Mathieu Duponchelle
86039dd5c1 webrtc-api example: do not rely on webpack / npm proxying websocket
Instead simply use the desired address directly from the reference
example, this makes it work out of the box without placing expectations
on the web server.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1674>
2024-07-30 16:29:54 +00:00
Loïc Le Page
5a1d12419f gstwebrtc-api: always include index file in dist for convenience
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1670>
2024-07-17 08:27:31 +00:00
François Laignel
34b791ff5e webrtc: add raw payload support
This commit adds support for raw payloads such as L24 audio to `webrtcsink` &
`webrtcsrc`.

Most changes take place within the `Codec` helper structure:

* A `Codec` can now advertise a depayloader. This also ensures that a format
  not only can be decoded when necessary, but it can also be depayloaded in the
  first place.
* It is possible to declare raw `Codec`s, meaning that their caps are compatible
  with a payloader and a depayloader without the need for an encoder and decoder.
* Previous accessor `has_decoder` was renamed as `can_be_received` to account
  for codecs which can be handled by an available depayloader with or without
  the need for a decoder.
* New codecs were added for the following formats:
  * L24, L16, L8 audio.
  * RAW video.

The `webrtc-precise-sync` examples were updated to demonstrate streaming of raw
audio or video.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1501>
2024-07-16 19:32:02 +00:00
François Laignel
6e9855c36b webrtcsink: fix property types for rav1enc
Fixes: https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/issues/572
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1667>
2024-07-12 18:59:20 +02:00
Sebastian Dröge
a8ccfe49d9 webrtc: Require livekit-protocol < 0.3.4 due to uncoordinated breaking changes
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1663>
2024-07-11 20:00:24 +03:00
Sebastian Dröge
98b28d69ce Update for new debug log macro syntax
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1658>
2024-07-08 11:25:23 +03:00
leonardo salvatore
f303992e0c webrtcsink: initial support for vpuenc_h264 encoder for imx8mp, default values set to cover a common streaming scenario
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1639>
2024-07-01 07:34:04 +00:00
Sebastian Dröge
47d62b6d78 Update for new clone/closure macro syntax
Also fix various weak/strong references in the webrtc plugin, and make
sure to pass the object to debug log functions in every place.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1625>
2024-06-21 11:54:58 +03:00
Sebastian Dröge
9b323a6519 Use Option::is_some_and(...) instead of Option::map_or(false, ...)
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1630>
2024-06-19 13:03:37 +00:00
Sebastian Dröge
23d998a1db Slightly improve code making use of element factories retrieved from an element
We can use `is_some_and(...)` instead of `map_or(false, ...)`.

Also in a few places the factory was retrieved multiple times, one time
with unwrapping and another time with handling the `None` case
correctly. Instead of unwrapping, move code to handle the `None` case.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1630>
2024-06-19 13:03:37 +00:00
Sebastian Dröge
ba70bb1154 deny: Add override for older tungstenite
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1603>
2024-06-06 10:34:12 +00:00
Sebastian Dröge
85c38107cf webrtc: Update to async-tungstenite 0.26
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1603>
2024-06-06 10:34:12 +00:00
Francisco Javier Velázquez-García
8fc652f208 webrtcsink: Refactor value retrieval to avoid lock poisoning
When setting an incorrect property name in settings,
start_stream_discovery_if_needed would panic because it attempts to
unwrap a poisoned lock for settings.

This refactor avoids that situation.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1594>
2024-05-31 08:10:23 +00:00
Francisco Javier Velázquez-García
568e8533fa webrtcsink: Fix typo in property name for av1enc
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1594>
2024-05-31 08:10:23 +00:00
Arun Raghavan
04e9e5284c webrtc: signaller: A couple of minor doc fixups
The expectation is `Returns:`, not `Return:`

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1525>
2024-05-30 22:16:46 +03:00
Arun Raghavan
1c54c77840 webrtcsink: Add a mechanism for SDP munging
Unfortunately, server implementations might have odd SDP-related quirks,
so let's allow clients a way to work around these oddities themselves.
For now, this means that a client can fix up the H.264 profile-level-id
as required by Twitch (whose media pipeline is more permissive than the
WHIP implementation).

Fixes: https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/issues/516
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1525>
2024-05-30 22:16:46 +03:00
Taruntej Kanakamalla
83f76280f5 net/webrtc: Example for whipserver
rudimentary sample to test multiple WHIP client connections

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1339>
2024-05-29 21:03:27 +00:00
Taruntej Kanakamalla
712d4757c3 net/webrtc/whip_signaller: multiple client support in the server
- generate a new session id for every new client
use the session id in the resource url

- remove the producer-peer-id property in the WhipServer signaler as it
is redundant to have producer id in a session having only one producer

- read the 'producer-peer-id' property on the signaller conditionally
if it exists else use the session id as producer id

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1339>
2024-05-29 21:03:27 +00:00
Taruntej Kanakamalla
de726ca8d2 net/webrtc: multi producer support in webrtcsrc
- Add a new structure Session
  - manage each producer using a session
  - avoid send EOS when a session terminates, instead keep running
    waiting for any new producer to connect

- Maintain a bin element per session
  - each session bin encapsulates webrtcbin and the decoder if needed
    as well as the parser and filter if requested by the application
    (through request-encoded-filter)
  - this will be helpful to cleanup the session's respective elements
    when the corresponding producer terminates the session

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1339>
2024-05-29 21:03:27 +00:00
François Laignel
4259d284bd webrtc: add android webrtcsrc example
This commit adds an Android `webrtcsrc` based example with the following
features:

* A first view allows retrieving the producer list from the signaller (peer ids
  are uuids which are too long to tap, especially using an onscreen keyboard).
* Selecting a producer opens a second view. The first available video stream is
  rendered on a native Surface. All the audio streams are rendered using
  `autoaudiosink`.

Available Settings:

* Signaller URI.
* A toggle to prefer hardware decoding for OPUS, otherwise the app defaults to
  raising `opusdec`'s rank. Hardware decoding was moved aside since it was found
  to crash the app on all tested devices (2 smartphones, 1 tv).

**Warning**: in order to ease testing, this demonstration application enables
unencrypted network communication. See `AndroidManifest.xml`.

The application uses the technologies currenlty proposed by Android Studio when
creating a new project:

* Kotlin as the default language, which is fully interoperable with Java and
  uses the same SDK.
* gradle 8.6.
* kotlin dialect for gradle. The structure is mostly the same as the previously
  preferred dialect, for which examples can be found online readily.
* However, JNI code generation still uses Makefiles (instead of CMake) due to
  the need to call [`gstreamer-1.0.mk`] for `gstreamer_android` generation.
  Note: on-going work on that front:
  - https://gitlab.freedesktop.org/gstreamer/cerbero/-/merge_requests/1466
  - https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/6794

Current limitations:

* x86 support is currently discarded as `gstreamer_android` libs generation
  fails (observed with `gstreamer-1.0-android-universal-1.24.3`).
* A selector could be added to let the user chose the video streams and
  possibly decide whether to render all audio streams or just select one.

Nice to have:

* Support for the synchronization features of the `webrtc-precise-sync-recv`
  example (NTP clock, RFC 7273).
* It could be nice to use Rust for the specific native code.

[`gstreamer-1.0.mk`]: https://gitlab.freedesktop.org/gstreamer/cerbero/-/blob/main/data/ndk-build/gstreamer-1.0.mk

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1578>
2024-05-24 16:14:13 +00:00
cdelguercio
c99cabfbc5 webrtcsink: Add VP9 parser after the encoder for VP9 too
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1572>
2024-05-23 10:16:59 +03:00
cdelguercio
f5a7de9dc3 webrtcsink: Support av1 via nvav1enc, av1enc, and rav1enc
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1572>
2024-05-23 10:16:59 +03:00
Robert Ayrapetyan
bac5845be1 webrtc: add support for insecure tls connections
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1553>
2024-05-16 19:34:57 +00:00
Sebastian Dröge
613ed56675 webrtcsink: Add a custom signaller example in Python
This re-implements the default webrtcsink/src signalling protocol in
Python for demonstration purposes.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1569>
2024-05-10 15:59:12 +00:00
Sebastian Dröge
e8e173d0d0 webrtc: Update Signallable interface to new interface definition API
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1570>
2024-05-10 14:13:55 +03:00
Mathieu Duponchelle
8861fc493b webrtcsink: improve error when no discovery pipeline runs
If for instance no encoder was found or the RTP plugin was missing,
it is possible that no discovery pipeline will run for a given stream.

Provide a more helpful error message for that case.

Fixes: https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/issues/534
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1560>
2024-05-06 11:39:48 +00:00
Sebastian Dröge
be3ae583bc Fix new Rust 1.78 clippy warnings
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1559>
2024-05-02 18:36:23 +03:00
François Laignel
542030fd82 webrtcsink: don't panic if input CAPS are not supported
If a user constrained the supported CAPS, for instance using `video-caps`:

```shell
gst-launch-1.0 videotestsrc ! video/x-raw,format=I420 ! x264enc \
    ! webrtcsink video-caps=video/x-vp8
```

... a panic would occur which was internally caught without the user being
informed except for the following message which was written to stderr:

> thread 'tokio-runtime-worker' panicked at net/webrtc/src/webrtcsink/imp.rs:3533:22:
>   expected audio or video raw caps: video/x-h264, [...] <br>
> note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

The pipeline kept running.

This commit converts the panic into an `Error` which bubbles up as an element
`StreamError::CodecNotFound` which can be handled by the application.
With the above `gst-launch`, this terminates the pipeline with:

> [...] ERROR  webrtcsink net/webrtc/src/webrtcsink/imp.rs:3771:gstrswebrtc::
>   webrtcsink:👿:BaseWebRTCSink::start_stream_discovery_if_needed::{{closure}}:<webrtcsink0>
> Error running discovery: Unsupported caps: video/x-h264, [...] <br>
> ERROR: from element /GstPipeline:pipeline0/GstWebRTCSink:webrtcsink0:
>   There is no codec present that can handle the stream's type. <br>
> Additional debug info: <br>
> net/webrtc/src/webrtcsink/imp.rs(3772): gstrswebrtc::webrtcsink:👿:BaseWebRTCSink::
> start_stream_discovery_if_needed::{{closure}} (): /GstPipeline:pipeline0/GstWebRTCSink:webrtcsink0:
> Failed to look up output caps: Unsupported caps: video/x-h264, [...] <br>
> Execution ended after 0:00:00.055716661 <br>
> Setting pipeline to NULL ...

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1540>
2024-04-14 23:09:09 +02:00
François Laignel
3fc38be5c4 webrtc: add missing tokio feature for precise sync examples
Clippy caught the missing feature `signal` which is used by the WebRTC precise
synchronization examples. When running `cargo` `check`, `build` or `clippy`
without `no-default-dependencies`, this feature was already present due to
dependents crates.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1541>
2024-04-14 16:50:33 +02:00