mirrors/gst-plugins-rs
mirrors/gst-plugins-rs
1
1
Fork 0
mirror of https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs.git synced 2025-02-27 10:06:21 +00:00
Code Issues Projects Releases Wiki Activity
3832 commits 14 branches 107 tags 62 MiB
Commit graph

51 commits

Author SHA1 Message Date
Mathieu Duponchelle
f08b99dcd3 aws/transcriber*: log all debug details from AWS errors 
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/2055>
 2025-02-10 11:16:44 +00:00
Mathieu Duponchelle
71121a2380 aws: reimplement two separate wrappers for translate and transcribe 
The original awstranscribe element has grown too complex when
integrating translations for reasons that in retrospect were wrong:

As awstranscribe outputs words one by one, I decided we wanted to
perform translations there with larger sentences if available, but an
alternative design where a separate translation element is composed
downstream is also possible, as long as that element accumulates words
and enough latency is set on the transcriber.

An important difference is that the new elements do not expose unsynced
pads, this use case is instead now served by simple messages on the bus.

The elements should otherwise be at feature parity with the original
element.

A higher-level bin is also provided for convenience (and usage within
transcriberbin): translationbin.

A transcriber element can be provided to this bin, which exposes an
always audio sink pad, and an always text sink pad (for the
transcripts).

Additional source pads can be requested for translations, for now the
bin always uses `awstranslate` as the translator, but this can be made
configurable.

This element is usable as a transcriber in `transcriberbin`.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/2055>
 2025-02-10 11:16:44 +00:00
Mathieu Duponchelle
c51a65d973 awstranscriber, speechmatics: store language tags on translation source pads 
In order to do so we need to activate the pad as soon as it is added,
which means we can no longer start the task at this point, instead wait
for stream-start to do so now.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/2029>
 2025-01-20 14:27:05 +00:00
Mathieu Duponchelle
9c844acba5 aws/transcriber: fix unsynced_translate_src_%u presence 
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1930>
 2024-11-29 22:09:37 +00:00
Mathieu Duponchelle
f16f8f69d5 aws/transcriber: don't adjust late item duration 
It makes for a better user experience to simply adjust the pts of a late
item, but to preserve its duration: for instance a speech synthesis
element might use the duration as a hint for speeding up the audio.

Future late items may also be similarly offset anyway.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1930>
 2024-11-29 22:09:37 +00:00
Mathieu Duponchelle
9972c83c60 aws/transcriber: put posting of warning messages behind property 
Repeated warning messages are fairly noisy with gst-launch, better make
this behavior opt-in.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1930>
 2024-11-29 22:09:37 +00:00
Mathieu Duponchelle
c57b74e269 awstranscriber: release matching unsynced pad along request pads 
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1930>
 2024-11-29 22:09:37 +00:00
Mathieu Duponchelle
b5bd7d047c awstranscribe: output original transcripts to separate pad 
When the transcriber is used in a live situation, it can be useful
to save a transcript for editing after the fact when producing a
VOD.

Each source pad now gets an "unsynced_" pendant. That unsynced pad
is pushed to from the context of the "live" source pad task. Flow
returns from the unsynced pads are ignored, we simply check the
last flow return before attempting to push the next transcript.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1915>
 2024-11-18 17:30:54 +00:00
Mathieu Duponchelle
5f8e8b4873 aws: add wrapper for the polly text to speech API 
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1899>
 2024-11-05 08:46:48 +00:00
Mathieu Duponchelle
79845fd99a awstranscriber: post warning message with details when item is late 
When the latency is configured to a value that is too low, items will be
pushed out with an adjusted timestamp, thus affecting synchronization.

It can be useful for the application to receive details about those
adjustments.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1793>
 2024-10-28 13:13:31 +00:00
Sebastian Dröge
7e59c3f0fd Remove once_cell dependency 
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1868>
 2024-10-21 17:53:18 +00:00
Sebastian Dröge
0e3d019e24 aws: Don't unnecessarily clone AWS behaviour version 
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1864>
 2024-10-20 19:53:15 +00:00
Sebastian Dröge
00a4398aee aws: Allow a deprecated BehaviourVersion for now 
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1864>
 2024-10-20 19:53:15 +00:00
Mathieu Duponchelle
bfc32cc692 net/aws: fix spurious dispatch failures 
Since https://github.com/awslabs/aws-sdk-rust/discussions/956, the AWS
SDK errors out HTTP streams that do not transfer data for more than 5
seconds.

This probably should be an opt-in bhevior as it clearly not generically
useful, but as it is we need to opt out.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1760>
 2024-09-05 07:43:23 +00:00
Mathieu Duponchelle
65508cfe75 net/aws: don't discard errors from transcribe loop 
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1760>
 2024-09-05 07:43:23 +00:00
Mathieu Duponchelle
66727188cf net/aws: fix sanity check in transcribe loop 
When we receive a new alternative we want to avoid iterating out of
bounds, but the comparison between the current index and the length of
the alternative should not log an error when partial_index == length, as
Vec::drain(length..) is valid, and it is completely valid for AWS to
send us a new alternative with as many items as we have already
dequeued.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1751>
 2024-08-26 11:37:08 +02:00
Sebastian Dröge
98b28d69ce Update for new debug log macro syntax 
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1658>
 2024-07-08 11:25:23 +03:00
Sebastian Dröge
9b323a6519 Use Option::is_some_and(...) instead of Option::map_or(false, ...) 
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1630>
 2024-06-19 13:03:37 +00:00
Sebastian Dröge
be3ae583bc Fix new Rust 1.78 clippy warnings 
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1559>
 2024-05-02 18:36:23 +03:00
François Laignel
a870d60621 aws: improve error message logs 
The `Display` and `Debug` trait for the AWS error messages are not very useful.

- `Display` only prints the high level error, e.g.: "service error".
- `Debug` prints all the fields in the error stack, resulting in hard to read
  messages with redudant or unnecessary information. E.g.:

> ServiceError(ServiceError { source: BadRequestException(BadRequestException {
> message: Some("1 validation error detected: Value 'test' at 'languageCode'
> failed to satisfy constraint: Member must satisfy enum value set: [ar-AE,
> zh-HK, en-US, ar-SA, zh-CN, fi-FI, pl-PL, no-NO, nl-NL, pt-PT, es-ES, th-TH,
> de-DE, it-IT, fr-FR, ko-KR, hi-IN, en-AU, pt-BR, sv-SE, ja-JP, ca-ES, es-US,
> fr-CA, en-GB]"), meta: ErrorMetadata { code: Some("BadRequestException"),
> message: Some("1 validation error detected: Value 'test' at 'languageCode'
> failed to satisfy constraint: Member must satisfy enum value set: [ar-AE,
> zh-HK, en-US, ar-SA, zh-CN, fi-FI, pl-PL, no-NO, nl-NL, pt-PT, es-ES, th-TH,
> de-DE, it-IT, fr-FR, ko-KR, hi-IN, en-AU, pt-BR, sv-SE, ja-JP, ca-ES, es-US,
> fr-CA, en-GB]"), extras: Some({"aws_request_id": "1b8bbafd-5b71-4ba5-8676-28432381e6a9"}) } }),
> raw: Response { status: StatusCode(400), headers: Headers { headers:
> {"x-amzn-requestid": HeaderValue { _private: H0("1b8bbafd-5b71-4ba5-8676-28432381e6a9") },
> "x-amzn-errortype": HeaderValue { _private:
> H0("BadRequestException:http://internal.amazon.com/coral/com.amazonaws.transcribe.streaming/") },
> "date": HeaderValue { _private: H0("Tue, 26 Mar 2024 17:41:31 GMT") },
> "content-type": HeaderValue { _private: H0("application/x-amz-json-1.1") },
> "content-length": HeaderValue { _private: H0("315") }} }, body: SdkBody {
> inner: Once(Some(b"{\"Message\":\"1 validation error detected: Value 'test'
> at 'languageCode' failed to satisfy constraint: Member must satisfy enum value
> set: [ar-AE, zh-HK, en-US, ar-SA, zh-CN, fi-FI, pl-PL, no-NO, nl-NL, pt-PT,
> es-ES, th-TH, de-DE, it-IT, fr-FR, ko-KR, hi-IN, en-AU, pt-BR, sv-SE, ja-JP,
> ca-ES, es-US, fr-CA, en-GB]\"}")), retryable: true }, extensions: Extensions {
> extensions_02x: Extensions, extensions_1x: Extensions } } })

This commit adopts the most informative and concise solution I could come up
with to log AWS errors. With the above error case, this results in:

> service error: Error { code: "BadRequestException", message: "1 validation
> error detected: Value 'test' at 'languageCode' failed to satisfy constraint:
> Member must satisfy enum value set: [ar-AE, zh-HK, en-US, ar-SA, zh-CN, fi-FI,
> pl-PL, no-NO, nl-NL, pt-PT, es-ES, th-TH, de-DE, it-IT, fr-FR, ko-KR, hi-IN,
> en-AU, pt-BR, sv-SE, ja-JP, ca-ES, es-US, fr-CA, en-GB]",
> aws_request_id: "a40a32a8-7b0b-4228-a348-f8502087a9f0" }

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1521>
 2024-03-26 20:05:32 +01:00
François Laignel
9f27bde36a aws: use fixed BehaviorVersion 
Quoting [`BehaviorVersion` documentation]:

> Over time, new best-practice behaviors are introduced. However, these
> behaviors might not be backwards compatible. For example, a change which
> introduces new default timeouts or a new retry-mode for all operations might
> be the ideal behavior but could break existing applications.

This commit uses `BehaviorVersion::v2023_11_09()`, which is the latest
major version at the moment. When a new major version is released, the method
will be deprecated, which will warn us of the new version and let us decide
when to upgrade, after any changes if required. This is safer that using
`latest()` which would silently use a different major version, possibly
breaking existing code.

[`BehaviorVersion` documentation]: https://docs.rs/aws-config/1.1.8/aws_config/struct.BehaviorVersion.html

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1520>
 2024-03-26 17:44:16 +01:00
Sebastian Dröge
4ad101b53b Use once_cell crate directly again 
The glib crate does not depend on it anymore and also does not re-export
it anymore.

Also switch some usages of OnceCell to OnceLock from std.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1441>
 2024-01-31 18:07:57 +02:00
Sebastian Dröge
dee27e35b7 Update to latest AWS SDK 
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1395>
 2023-11-17 11:22:29 +02:00
Mathieu Duponchelle
5371eb52ad Port to AWS SDK 0.57/0.35 
Co-authored-by: Sebastian Dröge <sebastian@centricular.com>
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1379>
 2023-11-03 15:13:45 +00:00
Sebastian Dröge
16b917abb1 Update for gst::Rank API changes 2023-11-02 14:10:59 +02:00
Jordan Petridis
a2d7f42138 Fix compilation after glib bindings changes 
loggable_error! can now expand variables and we no longer need
the format! on our side.

https://github.com/gtk-rs/gtk-rs-core/pull/1210

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1366>
 2023-10-22 01:20:56 +03:00
Sebastian Dröge
8d433761d1 Fix indentation of let-else blocks 
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1308>
 2023-08-29 12:21:38 +00:00
Bilal Elmoussaoui
dd2d7d9215 Use re-exported once_cell 
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1268>
 2023-07-06 17:50:49 +03:00
Seungha Yang
1f0ce101eb awstranscriber: Tone down log message 
It's not an ERROR case at all

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1263>
 2023-06-28 23:57:54 +09:00
Mathieu Duponchelle
6346d5608e net/aws/transcriber: track discont offset in input stream 
and add it up to subsequent transcripts.

This ensures synchronization is maintained even after the input stream
experiences a discontinuity and a gap in its timestamps.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1230>
 2023-06-02 08:55:11 +00:00
François Laignel
7ba0073052 use Pad builders for optional name definition 
Also, apply auto-naming in the following cases

* When building from a non wildcard-named template, the name of the template is
  automatically assigned to the Pad. User can override with a specific name by
  calling `name()` on the `PadBuilder`.
* When building with a target and no name was provided via the above, the
  GhostPad is named after the target.

See https://gitlab.freedesktop.org/gstreamer/gstreamer-rs/-/issues/448
Auto-naming discussion: https://gitlab.freedesktop.org/gstreamer/gstreamer-rs/-/merge_requests/1255#note_1891181

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1197>
 2023-05-12 12:55:31 +02:00
Mathieu Duponchelle
f366c20869 awstranscriber: fix what we send over for translations 
Prior to this commit, we were sending over words concatenated together
with no separators, for instance "Idon'twanttobeanemperor".

The translation service seems clever enough to translate the contents
anyway, but there is no reason to make its task harder than necessary,
and it didn't re-add separators when the target language was the same as
the source language, which resulted in less than ideal output.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1171>
 2023-04-10 20:47:12 +00:00
Mathieu Duponchelle
408fd2030c awstranscriber: slight debug improvement 
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1171>
 2023-04-10 20:47:12 +00:00
Guillaume Desmottes
403004a85e fix typos 
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1170>
 2023-04-10 13:35:32 +02:00
Seungha Yang
762fb86ce7 awstranscriber: Reset start_time per task 
Otherwise wrong start time can be assigned if the element is
reused with state change

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1159>
 2023-04-05 18:22:59 +00:00
Seungha Yang
4000d60305 awstranscriber: Avoid too large initial GAP event 
Initialized GstSegment.position is always zero

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1154>
 2023-04-03 13:05:15 +00:00
Sebastian Dröge
6fe806c2b5 aws: Update to AWS SDK 0.55/0.25 
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1152>
 2023-03-31 09:12:26 +00:00
François Laignel
2b32d00589 net/aws/transcriber: use two queues for sending transcript items 
* A queue dedicated to transcript items not intended for translation.
* A queue dedicated to transcript items intended for translation. The items are
  enqueued after a separator is detected or translate-lookahead was reached.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1137>
 2023-03-16 20:29:31 +01:00
François Laignel
5a5ca76d9d net/aws/transcriber: desambiguify SrcPad output items queue 
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1137>
 2023-03-16 12:41:07 +01:00
François Laignel
162db2f3b9 net/aws/transcriber: fix translate lookahead 
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1137>
 2023-03-16 12:39:15 +01:00
François Laignel
d5d6a4daf9 net/aws/transcriber: rename prop transcript-lookahead & TranslationSrcPad 
... as translate-lookahead and TranslateSrcPad.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1137>
 2023-03-16 12:37:31 +01:00
François Laignel
3b3f0c1a29 net/aws/transcriber: fix transcript-lookahead prop nick 
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1136>
 2023-03-14 21:11:33 +01:00
François Laignel
299e25ab3c net/aws/transcriber: translate: optional experimental translation tokenization 
This commit adds an optional experimental translation tokenization feature.
It can be activated using the `translation_src_%u` pads property
`tokenization-method`. For the moment, the feature is deactivated by default.

The Translate ws accepts '<span></span>' tags in the input and adds matching
tags in the output. When an 'id' is also provided as an attribute of the
'span', the matching output tag also uses this 'id'.

In the context of close captions, the 'id's are of little use. However, we can
take advantage of the spans in the output to identify translation chunks, which
more or less reflect the rythm of the input transcript.

This commit adds simples spans (no 'id') to the input Transcript Items and
parses the resulting spans in the translated output, assigning the timestamps
and durations sequentially from the input Transcript Items. Edge cases such as
absence of spans, nested spans were observed and are handled here. Similarly,
mismatches between the number of input and output items are taken care of by
some sort of reconcialiation.

Note that this is still experimental and requires further testings.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1109>
 2023-03-14 13:48:32 +00:00
François Laignel
743e97738f net/aws/transcriber: add translation request src pads 
This commit adds an optional transcript translation feature implemented as
request src Pads.

When requesting a src Pad, the user can specify the translation language code
using Pad properties 'language-code'.

The following properties are defined on the Element:

- 'transcribe-latency': formerly 'latency', defines the expected latency for
  the Transcribe webservice.
- 'translate-latency': defines the expected latency for the Translate
  webservice.
- 'transcript-lookahead': maximum transcript duration to send to translation
  when a transcript is hitting its deadline and no punctuation was found.

When the input and output languages are the same, only the 'transcribe-latency'
is used for the Pad. Otherwise, the resulting latency is the addition of
'transcribe-latency' and 'translate-latency'.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1109>
 2023-03-14 13:48:32 +00:00
François Laignel
b9cd71d8eb net/aws/transcriber: fix eos not being sent 
For eos to be sent from the srcpad task loop, we need to go through `dequeue`.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1122>
 2023-03-09 13:07:03 +01:00
François Laignel
2ea9f147ab net/aws/transcriber: fix deadlock when the pipeline is interrupted 
... also makes sure to abort the taks_iter Future.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1122>
 2023-03-09 13:07:03 +01:00
François Laignel
4a988aaeb8 net/aws/transcriber: use a TranscriberLoop struct 
This helps gather together the details related to the `TranscriberLoop`.
One difference with previous implementation is that the ws `Client` is
build each time the loop is started instead of being reused. With the new
approach, we don't keep the connection open after EOS and we should be
more resistant in case of a connection failure.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1104>
 2023-03-01 08:47:58 +00:00
François Laignel
f1a080c94e net/aws/transcriber: own transcription items 
So that we can avoid copying the content.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1104>
 2023-03-01 08:47:58 +00:00
François Laignel
36ae29d746 net/aws: enqueue transcribed buffers within the ws loop 
Instead of sending transcription events to the src pad loop, this commit
enqueues the transcribed buffers immediately in the ws loop, then notifies
the src pad loop. The src pad loop is only in charge of dequeuing the buffers.

This should help with upcoming evolutions.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1104>
 2023-03-01 08:47:58 +00:00
François Laignel
00153754bb net/aws: use aws-sdk-transcribestreaming 
Switch from manual webservice client impl to `aws-sdk-transcribestreaming`.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1104>
 2023-03-01 08:47:58 +00:00