Commit graph

33 commits

Author SHA1 Message Date
Sanchayan Maity
768fad2445 Migrate s3src and s3sink to use AWS SDK
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/759>
2022-06-14 08:03:49 +00:00
Arun Raghavan
09a697faef rusoto: s3sink: Expose property to control all timeout/retry durations
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/690>
2022-03-18 16:03:11 +05:30
Arun Raghavan
94c5cbbfb8 rusoto: s3sink: Make remaining requests bounded in time
This implements a default timeout and retry duration for the remaining
S3 requests that were still able to be blocked indefinitely. There are 3
classes of operations: multipart upload creation/abort (should not take
too long), uploads (duration depends on part size), multipart upload
completion (can take several minutes according to documentation).

We currently only expose the part upload times as configurable, and hard
code the rest. If it seems sensible, we can expose the other two sets of
parameters as well.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/690>
2022-03-18 16:03:10 +05:30
Arun Raghavan
5fe95afe87 s3src: Consolidate stream reading into get object retries
Previously, the actual reading from the streaming body of a GetObject
request was not within the same timeout/retry path as the dispatch of
the HTTP request itself. We consolidate these two into a single async
block and create a sum type to encapsulate the rusoto and std library
error paths within that future.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/690>
2022-03-18 16:02:20 +05:30
Sebastian Dröge
b38f6cc731 Remove now unnecessary Send+Sync impls for element/etc subclasses
This is now automatically implemented.
2022-02-28 18:56:58 +02:00
Arun Raghavan
c2aafa4f46 rusoto: s3sink: Implement timeout/retry for part uploads
Rusoto does not implement timeouts or retries for any of its HTTP
requests. This is particularly problematic for the part upload stage of
multipart uploads, as a blip in the network could cause part uploads to
freeze for a long duration and eventually bail.

To avoid this, for part uploads, we add (a) (configurable) timeouts for
each request, and (b) retries with exponential backoff, upto a
configurable duration.

It is not clear if/how we want to do this for other types of requests.
The creation of a multipart upload should be relatively quick, but the
completion of an upload might take several minutes, so there is no
one-size-fits-all configuration, necessarily.

It would likely make more sense to implement some sensible hard-coded
defaults for these other sorts of requests.
2022-02-23 13:53:39 -05:00
François Laignel
422ea740ca Update to gst::_log_macro_
See the details:
https://gitlab.freedesktop.org/gstreamer/gstreamer-rs/-/merge_requests/980
2022-02-21 20:50:01 +01:00
Sebastian Dröge
326449b3e6 Re-license LGPL-2.1 plugins to MPL-2
Fixes https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/issues/168
2022-01-15 21:05:11 +02:00
Sanchayan Maity
099a3f2114 rusoto: s3sink: Support aborting or completing multipart upload on error
A multipart upload should either be completed or aborted on error. In
the current state of things, a multipart upload would neither be
completed nor aborted, putting the onus on an external entity to take
care of finishing incomplete uploads or relying on a sane bucket
life cycle policy configured to abort incomplete multipart uploads.

An incomplete multipart upload still contributes to the storage costs as
long as it exists.

We introduce a property here to allow the user to select either aborting
or completing multipart uploads on error. Aborting the upload causes
whole of data to be discarded and the same upload ID is not usable for
uploading more parts to the same.

Completing an incomplete multipart upload can be useful in situations
like having a streamable MP4 where one might want to complete the upload
and have part of the data which was uploaded be preserved.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/618>
2021-12-07 18:29:52 +05:30
Guillaume Desmottes
0b348406ef s3sink: add metadata property
This property can be used to set metadata on the S3 storage object.
2021-11-22 17:03:24 +01:00
Guillaume Desmottes
11bef9066c s3sink: log when setting properties 2021-11-22 16:52:04 +01:00
Sebastian Dröge
55aad51141 Update for glib constructor renames
See https://github.com/gtk-rs/gtk-rs-core/pull/384
2021-11-20 14:31:06 +02:00
Sebastian Dröge
0a7d1639e7 Update to Rust edition 2021 and minimum supported Rust version to 1.56 2021-10-31 17:40:05 +02:00
Sebastian Dröge
b9541b2ca4 Update for GstObjectImpl API change 2021-10-23 12:31:33 +03:00
Sebastian Dröge
69bb09f7ad rusoto/s3: Allow passing custom AWS-compatible regions
For the region property this would be provided as
    `region-name+https://region.end/point`
while for the URI this unfortunately has to be base32 encoded to allow
usage as the host part of the URI.
2021-09-28 06:23:07 +00:00
Sebastian Dröge
502b336361 rusoto: Implement auth via explicit access-key/secret-access-key properties
This allows passing them explicitly as strings to the elements instead
of relying on system/per-user configuration.
2021-09-27 17:00:36 +03:00
Arun Raghavan
3a2d16f00c rusoto: s3sink: Bring back bucket, key and region properties
We don't want to drop these entirely while introducing the URI handler,
as that would break backwards compatibility.
2021-06-21 06:58:06 -04:00
François Laignel
5439f14e57 fix clippy warnings 2021-06-05 10:36:22 +02:00
François Laignel
8dfc872544 use gst::glib where applicable 2021-06-03 20:53:16 +02:00
Sanchayan Maity
bf5e231e5b rusoto: s3sink: Implement support for GstUriHandler interface
With the URI handler interface implemented, we can drop the old method
of specifying bucket, key and region. This also brings it in line with
how it is for s3src.
2021-05-01 13:55:49 +05:30
Sebastian Dröge
15cf738616 Update for Value trait refactoring 2021-04-25 15:48:55 +03:00
François Laignel
27bc5c89ca fix-getters-def 0.3.0 pass 2021-04-20 18:19:58 +02:00
François Laignel
7d17f88941 post fix-getters manual updates 2021-04-13 17:24:20 +02:00
François Laignel
06accc8d98 fix-getters-{def,calls} pass 2021-04-12 15:57:19 +02:00
Sebastian Dröge
2cada57efc Update for the subclassing glib/gstreamer bindings API changes 2021-03-09 17:07:13 +02:00
Sebastian Dröge
dc0c5f7611 Update for new #[glib::object_subclass] attribute macro 2021-03-07 18:27:00 +02:00
Sebastian Dröge
b649e9b076 Use gst::PARAM_FLAG_MUTABLE_PLAYING and others consistently everywhere
Fixes https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/issues/139
2021-01-31 15:43:00 +02:00
Sebastian Dröge
d4ce1a33f2 Update for glib/gstreamer bindings API changes 2021-01-25 14:43:05 +02:00
Sebastian Dröge
3d617371af Update for macro renames 2020-12-20 20:43:45 +02:00
Sebastian Dröge
ea6c05e16c Update everything for glib macro renamings 2020-12-18 00:44:49 +02:00
Sebastian Dröge
3c9f1c0d1d net: Update to 2018 edition 2020-11-23 10:28:33 +02:00
Sebastian Dröge
d56ae71e0e Update for ObjectImpl::get_property() being infallible now 2020-11-19 18:25:53 +02:00
Sebastian Dröge
d7404a7e1c net: Update for subclassing API changes 2020-11-15 18:25:42 +02:00