Alex Auvolat
f8b0817ddc
api: streaming signature: fix trailer parsing
2025-02-18 12:00:41 +01:00
Alex Auvolat
21c0dda16a
api: refactor: move checksumming code around again
2025-02-17 20:11:06 +01:00
Alex Auvolat
658541d812
api: use checksumming in api_common::signature for put/putpart
2025-02-17 19:54:25 +01:00
Alex Auvolat
c5df820e2c
api: start refactor of signature to calculate checksums earlier
2025-02-17 18:47:06 +01:00
Alex Auvolat
a04d6cd5b8
api: streaming: parse unsigned streaming bodies and payload trailers
2025-02-17 16:23:24 +01:00
Alex Auvolat
44a896f9b5
api: add logic to parse x-amz-content-sha256
2025-02-16 18:25:35 +01:00
Alex Auvolat
cee7560fc1
api: refactor: move checksum algorithms to common
2025-02-16 17:25:55 +01:00
Alex Auvolat
2f0c5ca220
signature: refactor: move constant defs to mod.rs
2025-02-16 16:34:18 +01:00
Alex
859b38b0d2
Merge pull request 'fix compilation warnings' ( #959 ) from fixes into main
...
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/959
2025-02-14 17:32:30 +00:00
Alex Auvolat
2729a71d9d
fix warning in garage test
2025-02-14 18:27:00 +01:00
Alex Auvolat
c9d00f5f7b
garage_api_s3: remove unused field in ListPartsQuery
2025-02-14 18:25:23 +01:00
Alex Auvolat
24470377c9
garage_model: fix warning about dead code
2025-02-14 18:12:14 +01:00
Alex Auvolat
5b26545abf
fix deprecated uses of chrono in lifecycle worker
2025-02-14 18:08:23 +01:00
Alex Auvolat
165f9316e2
s3api: return Location in CompleteMultipartUpload ( fix #852 )
...
NB. The location returned is not guaranteed to work in all cases.
This already fixes the parse issue in #852 .
2025-02-14 18:05:07 +01:00
Alex Auvolat
e4c9a8cd53
block manager: avoid deadlock in fix_block_location ( fix #845 )
2025-02-14 17:41:50 +01:00
Alex Auvolat
fdf4dad728
block resync: avoid saving blocks to draining nodes
2025-02-14 16:45:55 +01:00
Alex Auvolat
6820b69f30
block manager: improve read strategy to find blocks faster
2025-02-14 16:45:55 +01:00
Alex Auvolat
d0104b9f9b
block manager: write blocks only to currently active layout version ( fix #815 )
...
avoid wastefully writing blocks to nodes that will discard them as soon
as the layout migration is finished
2025-02-14 16:45:55 +01:00
Alex
3fe8db9e52
Merge pull request 'web_server.rs: Added bucket domain to observability' ( #608 ) from jpds/garage:domain-web-requests into main
...
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/608
2025-02-14 14:26:08 +00:00
Alex
627a37fe9f
Merge pull request 's3 api: parse x-id query parameter and warn of any inconsistency ( fix #822 )' ( #954 ) from fix-822 into main
...
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/954
2025-02-14 14:07:01 +00:00
Alex Auvolat
2f55889835
add configuration option to enable/disable monitoring bucket in web metrics
2025-02-14 14:59:00 +01:00
Jonathan Davies
8b9cc5ca3f
web_server.rs: Added bucket domain to observability.
2025-02-14 14:36:20 +01:00
Alex Auvolat
c1b39d9ba1
s3 api: parse x-id query parameter and warn of any inconsistency ( fix #822 )
2025-02-14 14:30:58 +01:00
Alex Auvolat
d84308c413
cli: return info of all nodes when doing garage stats -a ( fix #814 )
2025-02-14 14:11:41 +01:00
Alex Auvolat
62a3003cca
rename Condition into RedirectCondition in internal model
2025-02-14 13:45:32 +01:00
Alex Auvolat
3151695011
Merge branch 'next-v2' into 1686a/s3-redirects
2025-02-14 13:11:33 +01:00
trinity-1686a
f034e834fa
Merge branch 'main' into 1686a/s3-redirects
2025-02-08 16:01:11 +01:00
trinity-1686a
bf0f792418
add redirect_all to WebsiteConfig model
2025-02-08 15:36:48 +01:00
Baptiste Jonglez
a2e134f036
db-snapshot: propagate any node snapshot error through RPC call
...
In particular, it means that "garage meta snapshot --all" will get an exit
code of 1 if any node fails to snapshot.
This makes sure that any external tool trying to snapshot nodes (e.g. from
cron) will be aware of the failure.
Fix #920
2025-02-07 00:29:43 +01:00
Baptiste Jonglez
06aa4b604f
db-snapshot: Fix error reporting when using "garage meta snapshot --all"
...
Snapshot errors on remote nodes were not reported at all.
We now get proper error output such as:
0fa0f35be69528ab error: Internal error: DB error: LMDB: No space left on device (os error 28)
88d92e2971d14bae ok
Fix #920
2025-02-07 00:18:01 +01:00
Alex Auvolat
61f3de6496
Merge branch 'main' into next-v2
2025-02-05 21:00:19 +01:00
Alex Auvolat
af67626ab2
remove async_trait for TableRepair
2025-02-05 20:45:07 +01:00
Alex Auvolat
5475da8ea8
remove async_trait used in generic_server.rs
2025-02-05 20:31:34 +01:00
Alex Auvolat
620dc58560
remove async_trait for traits declared in garage_net
2025-02-05 20:22:16 +01:00
Alex Auvolat
7c8fc04b96
massively speed up compilation of garage_api_admin by not using async_trait
2025-02-05 19:37:38 +01:00
Alex Auvolat
f914db057a
cli_v2: implement LaunchRepairOperation and remove old stuff
2025-02-05 15:36:47 +01:00
Alex Auvolat
406b6da163
cli_v2: implement Get{Node,Cluster}Statistics
2025-02-05 15:06:10 +01:00
Alex Auvolat
9f468b4439
cli_v2: implement CreateMetadataSnapshot
2025-02-05 14:22:10 +01:00
Alex Auvolat
97be7b38fa
admin api: reorder things
2025-02-03 18:54:51 +01:00
Alex Auvolat
6a1079c412
admin api: impl RequestHandler for MetricsRequest
2025-02-03 18:54:51 +01:00
Alex Auvolat
b1629dd355
cli_v2: implement RetryBlockResync and PurgeBlocks
2025-02-03 18:54:51 +01:00
Alex Auvolat
d405a9f839
cli_v2: implement ListBlockErrors and GetBlockInfo
2025-02-03 18:54:51 +01:00
Alex Auvolat
7b9c047b11
cli_v2: add local_api_request with crazy type bound
2025-02-03 18:54:51 +01:00
Alex Auvolat
10bbb26b30
cli_v2: implement ListWorkers and GetWorkerInfo
2025-02-03 18:54:51 +01:00
Alex Auvolat
89ff9f5576
admin api: base infrastructure for local endpoints
...
admin api: rename EndpointHandler into RequestHandler to avoid confusion with RPC
wip: infrastructure for local api calls
admin api: fix things
admin api: first local endpoint to work with new scheme
admin api: implement SetWorkerVariable
2025-02-03 18:54:51 +01:00
Alex Auvolat
bdaf55ab3f
cli_v2: migrate cleanupincompleteuploads to Admin API
...
admin api: add CleanupIncompleteUploads spec
2025-02-03 18:54:51 +01:00
Alex Auvolat
ec1a475923
build with rust 1.82.0
2025-02-03 17:46:48 +01:00
Alex Auvolat
390a5d97fe
nix, ci: build with Crane
...
This removes our dependency on cargo2nix, which was causing us some
issues. Whereas cargo2nix creates one Nix derivation per crate, Crane
uses only two derivations:
1. Build dependencies only
2. Build the final binary
This means that during the second step, no caching can be done. For
instance, if we do a change in garage_model, we need to recompile all of
the Garage crates including those that do not depend on garage_model.
On the upside, this allows all of the Garage crates to be built at once
using cargo build logic, which is optimized for high parallelism and
better pipelining between all of the steps of the build. All in all,
this makes most builds faster than cargo2nix.
A few other changes have been made to the build scripts and CI:
- Unit tests are now run within a Nix derivation. In fact, we have
different derivations to run the tests using LMDB and Sqlite as
metadata db engines.
- For debug builds, most CI steps now run in parallel (with the notable
exception of the smoke test that runs after the build, which is
inevitable).
- We no longer pass the GIT_VERSION argument when building debug builds
and running the tests. This means that dev binaries and test
binaries don't know the exact version of Garage they are from. That
shouldn't be an issue in most cases.
- The not-dynamic.sh scripts has been fixed to fail if the file does not
exist.
2025-02-03 16:39:50 +01:00
Alex Auvolat
fe937c2901
Merge branch 'main' into next-v2
2025-02-01 19:07:17 +01:00
Alex Auvolat
d18c5ad0ff
fix tests
2025-01-31 19:12:51 +01:00
Alex Auvolat
3d5e9a027e
cargo defs: simplify and fix descriptions
2025-01-31 18:54:29 +01:00
Alex Auvolat
4563313f87
use cargo-shear to remove many unused dependencies between crates
2025-01-31 18:47:30 +01:00
Alex Auvolat
afa28706e5
split s3/cors.rs into also common/cors.rs
2025-01-31 18:42:14 +01:00
Alex Auvolat
84f1db91c4
fix things up
2025-01-31 18:34:57 +01:00
Alex Auvolat
9fa20d45be
wip: split garage_api into garage_api_{common,s3,k2v,admin}
2025-01-31 18:18:29 +01:00
Alex Auvolat
83f6928ff7
table::insert_many: avoid failure with zero items ( fix #915 )
2025-01-30 18:06:47 +01:00
Alex Auvolat
5a89350b38
cli_v2: fix garage status
2025-01-30 16:45:59 +01:00
Alex Auvolat
3caea5fc06
cli_v2: merge util.rs into layout.rs
2025-01-30 16:24:55 +01:00
Alex Auvolat
ebc0e9319e
cli_v2: error messages
2025-01-30 16:17:35 +01:00
Alex Auvolat
f8c6a8373d
convert cli key operations to admin rpc
2025-01-30 16:12:16 +01:00
Alex Auvolat
076ce04fe5
fix garage status output
2025-01-30 15:38:22 +01:00
Alex Auvolat
f37d5d2b08
admin api: convert most bucket operations
2025-01-30 13:36:25 +01:00
Alex Auvolat
819f4f0050
cli: migrate layout remove, apply, revert
2025-01-30 12:19:23 +01:00
Alex Auvolat
69ddaafc60
wip: migrate garage status and garage layout assign
2025-01-30 12:07:12 +01:00
Alex Auvolat
145130481e
wip: proxy admin api requests through admin rpc, prepare new cli
2025-01-30 10:44:08 +01:00
Alex Auvolat
19454c1679
admin api: remove log message
2025-01-29 19:47:37 +01:00
Alex Auvolat
1c03941b19
admin api: fix panic on GetKeyInfo with no args
2025-01-29 19:26:16 +01:00
Alex Auvolat
4f0b923c4f
admin api: small fixes
2025-01-29 19:26:16 +01:00
Alex Auvolat
420bbc162d
admin api: clearer syntax for AddBucketAlias and RemoveBucketAlias
2025-01-29 19:26:16 +01:00
Alex Auvolat
12ea4cda5f
admin api: merge calls to manage global/local aliases
2025-01-29 19:26:16 +01:00
Alex Auvolat
5fefbd94e9
admin api: rename allow/deny api calls in api v2
2025-01-29 19:26:16 +01:00
Alex Auvolat
ba810b2e81
admin api: rename bucket aliasing operations
2025-01-29 19:26:16 +01:00
Alex Auvolat
f8ed3fdbc4
fix test_website_check_domain
2025-01-29 19:26:16 +01:00
Alex Auvolat
4cb45bd398
admin api: fix CORS to work in browser
2025-01-29 19:26:16 +01:00
Alex Auvolat
f538dc34d3
admin api: make all requests and responses (de)serializable
2025-01-29 19:26:16 +01:00
Alex Auvolat
ed58f8b0fe
admin api: update semantics of some endpoints, and update doc
2025-01-29 19:26:16 +01:00
Alex Auvolat
5037b97dd4
admin api: add compatibility from v1/ to v2/
2025-01-29 19:26:16 +01:00
Alex Auvolat
af1a530834
admin api: refactor using macro
2025-01-29 19:26:16 +01:00
Alex Auvolat
c99bfe69ea
admin api: new router_v2 with unified path syntax
2025-01-29 19:26:16 +01:00
Alex Auvolat
831f2b0207
admin api: make all handlers impls of a single trait
2025-01-29 19:26:16 +01:00
Alex Auvolat
c1eb1610ba
admin api: create structs for all requests/responess in src/api/admin/api.rs
2025-01-29 19:26:16 +01:00
Quentin Dufour
5560a963e0
decrease write quorum
2025-01-29 19:25:58 +01:00
Alex Auvolat
9f3c7c3720
api: better handling of helper errors to distinguish error codes
2025-01-29 19:14:34 +01:00
Alex
a1d081ee84
Merge pull request 's3 api: make x-amz-meta-* headers lowercase ( fix #844 )' ( #938 ) from fix-844 into main
...
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/938
2025-01-27 19:32:19 +00:00
Alex Auvolat
e8fa89e834
s3 api: make x-amz-meta-* headers lowercase ( fix #844 )
2025-01-27 19:58:06 +01:00
Baptiste Jonglez
6d798c640f
WIP: fix crash in layout computation when changing all nodes of a zone to gateway mode
...
This change is probably not a proper fix, somebody with more expertise on
this code should look at it.
Here is how to reproduce the crash:
- start with a layout with two zones
- move all nodes of a zone to gateway mode: `garage layout assign fea54bcc081f318 -g`
- `garage layout show` will panic with a backtrace
Fortunately, the crash is only on the RPC client side, not on the Garage
server itself, and `garage layout revert` still works to go back to the
previous state.
As far as I can tell, this bug is present since Garage 0.9.0 which
includes the new layout assignation algorithm:
https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/296
2025-01-27 19:33:57 +01:00
Baptiste Jonglez
43402c9619
snapshot: sqlite: use a subdirectory for consistency with LMDB
...
Currently, taking a snapshot of the metadata database with sqlite creates
a sqlite file without extension with the following format:
snapshots/2025-01-26T15:29:17Z
This makes it hard to understand what kind of data this is, and is not
consistent with LMDB:
snapshots/2025-01-26T15:29:17Z/data.mdb
With this change, we now get a directory with a single db.sqlite file:
snapshots/2025-01-26T15:29:17Z/db.sqlite
2025-01-27 19:06:52 +01:00
Baptiste Jonglez
59c153d280
db-snapshot: allow to set directory where snapshots are stored
...
Fix #926
2025-01-27 18:33:55 +01:00
Stefan Majer
2eb9fcae20
Fix all typos
2025-01-16 13:22:00 +01:00
Alex Auvolat
295237476e
fix formatting to comply with latest rustfmt
2025-01-12 17:36:25 +01:00
Alex Auvolat
2aaba39ddc
refactor web_server.rs
2025-01-04 20:11:54 +01:00
Alex Auvolat
47467df83e
avoid handling status_code-related logic in api/s3/get.rs
2025-01-04 19:52:14 +01:00
Alex Auvolat
9b7fea4cb0
put bucket website: improve error message for redirectallrequests
2025-01-04 19:16:24 +01:00
Alex Auvolat
44ce6ae5b4
properly implement new bucket model using a migration
2025-01-04 18:50:49 +01:00
Alex Auvolat
22487ceddf
move Redirect::compute_target to standalone function in web_server.rs
2025-01-04 18:22:42 +01:00
Alex Auvolat
6ccfbb2986
remove obsolete RedirectAll struct
2025-01-04 17:04:17 +01:00
Vedad KAJTAZ
6689800986
Formatting with
2025-01-04 16:52:23 +01:00
Vedad KAJTAZ
6ca99fd02c
formatting
2025-01-04 14:46:42 +01:00
Vedad KAJTAZ
b568bb863d
Fix #907
2025-01-04 12:50:10 +01:00
trinity-1686a
c939d2a936
clippy
2024-12-22 15:26:06 +01:00
