Alex Auvolat
3d5e9a027e
cargo defs: simplify and fix descriptions
2025-01-31 18:54:29 +01:00
Alex Auvolat
4563313f87
use cargo-shear to remove many unused dependencies between crates
2025-01-31 18:47:30 +01:00
Alex Auvolat
afa28706e5
split s3/cors.rs into also common/cors.rs
2025-01-31 18:42:14 +01:00
Alex Auvolat
84f1db91c4
fix things up
2025-01-31 18:34:57 +01:00
Alex Auvolat
9fa20d45be
wip: split garage_api into garage_api_{common,s3,k2v,admin}
2025-01-31 18:18:29 +01:00
Alex Auvolat
83f6928ff7
table::insert_many: avoid failure with zero items ( fix #915 )
2025-01-30 18:06:47 +01:00
Alex Auvolat
5a89350b38
cli_v2: fix garage status
2025-01-30 16:45:59 +01:00
Alex Auvolat
3caea5fc06
cli_v2: merge util.rs into layout.rs
2025-01-30 16:24:55 +01:00
Alex Auvolat
ebc0e9319e
cli_v2: error messages
2025-01-30 16:17:35 +01:00
Alex Auvolat
f8c6a8373d
convert cli key operations to admin rpc
2025-01-30 16:12:16 +01:00
Alex Auvolat
076ce04fe5
fix garage status output
2025-01-30 15:38:22 +01:00
Alex Auvolat
f37d5d2b08
admin api: convert most bucket operations
2025-01-30 13:36:25 +01:00
Alex Auvolat
819f4f0050
cli: migrate layout remove, apply, revert
2025-01-30 12:19:23 +01:00
Alex Auvolat
69ddaafc60
wip: migrate garage status and garage layout assign
2025-01-30 12:07:12 +01:00
Alex Auvolat
145130481e
wip: proxy admin api requests through admin rpc, prepare new cli
2025-01-30 10:44:08 +01:00
Alex Auvolat
19454c1679
admin api: remove log message
2025-01-29 19:47:37 +01:00
Alex Auvolat
1c03941b19
admin api: fix panic on GetKeyInfo with no args
2025-01-29 19:26:16 +01:00
Alex Auvolat
4f0b923c4f
admin api: small fixes
2025-01-29 19:26:16 +01:00
Alex Auvolat
420bbc162d
admin api: clearer syntax for AddBucketAlias and RemoveBucketAlias
2025-01-29 19:26:16 +01:00
Alex Auvolat
12ea4cda5f
admin api: merge calls to manage global/local aliases
2025-01-29 19:26:16 +01:00
Alex Auvolat
5fefbd94e9
admin api: rename allow/deny api calls in api v2
2025-01-29 19:26:16 +01:00
Alex Auvolat
ba810b2e81
admin api: rename bucket aliasing operations
2025-01-29 19:26:16 +01:00
Alex Auvolat
f8ed3fdbc4
fix test_website_check_domain
2025-01-29 19:26:16 +01:00
Alex Auvolat
4cb45bd398
admin api: fix CORS to work in browser
2025-01-29 19:26:16 +01:00
Alex Auvolat
f538dc34d3
admin api: make all requests and responses (de)serializable
2025-01-29 19:26:16 +01:00
Alex Auvolat
ed58f8b0fe
admin api: update semantics of some endpoints, and update doc
2025-01-29 19:26:16 +01:00
Alex Auvolat
5037b97dd4
admin api: add compatibility from v1/ to v2/
2025-01-29 19:26:16 +01:00
Alex Auvolat
af1a530834
admin api: refactor using macro
2025-01-29 19:26:16 +01:00
Alex Auvolat
c99bfe69ea
admin api: new router_v2 with unified path syntax
2025-01-29 19:26:16 +01:00
Alex Auvolat
831f2b0207
admin api: make all handlers impls of a single trait
2025-01-29 19:26:16 +01:00
Alex Auvolat
c1eb1610ba
admin api: create structs for all requests/responess in src/api/admin/api.rs
2025-01-29 19:26:16 +01:00
Quentin Dufour
5560a963e0
decrease write quorum
2025-01-29 19:25:58 +01:00
Alex Auvolat
9f3c7c3720
api: better handling of helper errors to distinguish error codes
2025-01-29 19:14:34 +01:00
Alex
a1d081ee84
Merge pull request 's3 api: make x-amz-meta-* headers lowercase ( fix #844 )' ( #938 ) from fix-844 into main
...
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/938
2025-01-27 19:32:19 +00:00
Alex Auvolat
e8fa89e834
s3 api: make x-amz-meta-* headers lowercase ( fix #844 )
2025-01-27 19:58:06 +01:00
Baptiste Jonglez
6d798c640f
WIP: fix crash in layout computation when changing all nodes of a zone to gateway mode
...
This change is probably not a proper fix, somebody with more expertise on
this code should look at it.
Here is how to reproduce the crash:
- start with a layout with two zones
- move all nodes of a zone to gateway mode: `garage layout assign fea54bcc081f318 -g`
- `garage layout show` will panic with a backtrace
Fortunately, the crash is only on the RPC client side, not on the Garage
server itself, and `garage layout revert` still works to go back to the
previous state.
As far as I can tell, this bug is present since Garage 0.9.0 which
includes the new layout assignation algorithm:
https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/296
2025-01-27 19:33:57 +01:00
Baptiste Jonglez
43402c9619
snapshot: sqlite: use a subdirectory for consistency with LMDB
...
Currently, taking a snapshot of the metadata database with sqlite creates
a sqlite file without extension with the following format:
snapshots/2025-01-26T15:29:17Z
This makes it hard to understand what kind of data this is, and is not
consistent with LMDB:
snapshots/2025-01-26T15:29:17Z/data.mdb
With this change, we now get a directory with a single db.sqlite file:
snapshots/2025-01-26T15:29:17Z/db.sqlite
2025-01-27 19:06:52 +01:00
Baptiste Jonglez
59c153d280
db-snapshot: allow to set directory where snapshots are stored
...
Fix #926
2025-01-27 18:33:55 +01:00
Stefan Majer
2eb9fcae20
Fix all typos
2025-01-16 13:22:00 +01:00
Alex Auvolat
295237476e
fix formatting to comply with latest rustfmt
2025-01-12 17:36:25 +01:00
Alex Auvolat
2aaba39ddc
refactor web_server.rs
2025-01-04 20:11:54 +01:00
Alex Auvolat
47467df83e
avoid handling status_code-related logic in api/s3/get.rs
2025-01-04 19:52:14 +01:00
Alex Auvolat
9b7fea4cb0
put bucket website: improve error message for redirectallrequests
2025-01-04 19:16:24 +01:00
Alex Auvolat
44ce6ae5b4
properly implement new bucket model using a migration
2025-01-04 18:50:49 +01:00
Alex Auvolat
22487ceddf
move Redirect::compute_target to standalone function in web_server.rs
2025-01-04 18:22:42 +01:00
Alex Auvolat
6ccfbb2986
remove obsolete RedirectAll struct
2025-01-04 17:04:17 +01:00
Vedad KAJTAZ
6689800986
Formatting with
2025-01-04 16:52:23 +01:00
Vedad KAJTAZ
6ca99fd02c
formatting
2025-01-04 14:46:42 +01:00
Vedad KAJTAZ
b568bb863d
Fix #907
2025-01-04 12:50:10 +01:00
trinity-1686a
c939d2a936
clippy
2024-12-22 15:26:06 +01:00
trinity-1686a
65e9dde8c9
add tests
2024-12-22 15:20:09 +01:00
trinity-1686a
c9b733a4a6
support redirection on s3 endpoint
2024-12-14 17:46:27 +01:00
Renjaya Raga Zenta
4c1bf42192
feat: add use_local_tz configuration
...
Used in lifecycle_worker to determine midnight time
2024-11-23 05:51:12 +07:00
trinity-1686a
9bd9e392ba
fix bit/byte inversion in rpc secret error message
2024-11-07 00:29:26 +01:00
Alex Auvolat
7a143f46fc
Bump to version 1.0.1
2024-09-22 14:25:32 +02:00
Alex Auvolat
34453bc9c2
fix logic in garage layout skip-dead-nodes + fix typo ( fix #879 )
2024-09-22 13:47:27 +02:00
Julien Kritter
bd71728874
Tests: don't expect old value after transactional insert
2024-09-12 10:50:53 +02:00
Julien Kritter
51ced60366
Don't fetch old values in cross-partition transactional inserts
2024-09-12 10:26:28 +02:00
marvin-j97
8062ec7b4b
test: fix db tests
2024-09-04 19:24:36 +02:00
marvin-j97
eb416a02fb
dont assert deletion count in sqlite KV adapter
2024-09-04 18:51:51 +02:00
marvin-j97
74363c9060
perf(kv): dont retrieve values for write ops
...
see https://git.deuxfleurs.fr/Deuxfleurs/garage/issues/851
2024-09-04 18:45:17 +02:00
Alex Auvolat
baf32c9575
api servers: kill opened connections after SIGINT after 10s deadline ( fix #806 )
2024-08-25 20:04:56 +02:00
Alex Auvolat
aa7ce9e97c
fix build when lmdb feature is disabled ( fix #800 )
2024-08-25 11:42:37 +02:00
Alex
8d62616ec0
Merge pull request 'layout: discard old info when it is completely out-of-date ( fix #841 )' ( #861 ) from fix-841 into main
...
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/861
2024-08-24 11:12:39 +00:00
Alex
3e711bc110
Merge pull request 'don't modify postobject request before validating policy' ( #850 ) from trinity-1686a/garage:fix-acl-postobject into main
...
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/850
2024-08-24 10:49:14 +00:00
Alex Auvolat
7fb66b4944
layout: discard old info when it is completely out-of-date ( fix #841 )
2024-08-24 12:38:56 +02:00
Quentin Dufour
2a93ad0c84
force flag "no read ahead" on LMDB
2024-08-17 21:17:15 +02:00
trinity-1686a
f190032589
don't modify postobject request before validating policy
2024-08-10 20:10:47 +02:00
Maximilien R.
9302cd42f0
Improve error message for malformed RPC secret key
2024-08-08 23:05:24 +00:00
Florian Klink
a0f6bc5b7f
add rpc_public_addr_subnet config option
...
In case `rpc_public_addr` is not set, but autodiscovery is used, this
allows filtering the list of automatically discovered IPs to a specific
subnet.
For example, if nodes should pick *their* IP inside a specific subnet,
but you don't want to explicitly write the IP down (as it's dynamic, or
you want to share configs across nodes), you can use this option.
2024-06-05 08:41:36 +02:00
Florian Klink
15847a636a
cli: clarify garage block is node-local
...
Prevents some of the confusion from
https://git.deuxfleurs.fr/Deuxfleurs/garage/issues/810 .
2024-05-07 07:42:33 +00:00
Alex Auvolat
90e3c2af91
[next-0.10] small updates to mention Garage v0.9.4
2024-04-10 14:35:30 +02:00
Alex Auvolat
afad62939e
[next-0.10] bump version number to 1.0
2024-03-28 15:19:44 +01:00
Alex Auvolat
8bfc16ba7d
Merge branch 'main' into next-0.10
2024-03-28 15:01:05 +01:00
Alex Auvolat
85f580cbde
[fix-buffering] change request sending strategy and fix priorities
...
remove LAS, priorize new requests but otherwise just do standard queuing
2024-03-27 16:22:40 +01:00
Alex Auvolat
0d3e285d13
[fix-buffering] implement block_ram_buffer_max to avoid excessive RAM usage
2024-03-27 16:22:40 +01:00
Alex Auvolat
25c196f34d
[next-0.10] admin api: fix logic in get cluster status
2024-03-27 13:55:49 +01:00
Alex Auvolat
4eba32f29f
[next-0.10] layout helper: rename & clarify updates to update trackers
2024-03-27 13:47:06 +01:00
Alex Auvolat
32f1786f9f
[next-0.10] cache layout check result
2024-03-27 13:37:20 +01:00
Alex Auvolat
01a0bd5410
[next-0.10] remove impl Deref for LayoutHelper
2024-03-27 13:32:13 +01:00
Alex Auvolat
c0eeb0b0f3
[next-0.10] fixes to k2v rpc + comment fixes
2024-03-27 10:44:03 +01:00
Alex Auvolat
f7cd4eb600
Merge branch 'main' into next-0.10
2024-03-26 16:34:40 +01:00
Alex Auvolat
73551e9a2d
[disable-k2v-test] disable the other k2v poll test
2024-03-26 16:24:26 +01:00
Alex Auvolat
f267609343
[disable-k2v-test] disable tests::k2v::test_poll_item as it is not 100% reliable
2024-03-26 15:39:17 +01:00
Alex Auvolat
74949c69cb
[s3-checksum] implement x-amz-checksum-* headers
2024-03-26 15:01:34 +01:00
Alex Auvolat
3844110cd0
[net-fixes] netapp peer exchange: send only currently connected address
2024-03-21 10:50:44 +01:00
Alex Auvolat
961b4f9af3
[net-fixes] fix issues with local peer address ( fix #761 )
2024-03-21 10:45:34 +01:00
Alex Auvolat
5225a81dee
[net-fixes] peering: only count node IDs and not addresses in hash
2024-03-21 09:47:04 +01:00
Alex
e835196940
Merge pull request 'Add marker files in data directories ( fix #601 )' ( #785 ) from check-data-dir into main
...
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/785
2024-03-20 16:53:47 +00:00
Alex Auvolat
ba33bb31f1
[check-data-dir] add marker files in data directories ( fix #601 )
2024-03-20 15:20:25 +01:00
Alex Auvolat
84018be862
[syslog] warning when syslog support is not enabled
2024-03-20 14:39:04 +01:00
Alex Auvolat
fe8a7819fa
[syslog] Add support to logging to syslog
...
Original patch by Jakub Jirutka for Alpine Linux port.
2024-03-20 14:22:18 +01:00
Alex Auvolat
ce69dc302c
Merge branch 'main' into next-0.10
2024-03-19 17:17:46 +01:00
Alex Auvolat
783b586de9
[bucket-id-prefix] CLI: allow manipulating buckets by prefixes of their full IDs
2024-03-19 16:57:51 +01:00
Alex Auvolat
3eab639c14
[block-ref-repair] mention garage block repair-rc in documentation
2024-03-19 16:24:34 +01:00
Alex Auvolat
3165ab926c
[block-ref-repair] rename rc's rc field to rc_table
2024-03-19 16:20:22 +01:00
Alex Auvolat
dc0b78cdb8
[block-ref-repair] Block refcount recalculation and repair
...
- We always recalculate the reference count of a block before deleting
it locally, to make sure that it is indeed zero.
- If we had to fetch a remote block but we were not able to get it,
check that refcount is indeed > 0.
- Repair procedure that checks everything
2024-03-19 16:20:22 +01:00
Alex Auvolat
0038ca8a78
Merge branch 'main' into next-0.10
2024-03-18 20:19:30 +01:00
Alex Auvolat
b55f52a9b7
[sqlite-r2d2] run integration test with all db engines
2024-03-18 18:31:35 +01:00
Alex Auvolat
e8f9718ccd
[sqlite-r2d2] implement connection pooling in sqlite backend
2024-03-18 18:05:25 +01:00
Alex Auvolat
a68c37555d
[db-snapshot] add garage meta snapshot cli operation
2024-03-15 13:51:31 +01:00
Alex Auvolat
1e42808a59
[db-snapshot] implement meta_auto_snapshot_interval
2024-03-15 13:51:31 +01:00
Alex Auvolat
8dff278b72
[db-snapshot] Implement db snapshotting logic in garage_db
2024-03-15 10:57:22 +01:00
Alex Auvolat
7c86ff6c37
[disable-scrub] implement a disable_scrub configuration option
2024-03-14 17:01:16 +01:00
Alex
81191d2d92
Merge pull request 'Remove Sled' ( #767 ) from rm-sled into next-0.10
...
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/767
2024-03-12 10:45:57 +00:00
Alex Auvolat
2795b53b8b
[rm-sled] factorize some code in sqlite backend
2024-03-12 11:15:26 +01:00
Alex Auvolat
32aa246300
[rm-sled] Make proper use of pinning in LMDB adapter + comment unsafe
2024-03-08 17:39:17 +01:00
Alex Auvolat
b942949940
[rm-sled] Implement iterators in sqlite & lmdb transactions
...
with way too much unsafe code
2024-03-08 16:38:01 +01:00
Alex Auvolat
66c23890c1
[rm-sled] Implement some missing functionality in garage_db
2024-03-08 16:02:58 +01:00
Alex Auvolat
05c92204ec
[rm-sled] Remove counted_tree_hack
2024-03-08 15:09:57 +01:00
Alex
2128b5febd
Merge pull request 'Remove migration path from Garage v0.5' ( #766 ) from rm-migration into next-0.10
...
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/766
2024-03-08 13:43:42 +00:00
Alex Auvolat
44454aac01
[rm-sled] Remove the Sled database engine
2024-03-08 14:11:02 +01:00
Alex Auvolat
1ace34adbb
Merge branch 'main' into next-0.10
2024-03-08 13:57:10 +01:00
Alex Auvolat
f537f76681
[rm-migration] Remove migration path from Garage v0.5
2024-03-08 13:24:47 +01:00
Alex Auvolat
ec34728b27
[factor-db-open] Combine logic for opening db engines
2024-03-08 12:58:17 +01:00
Alex
20c0b4ffb2
Merge pull request 'ReplicationMode -> ConsistencyMode+ReplicationFactor' ( #750 ) from yuka/garage:split-consistency-mode into next-0.10
...
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/750
2024-03-07 16:32:52 +00:00
Alex Auvolat
3fcb54e3cf
[sse-c] Remove special case for Content-Type header
2024-03-07 15:43:48 +01:00
Alex Auvolat
fa4878bad6
[sse-c] Testing for SSE-C encryption
2024-03-07 15:43:48 +01:00
Alex Auvolat
57acc60082
[sse-c] Implement SSE-C encryption
2024-03-07 15:43:47 +01:00
Alex Auvolat
fe2dc5d51c
Merge branch 'main' into next-0.10
2024-03-07 14:00:34 +01:00
Alex Auvolat
eab2b81be2
[unicode-headers] allow utf-8 in headers + add test for object metadata
2024-03-07 13:42:01 +01:00
Yureka
c1769bbe69
ReplicationMode -> ConsistencyMode+ReplicationFactor
2024-03-07 12:45:33 +01:00
Yureka
6760895926
refactor: remove max_write_errors and max_faults
2024-03-04 18:39:56 +01:00
Alex Auvolat
bbde9bc912
Merge branch 'main' into next-0.10
2024-03-04 15:56:10 +01:00
Alex
3168bb34a0
Merge pull request 'add request context helper' ( #751 ) from yuka/garage:req-ctx into main
...
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/751
2024-03-04 14:51:05 +00:00
Alex Auvolat
8670140358
[rel-0.9.3] Bump version to 0.9.3
2024-03-04 14:00:55 +01:00
Alex Auvolat
c8e416aaa5
[test-presigned] Use a HeaderMap type for QueryMap
2024-03-04 13:33:14 +01:00
Yureka
fb55682c66
add request context helper
2024-03-04 13:26:39 +01:00
asonix
c94bf45cba
Store original-cased query keys alongside query values
2024-03-04 13:03:27 +01:00
asonix
7c4f3473af
Lowercase query parameter keys when parsing
2024-03-04 13:03:16 +01:00
Alex Auvolat
b6a91e549b
[test-presigned] Add API test for presigned requests
2024-03-04 13:02:07 +01:00
Alex Auvolat
6a7623e90d
[rel-0.9.2] Bump version to v0.9.2
2024-03-01 16:54:39 +01:00
Alex Auvolat
a36248a169
[fix-signed-headers] aws signatures v4: don't actually check Content-Type is signed
...
This page of the AWS docs indicate that Content-Type should be part of
the CanonicalHeaders (and therefore SignedHeaders) strings in signature
calculation:
https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-header-based-auth.html
However, testing with Minio Client revealed that it did not sign the
Content-Type header, and therefore we broke CI by expecting it to be
signed. With this commit, we don't mandate Content-Type to be signed
anymore, for better compatibility with the ecosystem. Testing against
the official behavior of S3 on AWS has not been done.
2024-03-01 13:12:18 +01:00
Alex Auvolat
6d33e721c4
[fix-auth-ct-eq] use consant time comparison for awsv4 signature verification
2024-02-29 13:07:18 +01:00
Alex Auvolat
eaac4924ef
[fix-auth-ct-eq] use argon2 hashing and verification for admin/metrics token checking
2024-02-29 13:07:15 +01:00
Alex Auvolat
90cab5b8f2
[fix-presigned] add comments and reorganize
2024-02-28 12:24:21 +01:00
Alex Auvolat
e9f759d4cb
[fix-presigned] presigned requests: allow x-amz-* query parameters to stand in for equivalent headers
2024-02-28 12:24:21 +01:00
Alex Auvolat
a5e4bfeae9
[fix-presigned] write comments
2024-02-28 12:24:21 +01:00
Alex Auvolat
4c1d42cc5f
[fix-presigned] add back anonymous request code path + refactoring
2024-02-28 12:24:21 +01:00
Alex Auvolat
2efa9c5a1a
[fix-presigned] PostObject: verify X-Amz-Algorithm
2024-02-28 12:24:20 +01:00
Alex Auvolat
a8cb8e8a8b
[fix-presigned] split presigned/normal signature verification
2024-02-28 12:24:13 +01:00
Alex Auvolat
4b978b7533
Merge branch 'main' into next-0.10
2024-02-26 18:55:24 +01:00
Alex Auvolat
b76c0c102e
[refactor-put] add ordering tag to blocks being sent to storage nodes
2024-02-26 18:35:11 +01:00
Alex Auvolat
babccd2ad3
[refactor-put] send several blocks in parallel to storage nodes
2024-02-26 18:22:37 +01:00
Alex Auvolat
3fe94cc14f
[refactor-put] rewrite read_and_put_block as a series of steps with channels
2024-02-26 17:55:37 +01:00
Alex Auvolat
3692af7052
Merge branch 'main' into next-0.10
2024-02-23 18:28:05 +01:00
Alex Auvolat
d640102b76
[split_getobject] GetObject: split out handle_get_full
2024-02-23 18:14:50 +01:00
Alex Auvolat
916c67ccf4
Merge branch 'main' into next-0.10
2024-02-23 16:50:34 +01:00
Alex Auvolat
6ee691e65f
[refactor-block] simplify some more
2024-02-23 12:50:10 +01:00
Alex Auvolat
e9c42bca34
[refactor-block] add DataBlockStream type
2024-02-23 12:22:29 +01:00
