mirrors/garage
1
0
Fork 0
mirror of https://git.deuxfleurs.fr/Deuxfleurs/garage.git synced 2024-11-25 09:31:00 +00:00
Code Issues Projects Releases Packages Wiki Activity
1770 commits 27 branches 74 tags 44 MiB
Commit graph

1031 commits

Author SHA1 Message Date
Alex Auvolat
3ecc17f8c5 new layout: use deterministic randomness for reproducible results 2023-09-21 11:21:35 +02:00
Alex Auvolat
0088599f52 new layout: fix clippy lints 2023-09-18 12:17:07 +02:00
Alex Auvolat
749b4865d0 new layout: improve display and fix comments 2023-09-18 12:07:45 +02:00
Alex Auvolat
015ccb39aa new layout: make zone_redundancy optionnal (if not set, is maximum) 2023-09-18 11:59:08 +02:00
Alex Auvolat
2e229d4430 new layout: improve output display 2023-09-12 17:24:51 +02:00
Alex Auvolat
91e764a2bf fix hang on shutdown 2023-09-12 14:35:48 +02:00
Alex Auvolat
aa79810596 Fix error when none of S3/K2V/WEB/ADMIN server is started (fix #613) 2023-09-12 14:35:19 +02:00
Alex Auvolat
fd7d8fec59 Merge branch 'main' into next 2023-09-11 23:09:20 +02:00
Alex Auvolat
9cfe55ab60 fix 32-bit build 2023-09-11 20:01:29 +02:00
Alex Auvolat
51abbb02d8 Merge branch 'main' into next 2023-09-11 20:00:02 +02:00
Alex Auvolat
d5bb50d738 use statvfs instead of mount list to determine free data/meta space (fix #611) 2023-09-11 19:08:24 +02:00
Alex Auvolat
f8b3883611 config: make block_size and sled_cache_capacity expressable as strings 2023-09-11 18:34:59 +02:00
Alex Auvolat
51b9731a08 make lmdb's map_size configurable (fix #628) 2023-09-11 18:03:44 +02:00
Alex Auvolat
ad6b1cc0be Merge branch 'main' into next 2023-09-11 13:14:18 +02:00
Alex Auvolat
ba7ac52c19 block repair: simpler/more robust iterator progress calculation 2023-09-11 12:31:34 +02:00
Alex Auvolat
9526328d38 scrub: clear saved checkpoint when canceling scrub 2023-09-11 12:10:48 +02:00
Alex Auvolat
7f9ba49c71 block manager: remove data_dir field 2023-09-11 11:57:36 +02:00
Alex Auvolat
de5d792181 block manager: fix indentation (why not detected by cargo fmt?) 2023-09-11 11:52:57 +02:00
Alex Auvolat
be91ef6294 block manager: fix bug where rebalance didn't delete old copies 2023-09-07 16:04:03 +02:00
Alex Auvolat
2657b5c1b9 block manager: fix bugs 2023-09-07 15:30:56 +02:00
Alex Auvolat
2f112ac682 correct free data space accounting for multiple data dirs on same fs 2023-09-07 14:42:20 +02:00
Alex Auvolat
6b008b5bd3 block manager: add rebalance operation to rebalance multi-hdd setups 2023-09-07 13:44:11 +02:00
Alex Auvolat
99ed18350f block manager: refactor and fix monitoring/statistics 2023-09-07 12:41:36 +02:00
Alex Auvolat
f38a31b330 block manager: avoid incorrect data_dir configs and avoid losing files 2023-09-06 17:49:30 +02:00
Alex Auvolat
e30865984a block manager: scrub checkpointing 2023-09-06 16:35:28 +02:00
Alex Auvolat
55c514999e block manager: fixes in layout 2023-09-06 16:35:28 +02:00
Alex Auvolat
a44f486931 block manager: refactoring & increase max worker count to 8 2023-09-06 16:35:28 +02:00
Alex Auvolat
3a74844df0 block manager: fix dir_not_empty 2023-09-06 16:35:28 +02:00
Alex Auvolat
93114a9747 block manager: refactoring 2023-09-06 16:35:28 +02:00
Alex Auvolat
fd00a47ddc table queue: increase batch size 2023-09-06 16:35:28 +02:00
Alex Auvolat
1b8c265c14 block manager: get rid of check_block_status 2023-09-06 16:35:28 +02:00
Alex Auvolat
a09f86729c block manager: move blocks in write_block if necessary 2023-09-06 16:35:28 +02:00
Alex Auvolat
887b3233f4 block manager: use data paths from layout 2023-09-06 16:35:28 +02:00
Alex Auvolat
6c420c0880 block manager: multi-directory layout computation 2023-09-06 16:35:28 +02:00
Alex Auvolat
71c0188055 block manager: skeleton for multi-hdd support 2023-09-06 16:35:28 +02:00
Alex Auvolat
4b4f2000f4 lifecycle: fix SkipBucket bug 2023-09-06 16:34:07 +02:00
Alex Auvolat
51eac97260 update version to 0.8.4 2023-09-05 23:28:12 +02:00
Alex Auvolat
e78566591b Revert netapp update, hold to version 0.5.2 that uses rmp-serde 0.15 2023-09-05 23:23:23 +02:00
Alex Auvolat
8e0c020bb9 lifecycle worker: correct small clippy lints 2023-09-04 11:33:44 +02:00
Alex Auvolat
1cdc321e28 lifecycle worker: don't get stuck on non-existent bucket 2023-08-31 11:36:30 +02:00
Alex Auvolat
f579d6d9b4 lifecycle worker: fix potential inifinite loop 2023-08-31 11:29:54 +02:00
Alex Auvolat
a00a52633f lifecycle worker: add log message when starting 2023-08-31 11:25:14 +02:00
Alex Auvolat
adbf5925de lifecycle worker: use queue_insert and process objects in batches 2023-08-31 11:19:26 +02:00
Alex Auvolat
1cfcc61de8 lifecycle worker: mitigate potential bugs + refactoring 2023-08-31 00:28:37 +02:00
Alex Auvolat
be03a4610f s3api: remove redundant serde rename attribute 2023-08-31 00:00:26 +02:00
Alex Auvolat
b2f679675e lifecycle worker: take into account disabled rules 2023-08-30 23:52:09 +02:00
Alex Auvolat
01c327a07a lifecycle worker: avoid building chrono's serde feature 2023-08-30 23:46:15 +02:00
Alex Auvolat
f0a395e2e5 s3 bucket apis: remove redundant call 2023-08-30 23:39:28 +02:00
Alex Auvolat
a1d57283c0 bucket_table: bucketparams::new doesn't need to be pub 2023-08-30 20:07:14 +02:00
Alex Auvolat
d2e94e36d6 lifecycle config: add missing line in merge() and remove tracing 2023-08-30 20:05:53 +02:00
Alex Auvolat
75ccc5a95c lifecycle config: store date as given, try to debug 2023-08-30 20:02:07 +02:00
Alex Auvolat
7200954318 lifecycle worker: add logging 2023-08-30 14:54:52 +02:00
Alex Auvolat
0f1849e1ac lifecycle worker: launch with the rest of Garage 2023-08-30 14:51:08 +02:00
Alex Auvolat
da8b224e24 lifecycle worker: skip entire bucket when no lifecycle config is set 2023-08-30 14:38:19 +02:00
Alex Auvolat
2996dc875f lifecycle worker: implement main functionality 2023-08-30 14:29:03 +02:00
Alex Auvolat
a2e0e34db5 lifecycle: skeleton for lifecycle worker 2023-08-30 12:41:11 +02:00
Alex Auvolat
f7b409f114 use a NaiveDate in data model, it serializes to string (iso 8601 format) 2023-08-30 11:24:01 +02:00
Alex Auvolat
abf011c290 lifecycle: implement validation into garage's internal data structure 2023-08-29 18:22:03 +02:00
Alex Auvolat
8041d9a827 s3: add xml structures to serialize/deserialize lifecycle configs 2023-08-29 17:44:17 +02:00
Alex Auvolat
0b83e0558e bucket_table: data model for lifecycle configuration 2023-08-29 17:00:41 +02:00
Alex Auvolat
2e90e1c124 Merge branch 'main' into next 2023-08-29 11:32:42 +02:00
Alex Auvolat
06369c8f4a add garage_db dependency in garage_rpc 2023-08-28 17:08:21 +02:00
Alex Auvolat
cece1be1bb bump version to 0.8.3 2023-08-28 13:17:26 +02:00
Alex Auvolat
769b6fe054 fix test_website_check_domain 2023-08-28 12:40:28 +02:00
Alex Auvolat
e66c78d6ea integration test: move json_body to root of crate 2023-08-28 12:32:57 +02:00
Alex Auvolat
9b4ce4a8ad admin api: refactor caddy check api code 2023-08-28 12:17:10 +02:00
Alex
2bbe2da5ad Merge pull request 'support index on path missing a trailing slash' (#612) from compat/index-without-trailing-slash into main 
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/612
 2023-08-28 10:15:01 +00:00
Alex Auvolat
c5cafa0000 web_server.rs: handle error properly and refactor 2023-08-28 12:05:14 +02:00
Jonathan Davies
d66d81ae2d cargo: Updated gethostname v0.2.3 -> v0.4.3. 2023-08-28 09:30:27 +00:00
Jonathan Davies
7d8296ec59 cargo: Updated pretty_env_logger v0.4.0 -> v0.5.0. 2023-08-28 09:30:27 +00:00
Jonathan Davies
f607ac6792 garage/api: cargo: Updated idna dependency to 0.4. 2023-08-28 09:30:27 +00:00
Jonathan Davies
96d1d81ab7 garage/db: cargo: Updated rusqlite to 0.29. 2023-08-28 09:30:27 +00:00
Jonathan Davies
5185701aa8 cargo: Updated: 
* addr2line v0.19.0 -> v0.20.0
 * async-compression v0.4.0 -> v0.4.1
 * clap v4.3.8 -> v4.3.19
 * hyper v0.14.26 -> v0.14.27
 * ipnet v2.7.2 -> v2.8.0
 * rmp v0.8.11 -> v0.8.12
 * serde v1.0.164 -> v1.0.188
 * tokio v1.29.0 -> v1.31.0
 * zstd v0.12.3+zstd.1.5.2 -> v0.12.4
 * Others in `cargo update`
 2023-08-28 09:30:27 +00:00
Quentin Dufour
63da1d2443
support index on path missing a trailing slash 2023-08-08 15:28:57 +02:00
Quentin Dufour
24e533f262
support {s3,web}.root_domains in /check endpoint 2023-08-08 11:05:42 +02:00
Jonathan Davies
59bfc68f2e post_object.rs: Fixed typos / grammar. 2023-08-01 15:31:39 +01:00
Florian Klink
c855284760 src/util: fix typo 2023-07-14 14:25:40 +03:00
Florian Klink
b1ca1784a1 src/garage/cli: fix typo 2023-07-14 14:25:33 +03:00
Jonathan Davies
aee0d97f22 cargo: Updated async-compression to 0.4. 2023-06-28 11:17:16 +01:00
Alex
e716320b0a Merge pull request 'cargo: roxmltree-0.18 and aws-sdk-s3-0.28 bump' (#591) from jpds/garage:roxmltree-0.18 into main 
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/591
 2023-06-27 17:20:58 +00:00
Alex
e466edbaec Merge pull request 'introduce dedicated return type for PollRange' (#590) from trinity-1686a/garage:k2v-client-poll-range-result into main 
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/590
 2023-06-27 08:28:26 +00:00
Jonathan Davies
a6da7e588f tests/bucket.rs: Adjusted as previously used function is now private. 2023-06-26 18:03:43 +01:00
trinity-1686a
e5835704b7 don't build sqlite by default 
`bundled-libs` is enabled by default, and causes sqlite to be built too,
even if the sqlite backend isn't enabled.
 2023-06-26 11:15:11 +02:00
Jonathan Davies
7f8bf2d801 src/garage/tests: Updated types for aws-sdk-s3 bump. 2023-06-25 21:31:35 +01:00
Jonathan Davies
4297233d3e garage/Cargo.toml: Updated aws-sdk-s3 to 0.28, added aws-config. 2023-06-25 21:17:15 +01:00
Jonathan Davies
b94ba47f29 api/Cargo.toml: Updated roxmltree to 0.18. 2023-06-24 14:15:26 +01:00
trinity-1686a
33b3cf8e22 introduce dedicated return type for PollRange 2023-06-24 10:17:20 +02:00
Alex Auvolat
8ef42c9609 admin docs: reformatting, key admin: add check 2023-06-14 17:19:25 +02:00
Alex Auvolat
a83a092c03 admin: uniformize layout api and improve code 2023-06-14 17:12:37 +02:00
Alex Auvolat
7895f99d3a admin and cli: hide secret keys unless asked 2023-06-14 16:56:15 +02:00
Alex Auvolat
4a82f6380e admin api: move all endpoints to v1/ by default (v0/ still supported) 2023-06-14 14:15:51 +02:00
Alex Auvolat
28cc9f178a admin api: make name optionnal for CreateKey 2023-06-14 13:56:37 +02:00
Alex Auvolat
35c108b85d admin api: switch GetClusterHealth to camelcase (fix #381 again) 2023-06-14 13:53:19 +02:00
Alex Auvolat
52376d47ca admin api: change cluster status/layout to use lists and not maps (fix #377) 2023-06-14 13:45:27 +02:00
Alex Auvolat
5670367126 multipartupload in test: add forgotten timestamp 2023-06-13 23:10:46 +02:00
Alex Auvolat
cda957b4b1 update netapp's rmp-serde dependency to v1.1 2023-06-13 17:34:49 +02:00
Alex Auvolat
90b2d43eb4 Merge branch 'main' into next 2023-06-13 17:14:11 +02:00
Alex Auvolat
eb9cecf05c Split garage/admin.rs into smaller files 2023-06-13 16:46:28 +02:00
Alex Auvolat
802ed75721 move admin.rs to admin/mod.rs, before splitting 2023-06-13 16:42:14 +02:00
Alex Auvolat
bf19a44fd9 admin API: add missing camelCase conversions (fix #381) 2023-06-13 16:15:50 +02:00
Alex Auvolat
7126f3e1d1 garage key import: add checks and --yes CLI flag (fix #278) 2023-06-13 15:56:48 +02:00
Alex Auvolat
942c1f1bfe multipart uploads: save timestamp 2023-06-13 10:48:22 +02:00
Alex Auvolat
1ea4937c8b fix timestamps wrapping around in garage block list-errors (fix #584) 2023-06-12 20:07:33 +02:00
Alex Auvolat
3d477906d4 properly delete multipart uploads after completion 2023-06-09 17:13:27 +02:00
Alex Auvolat
a6cc563bdd UploadPart: automatic cleanup of version (and reference blocked) when interrupted 2023-06-09 16:23:37 +02:00
Alex Auvolat
c14d3735e5 Add test for multipart uploads and fix part renumbering 2023-06-09 16:23:37 +02:00
Alex Auvolat
53bf2f070c undo sort_key() returning Cow 2023-06-09 16:23:37 +02:00
Alex Auvolat
412ab77b08 comments and clippy lint fixes 2023-06-09 16:23:37 +02:00
Alex Auvolat
511e07ecd4 fix mpu counter (add missing workers) and report info at appropriate places 2023-06-09 16:23:37 +02:00
Alex Auvolat
4ea53dc759 Add multipart upload repair 2023-06-09 16:23:37 +02:00
Alex Auvolat
058518c22b refactor repair workers with a trait 2023-06-09 16:23:37 +02:00
Alex Auvolat
8644376ac2 fix test; simplify code 2023-06-09 16:23:37 +02:00
Alex Auvolat
7ad7dae5d4 fix s3 list test 2023-06-09 16:23:37 +02:00
Alex Auvolat
75a0e01372 fix online repair 2023-06-09 16:23:37 +02:00
Alex Auvolat
bb176ebcb8 cargo fmt 2023-06-09 16:23:37 +02:00
Alex Auvolat
87be8eeb93 updaet block admin for new multipartupload models 2023-06-09 16:23:37 +02:00
Alex Auvolat
82e75c0e29 Adapt S3 API code to use new multipart upload models 
- Create and PutPart
- completemultipartupload
- upload part copy
- list_parts
 2023-06-09 16:23:37 +02:00
Alex Auvolat
38d6ac4295 New multipart upload table layout 2023-06-09 16:23:37 +02:00
Alex Auvolat
6005491cd8 Use Cow<[u8]> for sort keys 2023-06-09 16:23:37 +02:00
Alex Auvolat
e7e164a280 Make fsync an option for meta and data 2023-06-09 16:23:21 +02:00
Alex Auvolat
1e466b11eb Revert integration tests to using Sled as LMDB causes failures 2023-06-09 13:23:08 +02:00
Jonathan Davies
8a945ee996 payload.rs: Surround / in inverted commas. 2023-06-06 16:26:06 +01:00
Jonathan Davies
180992d0f1 payload.rs: Fixed typo in error message. 2023-06-06 16:25:29 +01:00
Alex Auvolat
8a74e1c2bd Split garage/admin.rs into smaller files 2023-06-06 15:39:15 +02:00
Roberto Hidalgo
ef8a7add08 set default for [consul-services] api 2023-05-22 08:57:15 -06:00
Roberto Hidalgo
b770504126 simplify code according to feedback 2023-05-22 08:57:15 -06:00
Roberto Hidalgo
6b69404f1a rename mode to consul_http_api 2023-05-22 08:57:15 -06:00
Roberto Hidalgo
011f473048 revert rpc/Cargo.toml 2023-05-22 08:57:15 -06:00
Roberto Hidalgo
fd7dbea5b8 follow feedback, fold into existing feature 2023-05-22 08:57:15 -06:00
Roberto Hidalgo
bd6485565e allow additional ServiceMeta, docs 2023-05-22 08:57:15 -06:00
Roberto Hidalgo
4d6e6fc155 cargo fmt 2023-05-22 08:57:15 -06:00
Roberto Hidalgo
02ba9016ab register consul services against local agent instead of catalog api 2023-05-22 08:57:15 -06:00
Alex Auvolat
c3d3b837eb bump k2v-client to v0.0.4 2023-05-22 10:47:15 +02:00
Alex Auvolat
130e01505b Fix k2v_client with unicode in partition keys 2023-05-22 10:45:09 +02:00
Alex Auvolat
e2ce5970c6 Add basic k2v_client integration tests 2023-05-22 10:45:06 +02:00
Alex Auvolat
644e872264 Port k2v-client to aws-sigv4 since rusoto_signature is deprecated 2023-05-19 12:08:29 +02:00
Alex Auvolat
4420db7310 add tracing to k2v-client 2023-05-18 11:18:21 +02:00
Alex Auvolat
746b0090e4 k2v signature verification: double urlencoding (see comment in source code) 2023-05-18 11:18:06 +02:00
Alex Auvolat
19639705e6 Mark sled as deprecated, make lmdb default, and improve sqlite and lmdb defaults 2023-05-17 14:30:53 +02:00
Alex Auvolat
217d429937 fix clippy lint in format-table crate 2023-05-17 13:06:37 +02:00
Alex Auvolat
a1cec2cd60 Split format_table into separate crate and reduce k2v-client dependencies 2023-05-17 13:01:37 +02:00
Alex Auvolat
16f2a32bb7 cargo fmt 2023-05-16 19:46:57 +02:00
Alex Auvolat
472444ed8e k2v-client 0.0.2 2023-05-16 19:46:57 +02:00
Alex Auvolat
bb03805b58 k2v-cli: fix sort_key being partition_key and specify which key 2023-05-16 19:46:57 +02:00
Alex Auvolat
e4f955d672 fix base64 uses 2023-05-16 19:46:56 +02:00
Alex
ea9b15f669 Merge pull request 'cargo: tokio-1.28 and hyper-0.14.26 update' (#569) from jpds/garage:tokio-1.28 into main 
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/569
 2023-05-11 10:16:33 +00:00
Jonathan Davies
c783194e8b *: apply clippy recommendations. 2023-05-09 20:49:34 +01:00
Jonathan Davies
0f0795103d block/Cargo.toml: Bump tokio-util to 0.7. 2023-05-09 14:33:21 +01:00
Jonathan Davies
c9d26e8c50 k2v-client/Cargo.toml: Make tokio dep match other packages. 2023-05-09 14:33:00 +01:00
Alex Auvolat
351d734e6c Merge branch 'main' into next 2023-05-09 12:40:08 +02:00