Commit graph

622 commits

Author SHA1 Message Date
Quentin Dufour
5a1fb7cce7
Improve integration part of the doc 2022-02-07 16:01:45 +01:00
Alex
1c0ba930b8 Reorganize documentation for new website (#213)
This PR should be merged after the new website is deployed.

- [x] Rename files
- [x] Add front matter section to all `.md` files in the book (necessary for Zola)
- [x] Change all internal links to use Zola's linking system that checks broken links
- [x] Some updates to documentation contents and organization

Co-authored-by: Alex Auvolat <alex@adnab.me>
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/213
Co-authored-by: Alex <alex@adnab.me>
Co-committed-by: Alex <alex@adnab.me>
2022-02-07 11:51:12 +01:00
Alex Auvolat
45d6d377d2
Remove website publishing (new website is now online) 2022-02-06 09:25:04 +01:00
Quentin Dufour
6f7ef11537 Generate and upload a JSON result 2022-02-05 22:09:43 +01:00
Alex Auvolat
241db1e1f5 Add URL field to JSON builds 2022-02-05 22:09:43 +01:00
Quentin Dufour
ecd76977ea Generate JSON build description 2022-02-05 22:09:43 +01:00
Alex Auvolat
935670690f
Probably fix test-smoke 2022-02-02 17:34:19 +01:00
Alex Auvolat
ae2f32baf1
Hide deleted key in bucket info (fix #211) 2022-02-02 17:12:48 +01:00
Quentin Dufour
f67029ce2a Improve testing conf + test CORS 2022-02-01 17:55:14 +01:00
Alex Auvolat
2760f1cb17
Add advice about --fast-list 2022-01-31 16:51:39 +01:00
Alex Auvolat
26849ed066
Add step to 0.6.0 migration guide 2022-01-27 14:31:25 +01:00
Alex Auvolat
c99f55c420
Add restriction on part ordering in CompleteMultipartUpload 2022-01-25 12:45:00 +01:00
Alex Auvolat
acdf893362
Fix partnumber 2022-01-25 12:25:23 +01:00
Alex Auvolat
338b1b83ee
Implement part_number for GetObject 2022-01-24 21:04:42 +01:00
Alex Auvolat
6dab836f3a
Multipart improvements
- support part_number for HeadObject
- add checks in complete_multipart_upload
2022-01-24 21:04:40 +01:00
Alex Auvolat
513a6b15f9
Handle OPTIONS on website endpoint 2022-01-24 12:32:28 +01:00
Alex Auvolat
ea7fb901eb
Implement {Put,Get,Delete}BucketCors and CORS in general
- OPTIONS request against API endpoint
- Returning corresponding CORS headers on API calls
- Returning corresponding CORS headers on website GET's
2022-01-24 11:58:00 +01:00
Trinity Pointard
820924534a use clamp instead of min(max()) 2022-01-24 11:56:59 +01:00
Quentin Dufour
94f0e7c135 Test ListParts endpoint with awscli 2022-01-21 10:42:35 +01:00
Quentin Dufour
440374524b Implement ListParts 2022-01-21 10:42:30 +01:00
Quentin Dufour
fe003d6fbc Add ListPartsResult structure 2022-01-20 16:38:55 +01:00
trinity-1686a
e55fa38c99 Add date verification to presigned urls (#196)
fix #96
fix #162 by returning Forbidden instead Bad Request

Co-authored-by: Trinity Pointard <trinity.pointard@gmail.com>
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/196
Co-authored-by: trinity-1686a <trinity.pointard@gmail.com>
Co-committed-by: trinity-1686a <trinity.pointard@gmail.com>
2022-01-18 12:22:31 +01:00
trinity-1686a
178e35f868 refactor s3_router and api_server to make unused Endpoint parameters more obvious 2022-01-17 15:50:24 +01:00
Alex Auvolat
7c049f1c94 Fix extreme value to be less extreme so that integration test works on 32bits 2022-01-17 12:56:29 +01:00
Jill
fdcddbe168 Fix Multipart Upload with WinSCP (#164) (#193)
Closes #164.

Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/193
Co-authored-by: Jill <kokakiwi@deuxfleurs.fr>
Co-committed-by: Jill <kokakiwi@deuxfleurs.fr>
2022-01-17 11:18:40 +01:00
Jill
b45dcc1925 Support STREAMING-AWS4-HMAC-SHA256-PAYLOAD (#64) (#156)
Closes #64.

Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/156
Co-authored-by: Jill <kokakiwi@deuxfleurs.fr>
Co-committed-by: Jill <kokakiwi@deuxfleurs.fr>
2022-01-17 10:55:31 +01:00
Alex Auvolat
60c0033c8b
Update documentation 2022-01-13 14:25:22 +01:00
Alex Auvolat
d4dd2e2640
Make use of website config, return error document on error 2022-01-13 14:25:19 +01:00
Alex Auvolat
9eb211948e
Allow setting index document and error document on the CLI 2022-01-13 14:25:19 +01:00
Alex Auvolat
3ea8ca1b9e
Implement GetBucketWebsite 2022-01-13 14:23:52 +01:00
Alex Auvolat
f7349f4005
Add quotes in returned etags 2022-01-13 14:03:33 +01:00
Alex Auvolat
1ee8f596ee
Testing for UploadPartCopies and bugfixes in AWS signatures 2022-01-13 14:03:30 +01:00
Alex Auvolat
6617a72220
Implement UploadPartCopy 2022-01-13 13:58:47 +01:00
Alex Auvolat
3770a34e3d
Implement x-amz-copy-if-xxx copy preconditions and return more headers on copy (fix #187) 2022-01-13 13:56:55 +01:00
Quentin
b4592a00fe Implement ListMultipartUploads (#171)
Implement ListMultipartUploads, also refactor ListObjects and ListObjectsV2.

It took me some times as I wanted to propose the following things:
  - Using an iterator instead of the loop+goto pattern. I find it easier to read and it should enable some optimizations. For example, when consuming keys of a common prefix, we do many [redundant checks](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/main/src/api/s3_list.rs#L125-L156) while the only thing to do is to [check if the following key is still part of the common prefix](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/feature/s3-multipart-compat/src/api/s3_list.rs#L476).
  - Try to name things (see ExtractionResult and RangeBegin enums) and to separate concerns (see ListQuery and Accumulator)
  - An IO closure to make unit tests possibles.
  - Unit tests, to track regressions and document how to interact with the code
  - Integration tests with `s3api`. In the future, I would like to move them in Rust with the aws rust SDK.

Merging of the logic of ListMultipartUploads and ListObjects was not a goal but a consequence of the previous modifications.

Some points that we might want to discuss:
  - ListObjectsV1, when using pagination and delimiters, has a weird behavior (it lists multiple times the same prefix) with `aws s3api` due to the fact that it can not use our optimization to skip the whole prefix. It is independant from my refactor and can be tested with the commented `s3api` tests in `test-smoke.sh`. It probably has the same weird behavior on the official AWS S3 implementation.
  - Considering ListMultipartUploads, I had to "abuse" upload id marker to support prefix skipping. I send an `upload-id-marker` with the hardcoded value `include` to emulate your "including" token.
  - Some ways to test ListMultipartUploads with existing software (my tests are limited to s3api for now).

Co-authored-by: Quentin Dufour <quentin@deuxfleurs.fr>
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/171
Co-authored-by: Quentin <quentin@dufour.io>
Co-committed-by: Quentin <quentin@dufour.io>
2022-01-12 19:04:55 +01:00
Alex Auvolat
9cb2e9e57c
Add documentation to migrate to 0.6 2022-01-10 14:42:58 +01:00
Alex Auvolat
3586c7257c
Clarify error message 2022-01-10 14:10:04 +01:00
Alex Auvolat
17ea28a438
Fix trivial bug in CLI 2022-01-10 12:38:33 +01:00
Alex Auvolat
8f39360f22
Update documentation 2022-01-07 17:14:37 +01:00
Alex Auvolat
7ee11f0eb6
Fix unit tests 2022-01-05 17:34:48 +01:00
Alex Auvolat
168a90dfb5
Fix some error codes 2022-01-05 17:07:36 +01:00
Alex Auvolat
fb1e31add0
Small CLI changes 2022-01-05 16:28:46 +01:00
Alex Auvolat
135858d067
Implement DeleteBucket 2022-01-05 16:28:19 +01:00
Alex Auvolat
8395030e48
Implement CreateBucket 2022-01-05 15:56:48 +01:00
Alex Auvolat
9431090b1e
Implement key allow|deny --create-bucket 2022-01-05 15:12:59 +01:00
Alex Auvolat
677ab60cc1
Small changes in key model and refactoring 2022-01-04 18:59:17 +01:00
Alex Auvolat
df35feba18
New buckets for 0.6.0: make bucket id a SK and not a HK, CLI updates 2022-01-04 12:53:14 +01:00
Alex Auvolat
1bcd6fabbd
New buckets for 0.6.0: small changes
- Fix bucket delete

- fix merge of bucket creation date

- Replace deletable with option in aliases
    Rationale: if two aliases point to conflicting bucket, resolving
    by making an arbitrary choice risks making data accessible when it
    shouldn't be. We'd rather resolve to deleting the alias until
    someone puts it back.
2022-01-04 12:52:47 +01:00
Alex Auvolat
ba7f268b99
Rename and change query filters 2022-01-04 12:52:46 +01:00
Alex Auvolat
de37658b94
Hopefully fix Nix build 2022-01-04 12:52:46 +01:00