mirrors/garage
1
0
Fork 0
mirror of https://git.deuxfleurs.fr/Deuxfleurs/garage.git synced 2024-11-21 15:41:02 +00:00
Code Issues Projects Releases Packages Wiki Activity
S3-compatible object store for small self-hosted geo-distributed deployments
821 commits 27 branches 74 tags 44 MiB
Rust 68.5%
Nix 28.3%
Clojure 1.3%
Shell 1.3%
Python 0.5%
Other 0.1%
Find a file
Alex 309d7aef3f Merge pull request 'performance improvements' (#342) from lx-perf-improvements into main 
Performance improvements included in this PR:

- [x] Use `Bytes` at a few places where appropriate, instead of `Vec<u8>`, to reduce the number of copies
  - [x] StreamChunker now accumulates incoming slices in a `Vec<Bytes>` instead of a `VecDeque<u8>`. Replaces calls to `.extend()` and `.drain()` that were quite costly by a simple `concat()` on a vec of slices which is much more optimized
- [x] Hashing (b2, sha256, md5) is now done on a Tokio thread dedicated to cpu-intensive tasks, using `spawn_blocking`
- [x] Block manager now uses 256 independant locks instead of one big lock for writing, reduces contention when writing several/many objects in parallel
- [x] Better LMDB defaults: we now put flags `NoSync` and `NoMetaSync` to avoid `fsync` at each transaction (extremely slow). Also increased number of LMDB readers to accomodate more intensive workloads

Other changes included in this PR:

- [x] Update to hashing and MAC crates: md5 and sha2 from 0.9 to 0.10, hmac from 0.10 to 0.12
- [x] switch to `tracing_subscriber` for logs, which allows to have timing of each event

Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/342
2022-09-12 16:38:43 +02:00
doc Document available build features 2022-09-06 17:16:45 +02:00
nix Inject GIT_VERSION even later 2022-09-07 18:30:15 +02:00
script Fail if compiled binary is dynamic 2022-07-26 18:27:46 +02:00
src Simplify+improve async hasher by using bounded channel 2022-09-12 16:23:43 +02:00
.dockerignore Build Docker image 2020-06-30 17:18:42 +02:00
.drone.yml Update .drone.yml signature 2022-09-05 16:09:01 +02:00
.gitattributes Add FOSDEM talk and move all .pdf files to Git LFS 2022-02-16 20:01:36 +01:00
.gitignore Work on API 2020-04-28 10:18:14 +00:00
Cargo.lock Merge branch 'main' into lx-perf-improvements 2022-09-08 15:49:17 +02:00
Cargo.nix Merge branch 'main' into lx-perf-improvements 2022-09-08 15:49:17 +02:00
Cargo.toml Use the new cargo feature resolver "2" 2022-09-06 01:14:19 +02:00
default.nix Configure structopt to report the right version 2022-08-11 10:21:45 +02:00
Dockerfile Extract toolchain build from the CI 2021-10-29 11:34:01 +02:00
k2v_test.py First implementation of K2V (#293) 2022-05-10 13:16:57 +02:00
LICENSE Switch to AGPL 2021-03-16 16:35:46 +01:00
Makefile Fix merge 2022-09-07 18:16:01 +02:00
README.md Improve how node roles are assigned in Garage 2021-11-16 16:05:53 +01:00
rustfmt.toml Fix the Sync issue. Details: 2020-04-10 22:01:48 +02:00
shell.nix Run clippy in nix, leveraging nix caching ability 2022-07-26 18:27:52 +02:00

README.md

Garage Build Status

Garage logo

[ Website and documentation | Binary releases | Git repository | Matrix channel ]

Garage is a lightweight S3-compatible distributed object store, with the following goals:

  • As self-contained as possible
  • Easy to set up
  • Highly resilient to network failures, network latency, disk failures, sysadmin failures
  • Relatively simple
  • Made for multi-datacenter deployments

Non-goals include:

  • Extremely high performance
  • Complete implementation of the S3 API
  • Erasure coding (our replication model is simply to copy the data as is on several nodes, in different datacenters if possible)

Our main use case is to provide a distributed storage layer for small-scale self hosted services such as Deuxfleurs.