2023-03-09 18:12:50 +00:00
# Backie 🚲
2021-06-24 09:58:02 +00:00
2023-03-07 16:52:26 +00:00
Async background job processing library with Diesel and Tokio. It's a heavily modified fork of [fang ](https://github.com/ayrat555/fang ).
2021-07-11 07:26:20 +00:00
2022-10-18 06:30:17 +00:00
## Key Features
2022-09-25 13:05:27 +00:00
Here are some of the fang's key features:
2023-03-07 16:52:26 +00:00
- Async workers: Workers are started as `tokio` tasks (async workers)
- Unique tasks: Tasks are not duplicated in the queue if they are unique
- Single-purpose workers: Tasks are stored in a single table but workers can be configured to execute only tasks of a specific type
- Retries: Tasks can be retried with a custom backoff mode
## Differences from Fang crate
2023-03-04 19:46:09 +00:00
- Supports only async processing
- Supports graceful shutdown
- The connection pool for the queue is provided by the user
2023-03-07 16:52:26 +00:00
- Tasks status is calculated based on the database state
- Tasks have a timeout and are retried if they are not completed in time
2023-03-04 19:46:09 +00:00
2021-06-24 09:58:02 +00:00
## Installation
1. Add this to your Cargo.toml
2022-08-03 08:37:53 +00:00
```toml
[dependencies]
2023-03-07 16:52:26 +00:00
backie = "0.10"
2022-08-05 07:06:32 +00:00
```
2023-03-04 18:07:17 +00:00
*Supports rustc 1.67+*
2022-07-01 06:03:27 +00:00
2023-03-07 16:52:26 +00:00
2. Create the `backie_tasks` table in the Postgres database. The migration can be found in [the migrations directory ](https://github.com/rafaelcaricio/backie/blob/master/migrations/2023-03-06-151907_create_backie_tasks/up.sql ).
2021-06-24 09:58:02 +00:00
## Usage
2023-03-07 16:52:26 +00:00
Every task must implement the `backie::RunnableTask` trait, Backie uses the information provided by the trait to
execute the task.
2021-06-24 09:58:02 +00:00
2023-03-07 16:52:26 +00:00
All implementations of `RunnableTask` must have unique names per project.
2021-06-24 09:58:02 +00:00
```rust
2023-03-07 16:52:26 +00:00
use backie::RunnableTask;
use backie::task::{TaskHash, TaskType};
use backie::queue::AsyncQueueable;
use serde::{Deserialize, Serialize};
use async_trait::async_trait;
2021-06-24 09:58:02 +00:00
2021-07-11 07:26:20 +00:00
#[derive(Serialize, Deserialize)]
2022-08-03 08:37:53 +00:00
#[serde(crate = "fang::serde")]
2022-07-19 13:49:43 +00:00
struct MyTask {
2023-03-07 16:52:26 +00:00
pub number: u16,
2022-08-03 08:37:53 +00:00
}
#[typetag::serde]
#[async_trait]
2023-03-07 16:52:26 +00:00
impl RunnableTask for MyTask {
async fn run(& self, _queueable: & mut dyn AsyncQueueable) -> Result< (), Error> {
Ok(())
}
// this func is optional
// Default task_type is common
fn task_type(& self) -> TaskType {
"my-task-type".into()
}
// If `uniq` is set to true and the task is already in the storage, it won't be inserted again
// The existing record will be returned for for any insertions operaiton
fn uniq(& self) -> Option< TaskHash > {
None
}
// the maximum number of retries. Set it to 0 to make it not retriable
// the default value is 20
fn max_retries(& self) -> i32 {
2022-09-25 13:05:27 +00:00
20
2023-03-07 16:52:26 +00:00
}
2022-09-25 13:05:27 +00:00
2023-03-07 16:52:26 +00:00
// backoff mode for retries
fn backoff(& self, attempt: u32) -> u32 {
2022-09-25 13:05:27 +00:00
u32::pow(2, attempt)
2023-03-07 16:52:26 +00:00
}
2022-08-03 08:37:53 +00:00
}
```
2022-09-03 08:22:12 +00:00
2022-07-19 13:49:43 +00:00
### Enqueuing a task
2021-06-24 09:58:02 +00:00
2023-03-07 16:52:26 +00:00
To enqueue a task use `AsyncQueueable::create_task` .
2022-08-03 08:37:53 +00:00
For Postgres backend.
```rust
2023-03-07 16:52:26 +00:00
use backie::queue::PgAsyncQueue;
2022-08-03 08:37:53 +00:00
2022-09-25 13:05:27 +00:00
// Create an AsyncQueue
2023-03-07 16:52:26 +00:00
let manager = AsyncDieselConnectionManager::< AsyncPgConnection > ::new("postgres://postgres:password@localhost/backie");
let pool = Pool::builder()
.max_size(1)
.min_idle(Some(1))
.build(manager)
.await
.unwrap();
let mut queue = PgAsyncQueue::new(pool);
// Publish the first example
let task = MyTask { number: 8 };
2022-08-03 08:37:53 +00:00
let task_returned = queue
2023-03-07 16:52:26 +00:00
.create_task(& task)
2022-08-03 08:37:53 +00:00
.await
.unwrap();
```
2021-06-24 09:58:02 +00:00
### Starting workers
2022-09-25 13:05:27 +00:00
Every worker runs in a separate `tokio` task. In case of panic, they are always restarted.
2022-08-04 16:47:17 +00:00
Use `AsyncWorkerPool` to start workers.
2022-08-03 08:37:53 +00:00
```rust
2023-03-07 16:52:26 +00:00
use backie::worker_pool::AsyncWorkerPool;
2022-08-03 08:37:53 +00:00
2022-08-04 16:47:17 +00:00
// Need to create a queue
2022-08-03 08:37:53 +00:00
// Also insert some tasks
2023-03-07 16:52:26 +00:00
let mut pool: AsyncWorkerPool< PgAsyncQueue > = AsyncWorkerPool::builder()
2022-08-03 08:37:53 +00:00
.number_of_workers(max_pool_size)
.queue(queue.clone())
2022-09-03 08:22:12 +00:00
// if you want to run tasks of the specific kind
2023-03-07 16:52:26 +00:00
.task_type("my_task_type".into())
2022-08-03 08:37:53 +00:00
.build();
pool.start().await;
```
2021-08-18 19:09:49 +00:00
Check out:
2023-03-07 16:52:26 +00:00
- [Simple Worker Example ](https://github.com/rafaelcaricio/backie/tree/master/examples/simple_worker ) - simple worker example
2021-07-31 07:40:11 +00:00
2021-07-04 06:07:29 +00:00
### Configuration
2023-03-07 16:52:26 +00:00
Use the `AsyncWorkerPool` builder:
```rust
let mut pool: AsyncWorkerPool< PgAsyncQueue > = AsyncWorkerPool::builder()
.number_of_workers(max_pool_size)
.queue(queue.clone())
.build();
```
2022-08-03 08:37:53 +00:00
2021-07-04 06:07:29 +00:00
### Configuring the type of workers
### Configuring retention mode
By default, all successfully finished tasks are removed from the DB, failed tasks aren't.
There are three retention modes you can use:
```rust
pub enum RetentionMode {
2022-09-03 11:05:58 +00:00
KeepAll, // doesn't remove tasks
RemoveAll, // removes all tasks
RemoveFinished, // default value
2021-07-04 06:07:29 +00:00
}
```
2022-09-03 08:22:12 +00:00
Set retention mode with worker pools `TypeBuilder` in both modules.
2021-07-04 06:07:29 +00:00
2021-06-24 09:58:02 +00:00
## Contributing
1. [Fork it! ](https://github.com/ayrat555/fang/fork )
2. Create your feature branch (`git checkout -b my-new-feature`)
3. Commit your changes (`git commit -am 'Add some feature'`)
4. Push to the branch (`git push origin my-new-feature`)
2022-09-25 13:05:27 +00:00
5. Create a new Pull Request
2021-06-24 09:58:02 +00:00
2021-12-05 06:19:08 +00:00
### Running tests locally
2022-09-03 08:22:12 +00:00
- Install diesel_cli.
2021-12-05 06:19:08 +00:00
```
cargo install diesel_cli
2022-09-03 08:22:12 +00:00
```
2022-09-25 13:05:27 +00:00
- Install docker on your machine.
2021-12-05 06:19:08 +00:00
2022-09-03 08:22:12 +00:00
- Run a Postgres docker container. (See in Makefile.)
```
make db
```
2021-12-05 06:19:08 +00:00
2022-09-03 08:22:12 +00:00
- Run the migrations
```
make diesel
```
2021-12-05 06:19:08 +00:00
2022-09-03 08:22:12 +00:00
- Run tests
```
make tests
```
2021-12-05 06:19:08 +00:00
2022-09-03 08:22:12 +00:00
- Run dirty//long tests, DB must be recreated afterwards.
```
make ignored
```
2021-12-05 06:19:08 +00:00
2022-09-25 13:05:27 +00:00
- Kill the docker container
2022-09-03 08:22:12 +00:00
```
make stop
2021-12-05 06:19:08 +00:00
```
2023-03-07 16:52:26 +00:00
## Thank Fang's authors
2021-06-24 09:58:02 +00:00
2023-03-07 16:52:26 +00:00
I would like to thank the authors of the fang crate which was the inspiration for this project.
2022-08-05 07:06:32 +00:00
2023-03-07 16:52:26 +00:00
- Ayrat Badykov (@ayrat555)
2022-08-03 08:37:53 +00:00
- Pepe Márquez (@pxp9)
2021-07-11 10:55:52 +00:00
2023-03-07 16:52:26 +00:00
[ci]: https://crates.io/crates/backie
[docs]: https://docs.rs/backie/
[ga-test]: https://github.com/rafaelcaricio/backie/actions/workflows/rust.yml/badge.svg
[ga-style]: https://github.com/rafaelcaricio/backie/actions/workflows/style.yml/badge.svg