2022-09-03 11:05:58 +00:00
< p align = "center" > < img src = "https://raw.githubusercontent.com/ayrat555/fang/master/logo.png" alt = "fang" height = "300px" > < / p >
2021-06-24 09:58:02 +00:00
2021-07-11 10:55:52 +00:00
[![Crates.io][s1]][ci] [![docs page][docs-badge]][docs] ![test][ga-test] ![style][ga-style]
2021-06-24 09:58:02 +00:00
# Fang
2022-07-19 13:49:43 +00:00
Background task processing library for Rust. It uses Postgres DB as a task queue.
2021-07-11 07:26:20 +00:00
2022-08-05 07:06:32 +00:00
## Features
2022-09-25 13:05:27 +00:00
Here are some of the fang's key features:
- Async and threaded workers
Workers can be started in threads (threaded workers) or `tokio` tasks (async workers)
- Scheduled tasks
Tasks can be scheduled at any time in the future
- Periodic (CRON) tasks
Tasks can be scheduled using cron expressions
- Unique tasks
Tasks are not duplicated in the queue if they are unique
- Single-purpose workers
Tasks are stored in a single table but workers can execute only tasks of the specific type
- Retries
Tasks can be retried with a custom backoff mode
2021-07-25 12:20:16 +00:00
2021-06-24 09:58:02 +00:00
## Installation
1. Add this to your Cargo.toml
2022-09-25 13:05:27 +00:00
#### the Blocking feature
2021-06-24 09:58:02 +00:00
```toml
[dependencies]
2022-09-03 08:22:12 +00:00
fang = { version = "0.9" , features = ["blocking"], default-features = false }
2021-06-24 09:58:02 +00:00
```
2022-09-25 13:05:27 +00:00
#### the Asynk feature
2022-08-03 08:37:53 +00:00
```toml
[dependencies]
2022-09-03 08:22:12 +00:00
fang = { version = "0.9" , features = ["asynk"], default-features = false }
2022-08-03 08:37:53 +00:00
```
2022-08-05 07:06:32 +00:00
#### Both features
```toml
2022-09-03 08:22:12 +00:00
fang = { version = "0.9" }
2022-08-05 07:06:32 +00:00
```
2022-07-01 06:03:27 +00:00
*Supports rustc 1.62+*
2022-09-25 13:05:27 +00:00
2. Create the `fang_tasks` table in the Postgres database. The migration can be found in [the migrations directory ](https://github.com/ayrat555/fang/blob/master/migrations/2022-08-20-151615_create_fang_tasks/up.sql ).
2021-06-24 09:58:02 +00:00
## Usage
2022-07-19 13:49:43 +00:00
### Defining a task
2021-06-24 09:58:02 +00:00
2022-08-04 17:11:58 +00:00
#### Blocking feature
2022-09-25 13:05:27 +00:00
Every task should implement the `fang::Runnable` trait which is used by `fang` to execute it.
2021-06-24 09:58:02 +00:00
```rust
2021-07-11 07:26:20 +00:00
use fang::Error;
use fang::Runnable;
use fang::typetag;
2021-07-31 07:40:11 +00:00
use fang::PgConnection;
2022-08-03 08:37:53 +00:00
use fang::serde::{Deserialize, Serialize};
2021-06-24 09:58:02 +00:00
2021-07-11 07:26:20 +00:00
#[derive(Serialize, Deserialize)]
2022-08-03 08:37:53 +00:00
#[serde(crate = "fang::serde")]
2022-07-19 13:49:43 +00:00
struct MyTask {
2021-07-11 07:26:20 +00:00
pub number: u16,
}
2021-06-24 09:58:02 +00:00
2021-07-11 07:26:20 +00:00
#[typetag::serde]
2022-07-19 13:49:43 +00:00
impl Runnable for MyTask {
2022-09-03 08:22:12 +00:00
fn run(& self, _queue: & dyn Queueable) -> Result< (), Error> {
2021-07-11 07:26:20 +00:00
println!("the number is {}", self.number);
2021-06-24 09:58:02 +00:00
2021-07-11 07:26:20 +00:00
Ok(())
2021-06-24 09:58:02 +00:00
}
2022-09-03 08:22:12 +00:00
2022-09-25 13:05:27 +00:00
// If `uniq` is set to true and the task is already in the storage, it won't be inserted again
// The existing record will be returned for for any insertions operaiton
2022-09-03 08:22:12 +00:00
fn uniq(& self) -> bool {
true
}
// This will be useful if you want to filter tasks.
2022-09-25 13:05:27 +00:00
// the default value is `common`
2022-09-03 08:22:12 +00:00
fn task_type(& self) -> String {
"my_task".to_string()
}
// This will be useful if you would like to schedule tasks.
2022-09-25 13:05:27 +00:00
// default value is None (the task is not scheduled, it's just executed as soon as it's inserted)
2022-09-03 08:22:12 +00:00
fn cron(& self) -> Option< Scheduled > {
let expression = "0/20 * * * Aug-Sep * 2022/1";
Some(Scheduled::CronPattern(expression.to_string()))
}
2022-09-25 13:05:27 +00:00
// the maximum number of retries. Set it to 0 to make it not retriable
// the default value is 20
fn max_retries(& self) -> i32 {
20
}
// backoff mode for retries
fn backoff(& self, attempt: u32) -> u32 {
u32::pow(2, attempt)
}
2021-07-11 07:26:20 +00:00
}
2021-06-24 09:58:02 +00:00
```
2022-07-19 13:49:43 +00:00
As you can see from the example above, the trait implementation has `#[typetag::serde]` attribute which is used to deserialize the task.
2021-06-24 09:58:02 +00:00
2022-09-25 13:05:27 +00:00
The second parameter of the `run` function is a struct that implements `fang::Queueable` . You can re-use it to manipulate the task queue, for example, to add a new job during the current job's execution. If you don't need it, just ignore it.
2021-07-31 07:40:11 +00:00
2022-08-03 08:37:53 +00:00
2022-08-04 17:11:58 +00:00
#### Asynk feature
Every task should implement `fang::AsyncRunnable` trait which is used by `fang` to execute it.
2022-08-03 08:37:53 +00:00
2022-09-25 13:05:27 +00:00
Be careful not to call two implementations of the AsyncRunnable trait with the same name, because it will cause a failure in the `typetag` crate.
2022-08-03 08:37:53 +00:00
```rust
use fang::AsyncRunnable;
use fang::asynk::async_queue::AsyncQueueable;
use fang::serde::{Deserialize, Serialize};
use fang::async_trait;
#[derive(Serialize, Deserialize)]
#[serde(crate = "fang::serde")]
struct AsyncTask {
pub number: u16,
}
#[typetag::serde]
#[async_trait]
impl AsyncRunnable for AsyncTask {
async fn run(& self, _queueable: & mut dyn AsyncQueueable) -> Result< (), Error> {
Ok(())
}
2022-09-25 13:05:27 +00:00
// this func is optional
// Default task_type is common
2022-08-03 08:37:53 +00:00
fn task_type(& self) -> String {
"my-task-type".to_string()
}
2022-09-03 08:22:12 +00:00
2022-09-25 13:05:27 +00:00
// If `uniq` is set to true and the task is already in the storage, it won't be inserted again
// The existing record will be returned for for any insertions operaiton
2022-09-03 08:22:12 +00:00
fn uniq(& self) -> bool {
true
}
// This will be useful if you would like to schedule tasks.
2022-09-25 13:05:27 +00:00
// default value is None (the task is not scheduled, it's just executed as soon as it's inserted)
2022-09-03 08:22:12 +00:00
fn cron(& self) -> Option< Scheduled > {
let expression = "0/20 * * * Aug-Sep * 2022/1";
Some(Scheduled::CronPattern(expression.to_string()))
}
2022-09-25 13:05:27 +00:00
// the maximum number of retries. Set it to 0 to make it not retriable
// the default value is 20
fn max_retries(& self) -> i32 {
20
}
// backoff mode for retries
fn backoff(& self, attempt: u32) -> u32 {
u32::pow(2, attempt)
}
2022-08-03 08:37:53 +00:00
}
```
2022-09-03 08:22:12 +00:00
2022-09-25 13:05:27 +00:00
In both modules, tasks can be scheduled to be executed once. Use `Scheduled::ScheduleOnce` enum variant.
2022-09-03 08:22:12 +00:00
2022-09-25 13:05:27 +00:00
Datetimes and cron patterns are interpreted in the UTC timezone. So you should introduce the offset to schedule in a different timezone.
2022-09-03 08:22:12 +00:00
Example:
2022-09-25 13:05:27 +00:00
If your timezone is UTC + 2 and you want to schedule at 11:00:
2022-09-03 08:22:12 +00:00
```rust
let expression = "0 0 9 * * * * ";
```
2022-07-19 13:49:43 +00:00
### Enqueuing a task
2021-06-24 09:58:02 +00:00
2022-09-25 13:05:27 +00:00
#### the Blocking feature
2022-08-04 16:47:17 +00:00
To enqueue a task use `Queue::enqueue_task`
2021-06-24 09:58:02 +00:00
```rust
2021-07-31 07:40:11 +00:00
use fang::Queue;
2021-06-24 09:58:02 +00:00
2022-09-03 08:22:12 +00:00
// create a r2d2 pool
2021-06-24 09:58:02 +00:00
2022-09-03 08:22:12 +00:00
// create a fang queue
2021-06-24 09:58:02 +00:00
2022-09-03 08:22:12 +00:00
let queue = Queue::builder().connection_pool(pool).build();
2021-06-24 09:58:02 +00:00
2022-09-03 08:22:12 +00:00
let task_inserted = queue.insert_task(&MyTask::new(1)).unwrap();
2021-07-04 06:07:29 +00:00
```
2021-06-24 09:58:02 +00:00
2022-09-25 13:05:27 +00:00
#### the Asynk feature
To enqueue a task use `AsyncQueueable::insert_task` .
2022-08-03 08:37:53 +00:00
For Postgres backend.
```rust
use fang::asynk::async_queue::AsyncQueue;
use fang::NoTls;
use fang::AsyncRunnable;
2022-09-25 13:05:27 +00:00
// Create an AsyncQueue
2022-08-03 08:37:53 +00:00
let max_pool_size: u32 = 2;
let mut queue = AsyncQueue::builder()
// Postgres database url
.uri("postgres://postgres:postgres@localhost/fang")
2022-08-05 07:06:32 +00:00
// Max number of connections that are allowed
2022-08-03 08:37:53 +00:00
.max_pool_size(max_pool_size)
.build();
// Always connect first in order to perform any operation
queue.connect(NoTls).await.unwrap();
```
2022-09-25 13:05:27 +00:00
As an easy example, we are using NoTls type. If for some reason you would like to encrypt Postgres requests, you can use [openssl ](https://docs.rs/postgres-openssl/latest/postgres_openssl/ ) or [native-tls ](https://docs.rs/postgres-native-tls/latest/postgres_native_tls/ ).
2022-08-03 08:37:53 +00:00
```rust
2022-09-25 13:05:27 +00:00
// AsyncTask from the first example
2022-08-03 08:37:53 +00:00
let task = AsyncTask { 8 };
let task_returned = queue
.insert_task(& task as & dyn AsyncRunnable)
.await
.unwrap();
```
2021-06-24 09:58:02 +00:00
### Starting workers
2022-09-25 13:05:27 +00:00
#### the Blocking feature
2021-06-24 10:15:10 +00:00
Every worker runs in a separate thread. In case of panic, they are always restarted.
2021-06-24 09:58:02 +00:00
2022-09-03 08:22:12 +00:00
Use `WorkerPool` to start workers. Use `WorkerPool::builder` to create your worker pool and run tasks.
2021-06-24 09:58:02 +00:00
```rust
use fang::WorkerPool;
2022-09-03 08:22:12 +00:00
use fang::Queue;
2021-06-24 09:58:02 +00:00
2022-09-03 08:22:12 +00:00
// create a Queue
2021-12-05 06:43:17 +00:00
2022-09-03 08:22:12 +00:00
let mut worker_pool = WorkerPool::< Queue > ::builder()
.queue(queue)
.number_of_workers(3_u32)
// if you want to run tasks of the specific kind
.task_type("my_task_type")
.build();
2021-12-05 06:43:17 +00:00
2022-09-03 08:22:12 +00:00
worker_pool.start();
2021-12-05 06:43:17 +00:00
```
2022-09-25 13:05:27 +00:00
#### the Asynk feature
Every worker runs in a separate `tokio` task. In case of panic, they are always restarted.
2022-08-04 16:47:17 +00:00
Use `AsyncWorkerPool` to start workers.
2022-08-03 08:37:53 +00:00
```rust
use fang::asynk::async_worker_pool::AsyncWorkerPool;
2022-08-04 16:47:17 +00:00
// Need to create a queue
2022-08-03 08:37:53 +00:00
// Also insert some tasks
let mut pool: AsyncWorkerPool< AsyncQueue < NoTls > > = AsyncWorkerPool::builder()
.number_of_workers(max_pool_size)
.queue(queue.clone())
2022-09-03 08:22:12 +00:00
// if you want to run tasks of the specific kind
.task_type("my_task_type")
2022-08-03 08:37:53 +00:00
.build();
pool.start().await;
```
2021-08-18 19:09:49 +00:00
Check out:
2022-09-03 08:22:12 +00:00
- [Simple Worker Example ](https://github.com/ayrat555/fang/tree/master/fang_examples/blocking/simple_worker ) - simple worker example
- [Simple Cron Worker Example ](https://github.com/ayrat555/fang/tree/master/fang_examples/blocking/simple_cron_worker ) - simple worker example
- [Simple Async Worker Example ](https://github.com/ayrat555/fang/tree/master/fang_examples/asynk/simple_async_worker ) - simple async worker example
- [Simple Cron Async Worker Example ](https://github.com/ayrat555/fang/tree/master/fang_examples/asynk/simple_cron_async_worker ) - simple async worker example
2022-09-25 13:05:27 +00:00
- [El Monitorro ](https://github.com/ayrat555/el_monitorro ) - telegram feed reader. It uses the Fang's blocking module to synchronize feeds and deliver updates to users.
- [weather_bot_rust ](https://github.com/pxp9/weather_bot_rust ) - A bot that provides weather info. It uses the Fang's asynk module to process updates from Telegram users and schedule weather info.
2021-07-31 07:40:11 +00:00
2021-07-04 06:07:29 +00:00
### Configuration
2022-08-04 17:11:58 +00:00
#### Blocking feature
2022-08-03 08:37:53 +00:00
2022-09-25 13:05:27 +00:00
Just use `TypeBuilder` for `WorkerPool` .
2021-07-04 06:07:29 +00:00
2022-08-04 17:11:58 +00:00
#### Asynk feature
2022-08-03 08:37:53 +00:00
2022-09-25 13:05:27 +00:00
Just use `TypeBuilder` for `AsyncWorkerPool` .
2022-08-03 08:37:53 +00:00
2021-07-04 06:07:29 +00:00
### Configuring the type of workers
### Configuring retention mode
By default, all successfully finished tasks are removed from the DB, failed tasks aren't.
There are three retention modes you can use:
```rust
pub enum RetentionMode {
2022-09-03 11:05:58 +00:00
KeepAll, // doesn't remove tasks
RemoveAll, // removes all tasks
RemoveFinished, // default value
2021-07-04 06:07:29 +00:00
}
```
2022-09-03 08:22:12 +00:00
Set retention mode with worker pools `TypeBuilder` in both modules.
2021-07-04 06:07:29 +00:00
### Configuring sleep values
2022-08-04 17:11:58 +00:00
#### Blocking feature
2022-09-25 13:05:27 +00:00
You can use use `SleepParams` to configure sleep values:
2021-07-04 06:07:29 +00:00
```rust
pub struct SleepParams {
2022-09-03 11:05:58 +00:00
pub sleep_period: Duration, // default value is 5 seconds
pub max_sleep_period: Duration, // default value is 15 seconds
pub min_sleep_period: Duration, // default value is 5 seconds
pub sleep_step: Duration, // default value is 5 seconds
2022-08-03 08:37:53 +00:00
}
2021-07-04 06:07:29 +00:00
```
If there are no tasks in the DB, a worker sleeps for `sleep_period` and each time this value increases by `sleep_step` until it reaches `max_sleep_period` . `min_sleep_period` is the initial value for `sleep_period` . All values are in seconds.
Use `set_sleep_params` to set it:
```rust
let sleep_params = SleepParams {
2022-08-18 11:39:10 +00:00
sleep_period: Duration::from_secs(2),
max_sleep_period: Duration::from_secs(6),
min_sleep_period: Duration::from_secs(2),
sleep_step: Duration::from_secs(1),
2021-07-04 06:07:29 +00:00
};
2022-08-03 08:37:53 +00:00
```
2022-09-03 08:22:12 +00:00
Set sleep params with worker pools `TypeBuilder` in both modules.
2022-08-03 08:37:53 +00:00
2021-06-24 09:58:02 +00:00
## Contributing
1. [Fork it! ](https://github.com/ayrat555/fang/fork )
2. Create your feature branch (`git checkout -b my-new-feature`)
3. Commit your changes (`git commit -am 'Add some feature'`)
4. Push to the branch (`git push origin my-new-feature`)
2022-09-25 13:05:27 +00:00
5. Create a new Pull Request
2021-06-24 09:58:02 +00:00
2021-12-05 06:19:08 +00:00
### Running tests locally
2022-09-03 08:22:12 +00:00
- Install diesel_cli.
2021-12-05 06:19:08 +00:00
```
cargo install diesel_cli
2022-09-03 08:22:12 +00:00
```
2022-09-25 13:05:27 +00:00
- Install docker on your machine.
2021-12-05 06:19:08 +00:00
2022-09-03 08:22:12 +00:00
- Run a Postgres docker container. (See in Makefile.)
```
make db
```
2021-12-05 06:19:08 +00:00
2022-09-03 08:22:12 +00:00
- Run the migrations
```
make diesel
```
2021-12-05 06:19:08 +00:00
2022-09-03 08:22:12 +00:00
- Run tests
```
make tests
```
2021-12-05 06:19:08 +00:00
2022-09-03 08:22:12 +00:00
- Run dirty//long tests, DB must be recreated afterwards.
```
make ignored
```
2021-12-05 06:19:08 +00:00
2022-09-25 13:05:27 +00:00
- Kill the docker container
2022-09-03 08:22:12 +00:00
```
make stop
2021-12-05 06:19:08 +00:00
```
2022-08-03 08:37:53 +00:00
## Authors
2021-06-24 09:58:02 +00:00
2022-08-03 08:37:53 +00:00
- Ayrat Badykov (@ayrat555)
2022-08-05 07:06:32 +00:00
2022-08-03 08:37:53 +00:00
- Pepe Márquez (@pxp9)
2021-07-11 10:55:52 +00:00
[s1]: https://img.shields.io/crates/v/fang.svg
[docs-badge]: https://img.shields.io/badge/docs-website-blue.svg
[ci]: https://crates.io/crates/fang
[docs]: https://docs.rs/fang/
[ga-test]: https://github.com/ayrat555/fang/actions/workflows/rust.yml/badge.svg
[ga-style]: https://github.com/ayrat555/fang/actions/workflows/style.yml/badge.svg
2021-12-05 06:19:08 +00:00
[signal-hook]: https://crates.io/crates/signal-hook