mirrors/bonfire-app
1
0
Fork 0
mirror of https://github.com/bonfire-networks/bonfire-app.git synced 2024-04-28 23:38:44 +00:00
Code Issues Projects Releases Wiki Activity

docs

Browse source
This commit is contained in:
Mayel de Borniol 2023-02-15 13:51:35 +13:00
parent 9f2f13a7b6
commit ff42ba28a6
4 changed files with 63 additions and 149 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

29
docs/ARCHITECTURE.md
View file

@ -2,11 +2,9 @@
## Hacking
Bonfire is an unusual piece of software, developed in an unusual way.
It originally started with requests by Moodle users to be able to share and collaborate on educational resources with their peers and has been forked and evolved a lot since then.
Bonfire is an unusual piece of software, developed in an unusual way. It originally started as a project to create a generic federation library/framework, while building an app for educators to share and collaborate on learning resources with their peers, and has been forked and evolved a lot since then.
Hacking on it is actually pretty fun. The codebase has a unique feeling to work with and we've relentlessly refactored to manage the ever-growing complexity that a distributed social networking toolkit implies.
That said, it is not easy to understand without context, which is what this document is here to provide.
Hacking on it is actually pretty fun. The codebase has a unique feeling to work with and we've relentlessly refactored to manage the ever-growing complexity that a distributed social networking toolkit implies. That said, it is not easy to understand without context, which is what this document is here to provide.
## Design Decisions
@ -24,33 +22,21 @@ Operational goals:
## Stack
Our main implementation language is [Elixir](https://www.elixir-lang.org/), which is designed for
building reliable systems. We have almost [our own dialect](./BONFIRE-FLAVOURED-ELIXIR.md).
Our main implementation language is [Elixir](https://www.elixir-lang.org/), which is designed for building reliable systems. We have almost [our own dialect](./BONFIRE-FLAVOURED-ELIXIR.md).
We use the [Phoenix](https://www.phoenixframework.org/) web framework with
[LiveView](https://hexdocs.pm/phoenix_live_view/) and [Surface](https://surface-ui.org/documentation)
for UI components and views.
We use the [Phoenix](https://www.phoenixframework.org/) web framework with [LiveView](https://hexdocs.pm/phoenix_live_view/) and [Surface](https://surface-ui.org/documentation) for UI components and views.
Surface is a different syntax for LiveView that is designed to be more convenient and understandable
to frontend developers, with extra compile time checks. Surface views and components are compiled
into LiveView code (so once you hit runtime, Surface in effect doesn't exist any more).
Surface is a different syntax for LiveView that is designed to be more convenient and understandable to frontend developers, with extra compile time checks. Surface views and components are compiled into LiveView code (so once you hit runtime, Surface in effect doesn't exist any more).
Some extensions use the [Absinthe](https://absinthe-graphql.org/) GraphQL library to expose an API.
## The Bonfire Environment
We like to think of bonfire as a comfortable way of developing software - there are a lot of
conveniences built in once you know how they all work. The gotcha is that while you don't know them, it can be a bit overwhelming. Don't worry, we've got your back.
* [Bonfire-flavoured Elixir](./BONFIRE-FLAVOURED-ELIXIR.md) - an introduction to the way write elixir.
* [Bonfire's Database: an Introduction](./DATABASE.md) - an overview of how our database is designed.
* [Boundaries](./BOUNDARIES.md) - an introduction to our access control system.
Note: these are still at the early draft stage, we expect to gradually improve documentation over time.
We like to think of bonfire as a comfortable way of developing software - there are a lot of conveniences built in once you know how they all work. The gotcha is that while you don't know them, it can be a bit overwhelming. Don't worry, we've got your back.
## Code Structure
The code is broadly composed of these namespaces, many of which are packaged as "extensions" which are in separate git repos, and are included in the app by way of mix dependencies.
The code is broadly composed namespaces such as these, many of which are packaged as "extensions" which live in separate git repositories, which are included in the app by way of mix dependencies:
- `Bonfire.*` - Core application logic (very little code).
- `Bonfire.*.*` - Bonfire extensions (eg `Bonfire.Social.Posts`) containing mostly context modules, APIs, and routes
@ -81,7 +67,6 @@ EconomicResources.many(deleted: true) # List any deleted resources
Context modules also have functions for creating, updating and deleting objects, as well as hooks for federating or indexing in the search engine.
Here is an incomplete sample of some of current extensions and modules:
- `Bonfire.Me.Accounts` (for managing and querying local user accounts)

14
docs/BONFIRE-FLAVOURED-ELIXIR.md
View file

@ -42,8 +42,7 @@ end
```
A few little extra features you might notice here:
* You can move the parameter into a subexpression, as in `2 |> double_fst(double(...), 1)` where
double will be called before the parameter is passed to `double_fst`.
* You can move the parameter into a subexpression, as in `2 |> double_fst(double(...), 1)` where double will be called before the parameter is passed to `double_fst`.
* You can use `...` multiple times, substituting it in multiple places.
* The right hand side need not even be a function call, you can use any expression with `...`.
@ -103,14 +102,15 @@ You will find the codebase uses this a lot, though the debugs are frequently com
`id = ulid(object) || raise(Bonfire.Fail, :not_found)`
You can use this special exception when you want to redirect the user to the login page rather than just show an error:
`user = current_user(assigns) || raise(Bonfire.Fail.Auth, :needs_login)`
Advantages include:
- standardised error messages (defaults are defined at https://github.com/bonfire-networks/bonfire_fail/blob/main/lib/runtime_config.ex#L16) which can be overridden in your app's config using `config :bonfire_fail, :common_errors`
- messages are defined in one place, which means no duplicated localisation efforts
- friendly error messages are defined in one place, which means no duplicated localisation efforts
- uses the elixir/OTP pattern of "let it crash"
- returns the correct HTTP code when applicable
- no need to wrap blocks in if/else or the like
- for users of the LiveView frontend, this will make the corresponding friendly error message appear in a flash overlay (if using `Bonfire.UI.Common.LiveHandlers` and/or `Bonfire.UI.Common.undead/3`)
Note that when users of the LiveView frontend, this will make the corresponding friendly error message appear in flash overlay.
You can also use this exception when you want to redirect the user to the login page:
`user = current_user(assigns) || raise(Bonfire.Fail.Auth, :needs_login)`

53
docs/BOUNDARIES.md
View file

@ -1,29 +1,20 @@
# Boundaries & Access Control
Boundaries is Bonfire's flexible framework for full
per-user/per-object/per-action access control. It makes it easy to
ensure that users may only see or do what they are supposed to.
Boundaries is Bonfire's flexible framework for full per-user/per-object/per-action access control. It makes it easy to ensure that users may only see or do what they are supposed to.
## Users and Circles
Ignoring any future bot support, boundaries ultimately apply to users.
Circles are a way of categorising users. Each user has their own set
of circles that they can add to and categorise other users in as they
please.
Circles are a way of categorising users. Each user has their own set of circles that they can add to and categorise other users in as they please.
Circles allow a user to categorise work colleagues differently from
friends, for example. They can choose to allow different interactions
from users in the two circles or limit which content each sees on a
per-item basis.
Circles allow a user to categorise work colleagues differently from friends, for example. They can choose to allow different interactions from users in the two circles or limit which content each sees on a per-item basis.
## Verbs
Verbs represent actions that the user could perform, such as reading a
post or replying to a message.
Verbs represent actions that the user could perform, such as reading a post or replying to a message.
Each verb has a unique ID, like the table IDs from `pointers`, which
must be known to the system through configuration.
Each verb has a unique ID, like the table IDs from `pointers`, which must be known to the system through configuration.
## Permissions
@ -37,9 +28,7 @@ Permissions can take one of three values:
`nil` represents `no answer` - in isolation, it is the same as `false`.
Because a user could be in more than one circle and each circle may
have a different permission, we need a way of combining permissions to
produce a final result permission. `nil` is treated differently here:
Because a user could be in more than one circle and each circle may have a different permission, we need a way of combining permissions to produce a final result permission. `nil` is treated differently here:
left | right | result
:------ | :------ | :-----
@ -53,34 +42,20 @@ left | right | result
`false` | `true` | `false`
`false` | `false` | `false`
To be considered granted, the result of combining the permissions must
be `true` (`nil` is as good as `false` again here).
To be considered granted, the result of combining the permissions must be `true` (`nil` is as good as `false` again here).
`nil` can thus be seen as a sort of `weak false`, being easily
overridden by a true, but also not by itself granting anything.
`nil` can thus be seen as a sort of `weak false`, being easily overridden by a true, but also not by itself granting anything.
At first glance, this may seem a little odd, but it gives us a little
additional flexibility which is useful for implementing features such
as blocks (where `false` is really useful!). With a little practice,
it feels quite natural to use.
At first glance, this may seem a little odd, but it gives us a little additional flexibility which is useful for implementing features such as blocks (where `false` is really useful!). With a little practice, it feels quite natural to use.
## ACLs and Grants
An ACL is "just" a collection of Grants.
An `ACL` is "just" a collection of `Grant`s.
Grants combine the ID of the ACL they exist in with a verb id, a user
or circle id and a permission, thus providing a decision about whether
a particular action is permitted for a particular user (or all users
in a particular circle).
Grants combine the ID of the ACL they exist in with a verb id, a user or circle id and a permission, thus providing a decision about whether a particular action is permitted for a particular user (or all users in a particular circle).
Conceptually, an ACL contains a grant for every user-or-circle/verb
combination, but most of the permissions are `nil`. We do not record
grants with `nil` permissions in the database, saving substantially
on storage space and compute requirements.
Conceptually, an ACL contains a grant for every user-or-circle/verb combination, but most of the permissions are `nil`. We do not record grants with `nil` permissions in the database, saving substantially on storage space and compute requirements.
## `Controlled` - Applying boundaries to an object
## Controlled - Applying boundaries to an object
An object is linked to one or more `ACL`s by the `Controlled`
multimixin, which pairs an object ID with an ACL ID. Because it is a
multimixin, a given object can have multiple ACLs applied. In the case
of overlap, permissions are combined in the manner described earlier.
An object is linked to one or more `ACL`s by the `Controlled` multimixin, which pairs an object ID with an ACL ID. Because it is a multimixin, a given object can have multiple ACLs applied. In the case of overlap, permissions are combined in the manner described earlier.

116
docs/DATABASE.md
View file

@ -1,23 +1,18 @@
# Bonfire's Database - an intro
Bonfire uses the excellent PostgreSQL database for most data storage. PostgreSQL allows us to make a
wide range of queries and to make them relatively fast while upholding data integrity guarantees.
Bonfire uses the excellent PostgreSQL database for most data storage. PostgreSQL allows us to make a wide range of queries and to make them relatively fast while upholding data integrity guarantees.
Postgres is a relational schema-led database - it expects you to pre-define tables and the fields
in each table (represented in tabular form, i.e. as a collection of tables with each table consisting
of a set of rows and columns). Fields can contain data or a reference to a row in another table.
This usually means that a field containing a reference has to be pre-defined with a foreign key
pointing to a specific field (typically a primary key, like an ID column) *in a specific table*.
Postgres is a relational schema-led database - it expects you to pre-define tables and the fields in each table (represented in tabular form, i.e. as a collection of tables with each table consisting of a set of rows and columns). Fields can contain data or a reference to a row in another table.
This usually means that a field containing a reference has to be pre-defined with a foreign key pointing to a specific field (typically a primary key, like an ID column) *in a specific table*.
A simple example would be a blogging app, which might have a `post` table with `author` field that references the `user` table.
A social network, by contrast, is actually a graph of objects. Objects need to be able to refer
to other objects by their ID without knowing their type.
A social network, by contrast, is actually a graph of objects. Objects need to be able to refer to other objects by their ID without knowing their type.
A simple example would be likes, you might have a `likes` table with `liked_object` field that references the `post` table. But you don't just have posts that can be liked, but also videos, images, polls, etc, each with their own table?
We needed the flexibility to have a foreign key that can reference any referenceable object.
We call our system `Pointers`.
We needed the flexibility to have a foreign key that can reference any referenceable object. We call our system `Pointers`.
This guide is a brief introduction to Pointers. It assumes some foundational knowledge:
@ -26,48 +21,37 @@ This guide is a brief introduction to Pointers. It assumes some foundational kno
* What a primary key is and why it's useful.
* Foreign keys and relationships between tables (1 to 1, 1 to Many, Many to 1, Many to Many).
* Views as virtual tables backed by a SQL query.
* Basic understanding of Elixir (enough to follow the examples).
* Basic working knowledge of the [Ecto](https://hexdocs.pm/ecto/Ecto.html) database library (schema and migration definitions)
* Basic working knowledge of the [Ecto](https://hexdocs.pm/ecto/Ecto.html) database library (schema and migration definitions)
## Identifying objects - the ULID type
All referenceable objects in the system have a unique ID (primary key) whose type is the
[`ULID`](https://github.com/ulid/spec). It's a lot like a `UUID` in that you can generate unique ones
independently of the database. It's also a little different, being made up of two parts:
All referenceable objects in the system have a unique ID (primary key) whose type is the [`ULID`](https://github.com/ulid/spec). It's a lot like a `UUID` in that you can generate unique ones independently of the database. It's also a little different, being made up of two parts:
* The current timestamp, to millisecond precision.
* Strong random padding for uniqueness.
This means that it naturally sorts by time to the millisecond (close enough for us), giving us a
performance advantage on queries ordered by a separate creation datetime field (by contrast, UUIDv4 is
randomly distributed).
This means that it naturally sorts by time to the millisecond (close enough for us), giving us a performance advantage on queries ordered by a separate creation datetime field (by contrast, UUIDv4 is randomly distributed).
If you've only worked with integer primary keys before, you are probably used to letting the
database dispense an ID for you. With `ULID` (or `UUID`), IDs can be known *before* they are stored,
greatly easing the process of storing a graph of data and allowing us to do more of the preparation
work outside of a transaction for increased performance.
If you've only worked with integer primary keys before, you are probably used to letting the database dispense an ID for you. With `ULID` (or `UUID`), IDs can be known *before* they are stored, greatly easing the process of storing a graph of data and allowing us to do more of the preparation work outside of a transaction for increased performance.
In PostgreSQL, we actually store `ULID`s as `UUID` columns, thanks to both being the same size
(and the lack of a `ULID` column type shipping with postgresql). You mostly will not notice this
because it's handled for you, but there are a few places it can come up:
In PostgreSQL, we actually store `ULID`s as `UUID` columns, thanks to both being the same size (and the lack of a `ULID` column type shipping with postgresql). You mostly will not notice this because it's handled for you, but there are a few places it can come up:
* Ecto debug and error output may show either binary values or UUID-formatted values.
* Hand-written SQL may need to convert table IDs to the `UUID` format before use.
## It's just a table
The `pointers` system is mostly based around a single table represented by the `Pointers.Pointer`
schema with the following fields:
The `pointers` system is mostly based around a single table represented by the `Pointers.Pointer` schema with the following fields:
* `id` (ULID) - the database-wide unique id for the object, primary key.
* `table_id` (ULID) - identifies the type of the object, references `Pointers.Table`.
* `deleted_at` (timestamp, default: `null`) - when the object was deleted.
Every object that is stored in the system will have a record in this table. It may also have records
in other tables (handy for storing more than 3 fields about the object!).
Every object that is stored in the system will have a record in this table. It may also have records in other tables (handy for storing more than 3 fields about the object!).
Don't worry about `Pointers.Table` for now, just know that every object type will have a
record there so `Pointers.Pointer.table_id` can reference it.
Don't worry about `Pointers.Table` for now, just know that every object type will have a record there so `Pointers.Pointer.table_id` can reference it.
## Mixins - storing data about objects
@ -78,14 +62,9 @@ record or not record information for each mixin. Sample mixins include:
* post content (containing the title, summary, and/or html body of a post or message)
* created (containing the id of the object creator)
In this way, they are reusable across different object types. One mixin may (or may not) be used
by any number of objects. This is mostly driven by the type of the object we are storing,
but can also be driven by user input.
In this way, they are reusable across different object types. One mixin may (or may not) be used by any number of objects. This is mostly driven by the type of the object we are storing, but can also be driven by user input.
Mixins are just tables too! The only requirement is they have a `ULID` primary key which references
`Pointers.Pointer`. The developer of the mixin is free to put whatever other fields they want in the
table, so long as they have that primary-key-as-reference (which will be automatically added for you
by the `mixin_schema` macro).
Mixins are just tables too! The only requirement is they have a `ULID` primary key which references `Pointers.Pointer`. The developer of the mixin is free to put whatever other fields they want in the table, so long as they have that primary-key-as-reference (which will be automatically added for you by the `mixin_schema` macro).
Here is a sample mixin definition for a user profile:
@ -117,11 +96,9 @@ We will cover dynamic configuration later. For now, you can use the OTP app that
## Multimixins
Multimixins are like mixins, except that where an object may have 0 or 1 of a particular mixins, an
object may have any number of a particular multimixin.
Multimixins are like mixins, except that where an object may have 0 or 1 of a particular mixins, an object may have any number of a particular multimixin.
For this to work, a multimixin must have a *compound primary key* which must contain an `id` column
referencing `Pointers.Pointer` and at least one other field which will collectively be unique.
For this to work, a multimixin must have a *compound primary key* which must contain an `id` column referencing `Pointers.Pointer` and at least one other field which will collectively be unique.
An example multimixin is used for publishing an item to feeds:
@ -140,29 +117,22 @@ defmodule Bonfire.Data.Social.FeedPublish do
end
```
Notice that this looks very similar to defining a mixin. Indeed, the only difference is the
`primary_key: true` in this line, which adds a second field to the compound primary key.
This results in ecto recording a compound primary key of `(id, feed_id)` for the schema (the id is
added for you as with regular mixins).
Notice that this looks very similar to defining a mixin. Indeed, the only difference is the `primary_key: true` in this line, which adds a second field to the compound primary key.
This results in ecto recording a compound primary key of `(id, feed_id)` for the schema (the id is added for you as with regular mixins).
## Declaring Object Types
### Picking a table id
The first step to declaring a type is picking a unique table ID in ULID format. You could just
generate one at the terminal, but since these IDs are special, we tend to assign a synthetic ULID
that are readable as words so they stand out in debug output.
The first step to declaring a type is picking a unique table ID in ULID format. You could just generate one at the terminal, but since these IDs are special, we tend to assign a synthetic ULID that are readable as words so they stand out in debug output.
For example, the ID for the `Feed` table is: `1TFEEDS0NTHES0V1S0FM0RTA1S`, which can be read as "It
feeds on the souls of mortals". Feel free to have a little fun coming up with them, it makes debug
output a little more cheery! The rules are:
For example, the ID for the `Feed` table is: `1TFEEDS0NTHES0V1S0FM0RTA1S`, which can be read as "It feeds on the souls of mortals". Feel free to have a little fun coming up with them, it makes debug output a little more cheery! The rules are:
* The alphabet is [Crockford's Base32](https://en.wikipedia.org/wiki/Base32#Crockford's_Base32).
* They must be 26 characters in length.
* The first character must be a digit in the range 0-7.
To help you with this, the `Pointers.ULID.synthesise!/1` method takes an alphanumeric
binary and tries to return you it transliterated into a valid ULID. Example usage:
To help you with this, the `Pointers.ULID.synthesise!/1` method takes an alphanumeric binary and tries to return you it transliterated into a valid ULID. Example usage:
```
iex(1)> Pointers.ULID.synthesise!("itfeedsonthesouls")
@ -201,38 +171,26 @@ defmodule Bonfire.Data.Social.Block do
end
```
It should look quite similar to a mixin definition, except that we `use` `Pointers.Virtual` this time
(passing an additional `table_id` argument) and we call the `virtual_schema` macro.
It should look quite similar to a mixin definition, except that we `use` `Pointers.Virtual` this time (passing an additional `table_id` argument) and we call the `virtual_schema` macro.
The primary limitation of a virtual is that you cannot put extra fields into one. This also means
that `belongs_to` is not generally permitted because it results in adding a field. `has_one` and
`has_many` work just fine as they do not cause the creation of fields in the schema.
The primary limitation of a virtual is that you cannot put extra fields into one. This also means that `belongs_to` is not generally permitted because it results in adding a field. `has_one` and `has_many` work just fine as they do not cause the creation of fields in the schema.
This is not usually a problem, as extra fields can be put into mixins or multimixins as appropriate.
Under the hood, a virtual has a view (in this example, called `bonfire_data_social_block`). It looks
like a table with just an id, but it's populated with all the ids of blocks that are not
deleted. When the view is inserted into, a record is created in the `pointers` table for you
transparently. When you delete from the view, the corresponding `pointers` entry is marked deleted
for you.
Under the hood, a virtual has a view (in this example, called `bonfire_data_social_block`). It looks like a table with just an id, but it's populated with all the ids of blocks that are not deleted. When the view is inserted into, a record is created in the `pointers` table for you transparently. When you delete from the view, the corresponding `pointers` entry is marked deleted for you.
### Pointables
The other, lesser used, type of object is called the Pointable. The major difference is that unlike
the simple case of virtuals, pointables are not backed by views, but by tables.
The other, lesser used, type of object is called the Pointable. The major difference is that unlike the simple case of virtuals, pointables are not backed by views, but by tables.
When a record is inserted into a pointable table, a copy is made in the `pointers` table for you
transparently. When you delete from the table, the the corresponding `pointers` entry is marked
deleted for you. In these ways, they behave very much like virtuals. By having a table, however, we
are free to add new fields.
When a record is inserted into a pointable table, a copy is made in the `pointers` table for you transparently. When you delete from the table, the the corresponding `pointers` entry is marked deleted for you. In these ways, they behave very much like virtuals. By having a table, however, we are free to add new fields.
Pointables pay for this flexibility by being slightly more expensive than virtuals:
* Records must be inserted into/deleted from two tables (the pointable's table and the `pointers` table).
* The pointable table needs its own primary key index.
Here is a definition of a pointable type (indicating an ActivityPub activity whose type we don't
recognise, stored as a JSON blob):
Here is a definition of a pointable type (indicating an ActivityPub activity whose type we don't recognise, stored as a JSON blob):
```elixir
defmodule Bonfire.Data.Social.APActivity do
@ -252,8 +210,7 @@ The choice of using a pointable instead of a virtual combined with one or more m
## Writing Migrations
Migrations are typically included along with the schemas as public APIs you can call within your
project's migrations.
Migrations are typically included along with the schemas as public APIs you can call within your project's migrations.
### Virtuals
@ -270,8 +227,7 @@ defmodule Bonfire.Data.Social.Post.Migration do
end
```
If you need to do more work, it can be a little trickier. Here's an example for `block`, which
also creates a unique index on another table:
If you need to do more work, it can be a little trickier. Here's an example for `block`, which also creates a unique index on another table:
```elixir
defmodule Bonfire.Data.Social.Block.Migration do
@ -300,8 +256,7 @@ defmodule Bonfire.Data.Social.Block.Migration do
end
```
Notice how we have to write our `up` and `down` versions separately to get the correct
ordering of operations. Handling down migrations can be a bit awkward in ecto.
Notice how we have to write our `up` and `down` versions separately to get the correct ordering of operations.
### Pointables
@ -477,6 +432,5 @@ scenarios by now:
* [bonfire_data_identity](https://github.com/bonfire-networks/bonfire_data_identity/)
* [bonfire_data_edges](https://github.com/bonfire-networks/bonfire_data_edges/) (feat. bonus triggers)
If you want to know exactly what's happening, I can only suggest reading the code for
[Pointers.Migration](https://github.com/bonfire-networks/pointers/blob/main/lib/migration.ex), it's
surprisingly readable.
If you want to know exactly what's happening, you may want to read the code for
[Pointers.Migration](https://github.com/bonfire-networks/pointers/blob/main/lib/migration.ex).