From ff42ba28a6de3a85e4cab872aeb79a2e1cf2add0 Mon Sep 17 00:00:00 2001 From: Mayel de Borniol Date: Wed, 15 Feb 2023 13:51:35 +1300 Subject: [PATCH] docs --- docs/ARCHITECTURE.md | 29 ++------ docs/BONFIRE-FLAVOURED-ELIXIR.md | 14 ++-- docs/BOUNDARIES.md | 53 ++++---------- docs/DATABASE.md | 116 ++++++++++--------------------- 4 files changed, 63 insertions(+), 149 deletions(-) diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index 094663e6bb..35d3a5b914 100755 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -2,11 +2,9 @@ ## Hacking -Bonfire is an unusual piece of software, developed in an unusual way. -It originally started with requests by Moodle users to be able to share and collaborate on educational resources with their peers and has been forked and evolved a lot since then. +Bonfire is an unusual piece of software, developed in an unusual way. It originally started as a project to create a generic federation library/framework, while building an app for educators to share and collaborate on learning resources with their peers, and has been forked and evolved a lot since then. -Hacking on it is actually pretty fun. The codebase has a unique feeling to work with and we've relentlessly refactored to manage the ever-growing complexity that a distributed social networking toolkit implies. -That said, it is not easy to understand without context, which is what this document is here to provide. +Hacking on it is actually pretty fun. The codebase has a unique feeling to work with and we've relentlessly refactored to manage the ever-growing complexity that a distributed social networking toolkit implies. That said, it is not easy to understand without context, which is what this document is here to provide. ## Design Decisions @@ -24,33 +22,21 @@ Operational goals: ## Stack -Our main implementation language is [Elixir](https://www.elixir-lang.org/), which is designed for -building reliable systems. We have almost [our own dialect](./BONFIRE-FLAVOURED-ELIXIR.md). +Our main implementation language is [Elixir](https://www.elixir-lang.org/), which is designed for building reliable systems. We have almost [our own dialect](./BONFIRE-FLAVOURED-ELIXIR.md). -We use the [Phoenix](https://www.phoenixframework.org/) web framework with -[LiveView](https://hexdocs.pm/phoenix_live_view/) and [Surface](https://surface-ui.org/documentation) -for UI components and views. +We use the [Phoenix](https://www.phoenixframework.org/) web framework with [LiveView](https://hexdocs.pm/phoenix_live_view/) and [Surface](https://surface-ui.org/documentation) for UI components and views. -Surface is a different syntax for LiveView that is designed to be more convenient and understandable -to frontend developers, with extra compile time checks. Surface views and components are compiled -into LiveView code (so once you hit runtime, Surface in effect doesn't exist any more). +Surface is a different syntax for LiveView that is designed to be more convenient and understandable to frontend developers, with extra compile time checks. Surface views and components are compiled into LiveView code (so once you hit runtime, Surface in effect doesn't exist any more). Some extensions use the [Absinthe](https://absinthe-graphql.org/) GraphQL library to expose an API. ## The Bonfire Environment -We like to think of bonfire as a comfortable way of developing software - there are a lot of -conveniences built in once you know how they all work. The gotcha is that while you don't know them, it can be a bit overwhelming. Don't worry, we've got your back. - -* [Bonfire-flavoured Elixir](./BONFIRE-FLAVOURED-ELIXIR.md) - an introduction to the way write elixir. -* [Bonfire's Database: an Introduction](./DATABASE.md) - an overview of how our database is designed. -* [Boundaries](./BOUNDARIES.md) - an introduction to our access control system. - -Note: these are still at the early draft stage, we expect to gradually improve documentation over time. +We like to think of bonfire as a comfortable way of developing software - there are a lot of conveniences built in once you know how they all work. The gotcha is that while you don't know them, it can be a bit overwhelming. Don't worry, we've got your back. ## Code Structure -The code is broadly composed of these namespaces, many of which are packaged as "extensions" which are in separate git repos, and are included in the app by way of mix dependencies. +The code is broadly composed namespaces such as these, many of which are packaged as "extensions" which live in separate git repositories, which are included in the app by way of mix dependencies: - `Bonfire.*` - Core application logic (very little code). - `Bonfire.*.*` - Bonfire extensions (eg `Bonfire.Social.Posts`) containing mostly context modules, APIs, and routes @@ -81,7 +67,6 @@ EconomicResources.many(deleted: true) # List any deleted resources Context modules also have functions for creating, updating and deleting objects, as well as hooks for federating or indexing in the search engine. - Here is an incomplete sample of some of current extensions and modules: - `Bonfire.Me.Accounts` (for managing and querying local user accounts) diff --git a/docs/BONFIRE-FLAVOURED-ELIXIR.md b/docs/BONFIRE-FLAVOURED-ELIXIR.md index 1ddd8ffed2..c79f5e2726 100644 --- a/docs/BONFIRE-FLAVOURED-ELIXIR.md +++ b/docs/BONFIRE-FLAVOURED-ELIXIR.md @@ -42,8 +42,7 @@ end ``` A few little extra features you might notice here: -* You can move the parameter into a subexpression, as in `2 |> double_fst(double(...), 1)` where - double will be called before the parameter is passed to `double_fst`. +* You can move the parameter into a subexpression, as in `2 |> double_fst(double(...), 1)` where double will be called before the parameter is passed to `double_fst`. * You can use `...` multiple times, substituting it in multiple places. * The right hand side need not even be a function call, you can use any expression with `...`. @@ -103,14 +102,15 @@ You will find the codebase uses this a lot, though the debugs are frequently com `id = ulid(object) || raise(Bonfire.Fail, :not_found)` +You can use this special exception when you want to redirect the user to the login page rather than just show an error: +`user = current_user(assigns) || raise(Bonfire.Fail.Auth, :needs_login)` + Advantages include: - standardised error messages (defaults are defined at https://github.com/bonfire-networks/bonfire_fail/blob/main/lib/runtime_config.ex#L16) which can be overridden in your app's config using `config :bonfire_fail, :common_errors` -- messages are defined in one place, which means no duplicated localisation efforts +- friendly error messages are defined in one place, which means no duplicated localisation efforts - uses the elixir/OTP pattern of "let it crash" +- returns the correct HTTP code when applicable - no need to wrap blocks in if/else or the like +- for users of the LiveView frontend, this will make the corresponding friendly error message appear in a flash overlay (if using `Bonfire.UI.Common.LiveHandlers` and/or `Bonfire.UI.Common.undead/3`) -Note that when users of the LiveView frontend, this will make the corresponding friendly error message appear in flash overlay. - -You can also use this exception when you want to redirect the user to the login page: -`user = current_user(assigns) || raise(Bonfire.Fail.Auth, :needs_login)` diff --git a/docs/BOUNDARIES.md b/docs/BOUNDARIES.md index da0b8df22d..4b72becacd 100644 --- a/docs/BOUNDARIES.md +++ b/docs/BOUNDARIES.md @@ -1,29 +1,20 @@ # Boundaries & Access Control -Boundaries is Bonfire's flexible framework for full -per-user/per-object/per-action access control. It makes it easy to -ensure that users may only see or do what they are supposed to. +Boundaries is Bonfire's flexible framework for full per-user/per-object/per-action access control. It makes it easy to ensure that users may only see or do what they are supposed to. ## Users and Circles Ignoring any future bot support, boundaries ultimately apply to users. -Circles are a way of categorising users. Each user has their own set -of circles that they can add to and categorise other users in as they -please. +Circles are a way of categorising users. Each user has their own set of circles that they can add to and categorise other users in as they please. -Circles allow a user to categorise work colleagues differently from -friends, for example. They can choose to allow different interactions -from users in the two circles or limit which content each sees on a -per-item basis. +Circles allow a user to categorise work colleagues differently from friends, for example. They can choose to allow different interactions from users in the two circles or limit which content each sees on a per-item basis. ## Verbs -Verbs represent actions that the user could perform, such as reading a -post or replying to a message. +Verbs represent actions that the user could perform, such as reading a post or replying to a message. -Each verb has a unique ID, like the table IDs from `pointers`, which -must be known to the system through configuration. +Each verb has a unique ID, like the table IDs from `pointers`, which must be known to the system through configuration. ## Permissions @@ -37,9 +28,7 @@ Permissions can take one of three values: `nil` represents `no answer` - in isolation, it is the same as `false`. -Because a user could be in more than one circle and each circle may -have a different permission, we need a way of combining permissions to -produce a final result permission. `nil` is treated differently here: +Because a user could be in more than one circle and each circle may have a different permission, we need a way of combining permissions to produce a final result permission. `nil` is treated differently here: left | right | result :------ | :------ | :----- @@ -53,34 +42,20 @@ left | right | result `false` | `true` | `false` `false` | `false` | `false` -To be considered granted, the result of combining the permissions must -be `true` (`nil` is as good as `false` again here). +To be considered granted, the result of combining the permissions must be `true` (`nil` is as good as `false` again here). -`nil` can thus be seen as a sort of `weak false`, being easily -overridden by a true, but also not by itself granting anything. +`nil` can thus be seen as a sort of `weak false`, being easily overridden by a true, but also not by itself granting anything. -At first glance, this may seem a little odd, but it gives us a little -additional flexibility which is useful for implementing features such -as blocks (where `false` is really useful!). With a little practice, -it feels quite natural to use. +At first glance, this may seem a little odd, but it gives us a little additional flexibility which is useful for implementing features such as blocks (where `false` is really useful!). With a little practice, it feels quite natural to use. ## ACLs and Grants -An ACL is "just" a collection of Grants. +An `ACL` is "just" a collection of `Grant`s. -Grants combine the ID of the ACL they exist in with a verb id, a user -or circle id and a permission, thus providing a decision about whether -a particular action is permitted for a particular user (or all users -in a particular circle). +Grants combine the ID of the ACL they exist in with a verb id, a user or circle id and a permission, thus providing a decision about whether a particular action is permitted for a particular user (or all users in a particular circle). -Conceptually, an ACL contains a grant for every user-or-circle/verb -combination, but most of the permissions are `nil`. We do not record -grants with `nil` permissions in the database, saving substantially -on storage space and compute requirements. +Conceptually, an ACL contains a grant for every user-or-circle/verb combination, but most of the permissions are `nil`. We do not record grants with `nil` permissions in the database, saving substantially on storage space and compute requirements. -## `Controlled` - Applying boundaries to an object +## Controlled - Applying boundaries to an object -An object is linked to one or more `ACL`s by the `Controlled` -multimixin, which pairs an object ID with an ACL ID. Because it is a -multimixin, a given object can have multiple ACLs applied. In the case -of overlap, permissions are combined in the manner described earlier. +An object is linked to one or more `ACL`s by the `Controlled` multimixin, which pairs an object ID with an ACL ID. Because it is a multimixin, a given object can have multiple ACLs applied. In the case of overlap, permissions are combined in the manner described earlier. \ No newline at end of file diff --git a/docs/DATABASE.md b/docs/DATABASE.md index 203f46c450..509c876494 100644 --- a/docs/DATABASE.md +++ b/docs/DATABASE.md @@ -1,23 +1,18 @@ # Bonfire's Database - an intro -Bonfire uses the excellent PostgreSQL database for most data storage. PostgreSQL allows us to make a -wide range of queries and to make them relatively fast while upholding data integrity guarantees. +Bonfire uses the excellent PostgreSQL database for most data storage. PostgreSQL allows us to make a wide range of queries and to make them relatively fast while upholding data integrity guarantees. -Postgres is a relational schema-led database - it expects you to pre-define tables and the fields -in each table (represented in tabular form, i.e. as a collection of tables with each table consisting -of a set of rows and columns). Fields can contain data or a reference to a row in another table. -This usually means that a field containing a reference has to be pre-defined with a foreign key -pointing to a specific field (typically a primary key, like an ID column) *in a specific table*. +Postgres is a relational schema-led database - it expects you to pre-define tables and the fields in each table (represented in tabular form, i.e. as a collection of tables with each table consisting of a set of rows and columns). Fields can contain data or a reference to a row in another table. + +This usually means that a field containing a reference has to be pre-defined with a foreign key pointing to a specific field (typically a primary key, like an ID column) *in a specific table*. A simple example would be a blogging app, which might have a `post` table with `author` field that references the `user` table. -A social network, by contrast, is actually a graph of objects. Objects need to be able to refer -to other objects by their ID without knowing their type. +A social network, by contrast, is actually a graph of objects. Objects need to be able to refer to other objects by their ID without knowing their type. A simple example would be likes, you might have a `likes` table with `liked_object` field that references the `post` table. But you don't just have posts that can be liked, but also videos, images, polls, etc, each with their own table? -We needed the flexibility to have a foreign key that can reference any referenceable object. -We call our system `Pointers`. +We needed the flexibility to have a foreign key that can reference any referenceable object. We call our system `Pointers`. This guide is a brief introduction to Pointers. It assumes some foundational knowledge: @@ -26,48 +21,37 @@ This guide is a brief introduction to Pointers. It assumes some foundational kno * What a primary key is and why it's useful. * Foreign keys and relationships between tables (1 to 1, 1 to Many, Many to 1, Many to Many). * Views as virtual tables backed by a SQL query. + * Basic understanding of Elixir (enough to follow the examples). - * Basic working knowledge of the [Ecto](https://hexdocs.pm/ecto/Ecto.html) database library (schema and migration definitions) +* Basic working knowledge of the [Ecto](https://hexdocs.pm/ecto/Ecto.html) database library (schema and migration definitions) ## Identifying objects - the ULID type -All referenceable objects in the system have a unique ID (primary key) whose type is the -[`ULID`](https://github.com/ulid/spec). It's a lot like a `UUID` in that you can generate unique ones -independently of the database. It's also a little different, being made up of two parts: +All referenceable objects in the system have a unique ID (primary key) whose type is the [`ULID`](https://github.com/ulid/spec). It's a lot like a `UUID` in that you can generate unique ones independently of the database. It's also a little different, being made up of two parts: * The current timestamp, to millisecond precision. * Strong random padding for uniqueness. -This means that it naturally sorts by time to the millisecond (close enough for us), giving us a -performance advantage on queries ordered by a separate creation datetime field (by contrast, UUIDv4 is -randomly distributed). +This means that it naturally sorts by time to the millisecond (close enough for us), giving us a performance advantage on queries ordered by a separate creation datetime field (by contrast, UUIDv4 is randomly distributed). -If you've only worked with integer primary keys before, you are probably used to letting the -database dispense an ID for you. With `ULID` (or `UUID`), IDs can be known *before* they are stored, -greatly easing the process of storing a graph of data and allowing us to do more of the preparation -work outside of a transaction for increased performance. +If you've only worked with integer primary keys before, you are probably used to letting the database dispense an ID for you. With `ULID` (or `UUID`), IDs can be known *before* they are stored, greatly easing the process of storing a graph of data and allowing us to do more of the preparation work outside of a transaction for increased performance. -In PostgreSQL, we actually store `ULID`s as `UUID` columns, thanks to both being the same size -(and the lack of a `ULID` column type shipping with postgresql). You mostly will not notice this -because it's handled for you, but there are a few places it can come up: +In PostgreSQL, we actually store `ULID`s as `UUID` columns, thanks to both being the same size (and the lack of a `ULID` column type shipping with postgresql). You mostly will not notice this because it's handled for you, but there are a few places it can come up: * Ecto debug and error output may show either binary values or UUID-formatted values. * Hand-written SQL may need to convert table IDs to the `UUID` format before use. ## It's just a table -The `pointers` system is mostly based around a single table represented by the `Pointers.Pointer` -schema with the following fields: +The `pointers` system is mostly based around a single table represented by the `Pointers.Pointer` schema with the following fields: * `id` (ULID) - the database-wide unique id for the object, primary key. * `table_id` (ULID) - identifies the type of the object, references `Pointers.Table`. * `deleted_at` (timestamp, default: `null`) - when the object was deleted. -Every object that is stored in the system will have a record in this table. It may also have records -in other tables (handy for storing more than 3 fields about the object!). +Every object that is stored in the system will have a record in this table. It may also have records in other tables (handy for storing more than 3 fields about the object!). -Don't worry about `Pointers.Table` for now, just know that every object type will have a -record there so `Pointers.Pointer.table_id` can reference it. +Don't worry about `Pointers.Table` for now, just know that every object type will have a record there so `Pointers.Pointer.table_id` can reference it. ## Mixins - storing data about objects @@ -78,14 +62,9 @@ record or not record information for each mixin. Sample mixins include: * post content (containing the title, summary, and/or html body of a post or message) * created (containing the id of the object creator) -In this way, they are reusable across different object types. One mixin may (or may not) be used -by any number of objects. This is mostly driven by the type of the object we are storing, -but can also be driven by user input. +In this way, they are reusable across different object types. One mixin may (or may not) be used by any number of objects. This is mostly driven by the type of the object we are storing, but can also be driven by user input. -Mixins are just tables too! The only requirement is they have a `ULID` primary key which references -`Pointers.Pointer`. The developer of the mixin is free to put whatever other fields they want in the -table, so long as they have that primary-key-as-reference (which will be automatically added for you -by the `mixin_schema` macro). +Mixins are just tables too! The only requirement is they have a `ULID` primary key which references `Pointers.Pointer`. The developer of the mixin is free to put whatever other fields they want in the table, so long as they have that primary-key-as-reference (which will be automatically added for you by the `mixin_schema` macro). Here is a sample mixin definition for a user profile: @@ -117,11 +96,9 @@ We will cover dynamic configuration later. For now, you can use the OTP app that ## Multimixins -Multimixins are like mixins, except that where an object may have 0 or 1 of a particular mixins, an -object may have any number of a particular multimixin. +Multimixins are like mixins, except that where an object may have 0 or 1 of a particular mixins, an object may have any number of a particular multimixin. -For this to work, a multimixin must have a *compound primary key* which must contain an `id` column -referencing `Pointers.Pointer` and at least one other field which will collectively be unique. +For this to work, a multimixin must have a *compound primary key* which must contain an `id` column referencing `Pointers.Pointer` and at least one other field which will collectively be unique. An example multimixin is used for publishing an item to feeds: @@ -140,29 +117,22 @@ defmodule Bonfire.Data.Social.FeedPublish do end ``` -Notice that this looks very similar to defining a mixin. Indeed, the only difference is the -`primary_key: true` in this line, which adds a second field to the compound primary key. -This results in ecto recording a compound primary key of `(id, feed_id)` for the schema (the id is -added for you as with regular mixins). +Notice that this looks very similar to defining a mixin. Indeed, the only difference is the `primary_key: true` in this line, which adds a second field to the compound primary key. +This results in ecto recording a compound primary key of `(id, feed_id)` for the schema (the id is added for you as with regular mixins). ## Declaring Object Types ### Picking a table id -The first step to declaring a type is picking a unique table ID in ULID format. You could just -generate one at the terminal, but since these IDs are special, we tend to assign a synthetic ULID -that are readable as words so they stand out in debug output. +The first step to declaring a type is picking a unique table ID in ULID format. You could just generate one at the terminal, but since these IDs are special, we tend to assign a synthetic ULID that are readable as words so they stand out in debug output. -For example, the ID for the `Feed` table is: `1TFEEDS0NTHES0V1S0FM0RTA1S`, which can be read as "It -feeds on the souls of mortals". Feel free to have a little fun coming up with them, it makes debug -output a little more cheery! The rules are: +For example, the ID for the `Feed` table is: `1TFEEDS0NTHES0V1S0FM0RTA1S`, which can be read as "It feeds on the souls of mortals". Feel free to have a little fun coming up with them, it makes debug output a little more cheery! The rules are: * The alphabet is [Crockford's Base32](https://en.wikipedia.org/wiki/Base32#Crockford's_Base32). * They must be 26 characters in length. * The first character must be a digit in the range 0-7. -To help you with this, the `Pointers.ULID.synthesise!/1` method takes an alphanumeric -binary and tries to return you it transliterated into a valid ULID. Example usage: +To help you with this, the `Pointers.ULID.synthesise!/1` method takes an alphanumeric binary and tries to return you it transliterated into a valid ULID. Example usage: ``` iex(1)> Pointers.ULID.synthesise!("itfeedsonthesouls") @@ -201,38 +171,26 @@ defmodule Bonfire.Data.Social.Block do end ``` -It should look quite similar to a mixin definition, except that we `use` `Pointers.Virtual` this time -(passing an additional `table_id` argument) and we call the `virtual_schema` macro. +It should look quite similar to a mixin definition, except that we `use` `Pointers.Virtual` this time (passing an additional `table_id` argument) and we call the `virtual_schema` macro. -The primary limitation of a virtual is that you cannot put extra fields into one. This also means -that `belongs_to` is not generally permitted because it results in adding a field. `has_one` and -`has_many` work just fine as they do not cause the creation of fields in the schema. +The primary limitation of a virtual is that you cannot put extra fields into one. This also means that `belongs_to` is not generally permitted because it results in adding a field. `has_one` and `has_many` work just fine as they do not cause the creation of fields in the schema. This is not usually a problem, as extra fields can be put into mixins or multimixins as appropriate. -Under the hood, a virtual has a view (in this example, called `bonfire_data_social_block`). It looks -like a table with just an id, but it's populated with all the ids of blocks that are not -deleted. When the view is inserted into, a record is created in the `pointers` table for you -transparently. When you delete from the view, the corresponding `pointers` entry is marked deleted -for you. +Under the hood, a virtual has a view (in this example, called `bonfire_data_social_block`). It looks like a table with just an id, but it's populated with all the ids of blocks that are not deleted. When the view is inserted into, a record is created in the `pointers` table for you transparently. When you delete from the view, the corresponding `pointers` entry is marked deleted for you. ### Pointables -The other, lesser used, type of object is called the Pointable. The major difference is that unlike -the simple case of virtuals, pointables are not backed by views, but by tables. +The other, lesser used, type of object is called the Pointable. The major difference is that unlike the simple case of virtuals, pointables are not backed by views, but by tables. -When a record is inserted into a pointable table, a copy is made in the `pointers` table for you -transparently. When you delete from the table, the the corresponding `pointers` entry is marked -deleted for you. In these ways, they behave very much like virtuals. By having a table, however, we -are free to add new fields. +When a record is inserted into a pointable table, a copy is made in the `pointers` table for you transparently. When you delete from the table, the the corresponding `pointers` entry is marked deleted for you. In these ways, they behave very much like virtuals. By having a table, however, we are free to add new fields. Pointables pay for this flexibility by being slightly more expensive than virtuals: * Records must be inserted into/deleted from two tables (the pointable's table and the `pointers` table). * The pointable table needs its own primary key index. -Here is a definition of a pointable type (indicating an ActivityPub activity whose type we don't -recognise, stored as a JSON blob): +Here is a definition of a pointable type (indicating an ActivityPub activity whose type we don't recognise, stored as a JSON blob): ```elixir defmodule Bonfire.Data.Social.APActivity do @@ -252,8 +210,7 @@ The choice of using a pointable instead of a virtual combined with one or more m ## Writing Migrations -Migrations are typically included along with the schemas as public APIs you can call within your -project's migrations. +Migrations are typically included along with the schemas as public APIs you can call within your project's migrations. ### Virtuals @@ -270,8 +227,7 @@ defmodule Bonfire.Data.Social.Post.Migration do end ``` -If you need to do more work, it can be a little trickier. Here's an example for `block`, which -also creates a unique index on another table: +If you need to do more work, it can be a little trickier. Here's an example for `block`, which also creates a unique index on another table: ```elixir defmodule Bonfire.Data.Social.Block.Migration do @@ -300,8 +256,7 @@ defmodule Bonfire.Data.Social.Block.Migration do end ``` -Notice how we have to write our `up` and `down` versions separately to get the correct -ordering of operations. Handling down migrations can be a bit awkward in ecto. +Notice how we have to write our `up` and `down` versions separately to get the correct ordering of operations. ### Pointables @@ -477,6 +432,5 @@ scenarios by now: * [bonfire_data_identity](https://github.com/bonfire-networks/bonfire_data_identity/) * [bonfire_data_edges](https://github.com/bonfire-networks/bonfire_data_edges/) (feat. bonus triggers) -If you want to know exactly what's happening, I can only suggest reading the code for -[Pointers.Migration](https://github.com/bonfire-networks/pointers/blob/main/lib/migration.ex), it's -surprisingly readable. +If you want to know exactly what's happening, you may want to read the code for +[Pointers.Migration](https://github.com/bonfire-networks/pointers/blob/main/lib/migration.ex). \ No newline at end of file