up

2024-05-18 00:52:40 +00:00 · 2021-05-22 20:41:36 +02:00 · 2021-05-22 20:41:36 +02:00 · ace19ed0e2
parent b77fb7e450
commit ace19ed0e2
6 changed files with 379 additions and 17 deletions
--- a/2
+++ b/2
@ -281,7 +281,7 @@ rel.run: init docker.stop.web ## Run the app in Docker & starts a new `iex` cons
 rel.run.bg: init docker.stop.web ## Run the app in Docker, and keep running in the background
 	@docker-compose -p $(APP_REL_CONTAINER) -f $(APP_REL_DOCKERCOMPOSE) up -d

-rel.stop: init ## Run the app in Docker, and keep running in the background
+rel.stop: init ## Stop the running release
 	@docker-compose -p $(APP_REL_CONTAINER) -f $(APP_REL_DOCKERCOMPOSE) stop

 rel.shell: init docker.stop.web ## Runs a simple shell inside of the container, useful to explore the image
--- a/docs/ARCHITECTURE.md
+++ b/docs/ARCHITECTURE.md
@ -0,0 +1,167 @@
+# Bonfire Architecture
+
+## Hacking
+
+This is an unusual piece of software, developed in an unusual
+way. It started with requests by Moodle users to be able to share and
+collaborate on educational resources with their peers.
+
+Hacking on it is actually pretty fun. The codebase has a unique
+feeling to work with and we've relentlessly refactored to manage the
+ever-growing complexity that a distributed social network
+implies. This said, it is not easy to understand without context,
+which is what this section is here to provide.
+
+## Design Decisions
+
+Feature goals:
+
+- Flexibility for developers and deployments.
+- Integrated federation with the existing fediverse.
+
+Operational goals:
+
+- Easy to set up and run.
+- Light on resources for small deployments.
+- Scalable for large deployments.
+
+Operationally, there's a tension between wanting to be able to scale
+instances and not wanting to burden small instances with
+high resource requirements or difficult setup.
+
+There are no easy answers to this. Our current solution is heavily
+reliant on postgresql. We will monitor perforamnce and scaling and
+continually evolve our strategy.
+
+## Stack
+
+Our implementation language is [Elixir](https://www.elixir-lang.org/),
+a language designed for building reliable systems. We use the
+[Phoenix](https://www.phoenixframework.org/) web framework.
+
+Some extensions use LiveView and Surface for UI components and views.
+
+Some extensions use the [Absinthe](https://absinthe-graphql.org/) GraphQL library to deliver a GraphQL API.
+
+We like our stack and we have no interest in rewriting in PHP, thanks
+for not asking.
+
+## Code Structure
+
+The server code is broadly composed of these parts.
+
+- `Bonfire.*` - Core application logic (very little code).
+- `Bonfire.*.*` - Bonfire extensions (eg `Bonfire.Social.Posts`)
+- `Bonfire.Data.*` - Schemas etc.
+- `Bonfire.Web.*` - Phoenix webapp
+- `Bonfire.GraphQL.*` - Optional GraphQL API.
+- `ActivityPub` - ActivityPub S2S models, logic and various helper modules (adapted Pleroma code)
+- `ActivityPubWeb` - ActivityPub S2S REST endpoints, activity ingestion and push federation facilities (adapted Pleroma code)
+- `ValueFlows.*` - economic extensions
+
+### `Bonfire`
+
+This namespace contains the core business logic. Every `Bonfire` object type has at least context module (e. g. `Bonfire.Communities`), a model/schema module (`Bonfire.Communities.Community`) and a queries module (`Bonfire.Communities.Queries`).
+
+All `Bonfire` objects use an ULID as their primary key. We use the pointers library (`Bonfire.Common.Pointers`) to reference any object by its primary key without knowing what type it is beforehand. This is very useful as we allow for example following or liking many different types of objects and this approach allows us to store the context of the like/follow by only storing its primary key (see `Bonfire.Follows.Follow`) for an example.
+
+All context modules have a `one/1` and `many/1` function for fetching objects. These take a keyword list as filters as arguments allowing objects to be fetched by arbitrary criteria defined in the queries modules.
+
+Examples:
+
+```
+Communities.one(username: "bob") # Fetching by username
+Collections.many(community: "01E9TQP93S8XFSV2ZATX1FQ528") # Fetching collections by its parent community
+Resources.many(deleted: nil) # Fetching all undeleted communities
+```
+
+Context modules also have functions for creating, updating and deleting objects. These actions are passed to the AP layer via the `Bonfire.Workers.APPublishWorker` module.
+
+#### Contexts
+
+The `Bonfire` namespace is occupied mostly by contexts. These are
+top level modules which comprise a grouping of:
+
+- A top level library module
+- Additional library modules
+- OTP services
+- Ecto schemas
+
+Here are some of the current extensions, or modules therin:
+
+- `Bonfire.Boundaries` (for managing and querying email whitelists)
+- `Bonfire.Social.FeedActivities` and `Bonfire.Social.Activities` and`Bonfire.Social.Activities` (for managing and querying activities, the unit of a feed)
+- `Bonfire.Social.Feeds` (for managing and querying feeds)
+- `Bonfire.Social.Flags` (for managing and querying flags)
+- `Bonfire.Social.Follows` (for managing and querying follows)
+- `Bonfire.Instance` (for managing the local instance)
+- `Bonfire.Me.Mails` (for rendering and sending emails)
+- `Bonfire.Social.Threads` (for managing and querying threads and comments)
+- `Bonfire.Me.Users` (for managing and querying both local and remote users)
+- `Bonfire.Files` (for managing uploaded content)
+- `Bonfire.Me.Characters` (a shared abstraction over users, communities, collections, and other objects that need to have feeds and act as an actor in ActivityPub land)
+- `Bonfire.Search` (local search indexing and search API, powered by Meili)
+
+#### Additional extensions and modules
+
+- `Bonfire.Application` (OTP application)
+- `Bonfire.Federate.ActivityPub` (ActivityPub integration)
+- `Bonfire.Common` (stuff that gets used everywhere)
+- `Bonfire.GraphQL` (GraphQL abstractions)
+- `Bonfire.Queries` (Helpers for making queries)
+- `Bonfire.Repo` (Ecto repository)
+
+### `Bonfire.Web`
+
+Structure:
+
+- Endpoint
+- Router
+- Controllers
+- Views
+- Plugs
+- GraphQL
+  - Schemas
+  - Resolvers
+  - Middleware
+  - Pipeline
+  - Flows
+
+See the [GraphQL API documentation](./GRAPHQL.md)
+
+### `ActivityPub`
+
+This namespace handles the ActivityPub logic and stores AP activitities. It is largely adapted Pleroma code with some modifications, for example merging of the activity and object tables and new actor object abstraction.
+
+It also contains some functionality that isn't part of the AP spec but is required for federation:
+
+- `ActivityPub.Keys` - Generating and handling RSA keys for messagage signing
+- `ActivityPub.Signature` - Adapter for the HTTPSignature library
+- `ActivityPub.WebFinger` - Implementation of the WebFinger protocol
+- `ActivityPub.HTTP` - Module for making HTTP requests (wrapper around tesla)
+- `ActivityPub.Instances` - Module for storing reachability information about remote instances
+
+`ActivityPub` contains the main API and is documented there. `ActivityPub.Adapter` defines callback functions for the AP library.
+
+### `ActivityPubWeb`
+
+This namespace contains the AP S2S REST API, the activity ingestion pipeline (`ActivityPubWeb.Transmogrifier`) and the push federation facilities (`ActivityPubWeb.Federator`, `ActivityPubWeb.Publisher` and others). The outgoing federation module is designed in a modular way allowing federating through different protocols in the future.
+
+### `ActivityPub` interaction in our application logic
+
+The callback functions defined in `ActivityPub.Adapter` are implemented in `Bonfire.ActivityPub.Adapter`. Facilities for calling the ActivityPub API are implemented in `Bonfire.ActivityPub.Publisher`. When implementing federation for a new object type it needs to be implemented both ways: both for outgoing federation in `Bonfire.ActivityPub.Publisher` and for incoming federation in `Bonfire.ActivityPub.Adapter`.
+
+## Naming
+
+It is said that naming is one of the four hard problems of computer
+science (along with cache management and off-by-one errors). We don't
+claim our scheme is the best, but we do strive for consistency.
+
+Naming rules:
+
+- Context names all begin `Bonfire.` and are named in plural where possible.
+- Everything within a context begins with the context name and a `.`
+- Ecto schemas should be named in the singular
+- Database tables should be named in the singular
+- Acronyms in module names should be all uppercase
+- OTP services should have the `Service` suffix (without a preceding `.`)
--- a/docs/DEPLOY.md
+++ b/docs/DEPLOY.md
@ -0,0 +1,181 @@
+# Backend Configuration and Deployment
+
+# WARNING: Bonfire is still under heavy development and is not ready to be deployed or used other than for development and testing purposes. 
+
+_These instructions are for setting up Bonfire in production. If you want to run the backend in development, please refer to our [Developer Guide](./HACKING.md)!_
+
+---
+
+## Step 0 - Configure your database
+
+You must provide a postgresql database for data storage. We require postgres 9.4 or above.
+
+If you are running in a restricted environment such as Amazon RDS, you will need to execute some sql against the database:
+
+```
+CREATE EXTENSION IF NOT EXISTS citext;
+```
+
+## Step 1 - Configure the backend
+
+The app needs some environment variables to be configured in order to work (a list of which can be found in the file `config/docker.env` in this same repository).
+
+In the `${FLAVOUR}/config/` directory, there are following default config files:
+
+- `config.exs`: default base configuration, which itself loads many other config files, such as one for each installed Bonfire extension.
+- `dev.exs`: default extra configuration for `MIX_ENV=dev`
+- `prod.exs`: default extra configuration for `MIX_ENV=prod`
+- `runtime.exs`: extra configuration which is loaded at runtime (vs the others which are only loaded once at compile time, i.e. when you build a release)
+
+You should NOT have to modify the files above. Instead, overload any settings from the above files using env variables (a list of which can be found in the file `${FLAVOUR}/config/prod/public.env` and `${FLAVOUR}/config/prod/secrets.env`  in this same repository).
+
+`MAIL_DOMAIN` and `MAIL_KEY` are needed to configure transactional email, you can for example sign up at [Mailgun](https://www.mailgun.com/) and then configure the domain name and key.
+
+---
+
+## Step 2 - Install
+
+---
+
+### Option A - Install using Docker containers (recommended)
+
+The easiest way to launch the docker image is using the make commands.
+The `docker-compose.release.yml` uses `config/prod/public.env` and `config/prod/secrets.env` to launch a container with the necessary environment variables along with its dependencies, currently that means an extra postgres container. You may want to add a webserver / reverse proxy yourself.
+
+#### Install with docker-compose
+
+1. Make sure you have [Docker](https://www.docker.com/), a recent [docker-compose](https://docs.docker.com/compose/install/#install-compose) (which supports v3 configs), and [make](https://www.gnu.org/software/make/) installed:
+
+```sh
+$ docker version
+Docker version 18.09.1-ce
+
+$ docker-compose -v
+docker-compose version 1.23.2
+
+$ make --version
+GNU Make 4.2.1
+...
+```
+
+2. Clone this repository and change into the directory:
+
+```sh
+$ git clone git@gitlab.com:Bonfire/Server.git bonfire-backend
+$ cd bonfire-backend
+```
+
+3. Build the docker image.
+
+```
+$ make rel-build
+
+$ make rel-tag-latest
+```
+
+4. Start the docker containers with docker-compose:
+
+```sh
+$ make rel-run
+```
+
+5. The backend should now be running at [http://localhost:4000/](http://localhost:4000/).
+
+6. If that worked, start the app as a daemon next time:
+
+```sh
+$ make rel-run-bg
+```
+
+#### Docker commands
+
+- `docker-compose run --rm backend bin/bonfire` returns all the possible commands
+- `docker-compose run --rm backend /bin/sh` runs a simple shell inside of the container, useful to explore the image
+- `docker-compose run --rm backend bin/bonfire start_iex` starts a new `iex` console
+- `docker-compose run backend bin/bonfire remote` runs an `iex` console when the service is already running.
+
+There are some useful database-related release tasks under `Bonfire.Repo.ReleaseTasks.` that can be run in an `iex` console:
+
+- `migrate` runs all up migrations
+- `rollback(step)` roll back to step X
+- `rollback_to(version)` roll back to a specific version
+- `rollback_all` rolls back all migrations back to zero (caution: this means loosing all data)
+
+For example:
+`iex> Bonfire.Repo.ReleaseTasks.migrate` to create your database if it doesn't already exist.
+
+#### Building a Docker image
+
+The Dockerfile uses the [multistage build](https://docs.docker.com/develop/develop-images/multistage-build/) feature to make the image as small as possible. It is a very common release using OTP releases. It generates the release which is later copied into the final image.
+
+There is a `Makefile` with two relavant commands:
+
+- `make rel-build` which builds the docker image in `bonfire/bonfire:latest` and `bonfire/bonfire:$VERSION-$BUILD`
+- `make rel-run` which can be used to run the docker built docker image instead of using `docker-compose`
+
+---
+
+### Option B - Manual installation without Docker
+
+#### Dependencies
+
+- Postgres version 9.6 or newer
+- Build tools
+- Elixir version 1.9.0 with OTP 22 (or possibly newer). If your distribution only has an old version available, check [Elixir's install page](https://elixir-lang.org/install.html) or use a tool like [asdf](https://github.com/asdf-vm/asdf) (run `asdf install` in this directory).
+
+#### Quickstart
+
+The quick way to get started with building a release, assuming that elixir and erlang are already installed.
+
+```bash
+$ export MIX_ENV=prod
+$ mix deps.get
+$ mix release
+# TODO: load required env variables
+$ _build/prod/rel/bonfire/bin/bonfire eval 'Bonfire.Repo.ReleaseTasks.migrate()'
+# DB migrated
+$ _build/prod/rel/bonfire/bin/bonfire start
+# App started in foreground
+```
+
+See the section on [Runtime Configuration](#runtime-configuration) for information on exporting environment variables.
+
+#### B-1. Building the release
+
+- Clone this repo.
+- Make sure you have erlang and elixir installed (check `Dockerfile` for what version we're currently using)
+- Run `mix deps.get` to install elixir dependencies.
+- From here on out, you may want to consider what your `MIX_ENV` is set to. For production, ensure that you either export `MIX_ENV=prod` or use it for each command. Continuing, we are assuming `MIX_ENV=prod`.
+- Run `mix release` to create an elixir release. This will create an executable in your `_build/prod/rel/bonfire` directory. We will be using the `bin/bonfire` executable from here on.
+
+#### B-2. Running the release
+
+- Export all required environment variables. See [Runtime Configuration](#runtime-configuration) section.
+
+- Create a database, if one is not created already with `bin/bonfire eval 'Bonfire.ReleaseTasks.create_db()'`.
+- You will likely also want to run the migrations. This is done similarly with `bin/bonfire eval 'Bonfire.Repo.ReleaseTasks.migrate()'`.
+- If you’re using RDS or some other locked down DB, you may need to run `CREATE EXTENSION IF NOT EXISTS citext WITH SCHEMA public;` on your database with elevated privileges.
+
+* You can check if your instance is configured correctly by running it with `bonfire start`
+
+* To run the instance as a daemon, use `bin/bonfire daemon`.
+
+#### B-3. Adding HTTPS
+
+The common and convenient way for adding HTTPS is by using Nginx or Caddyserver as a reverse proxy.
+
+Caddyserver handles generating and setting up HTTPS certificates automatically, but if you need TLS/SSL certificates for nginx, you can look get some for free with [letsencrypt](https://letsencrypt.org/). The simplest way to obtain and install a certificate is to use [Certbot.](https://certbot.eff.org). Depending on your specific setup, certbot may be able to get a certificate and configure your web server automatically.
+
+#### Runtime configuration
+
+You will need to load the required environment variables for the release to run properly.
+
+See [`config/releases.exs`](config/releases.exs) for all used variables. Consider also viewing there [`config/docker.env`](config/docker.env) file for some examples of values.
+
+---
+
+## Step 3 - Run
+
+By default, the backend listens on port 4000 (TCP), so you can access it on http://localhost:4000/ (if you are on the same machine). In case of an error it will restart automatically.
+
+Once you've signed up, you will automatically be an instance admin if you were the first to register.
--- a/flavours/classic/config/bonfire_boundaries.exs
+++ b/flavours/classic/config/bonfire_boundaries.exs
@ -7,7 +7,7 @@ config :bonfire,
  verbs: %{
    read:    "0EAD1NGSVTTER1YFVNDAMENTA1",
    see:     "0BSERV1NG11ST1NGSEX1STENCE",
-    create:  "CREATE0RP0STBRANDNEW0BJECT",
+    create:  "4REATE0RP0STBRANDNEW0BJECT",
    edit:    "CHANG1NGVA1VES0FPR0PERT1ES",
    delete:  "MAKESTVFFG0AWAYPERMANENT1Y",
    follow:  "T0SVBSCR1BET0THE0VTPVT0F1T",
--- a/flavours/classic/config/bonfire_data.exs
+++ b/flavours/classic/config/bonfire_data.exs
@ -58,7 +58,31 @@ config :bonfire_data_access_control,

 alias Pointers.{Pointer, Table}

-config :pointers, Pointer, []
+config :pointers, Pointer,
+  has_one:  [controlled:     {Controlled, foreign_key: :id}],
+  has_one:  [created: {Created, foreign_key: :id}],
+  has_one:  [activity: {Activity, foreign_key: :object_id, references: :id}], # needs ON clause
+  has_one:  [post_content: {PostContent, foreign_key: :id}],
+  has_one:  [like_count:   {LikeCount,   foreign_key: :id}],
+  has_many: [likes: {Like, foreign_key: :liked_id, references: :id}],
+  has_one:  [my_like: {Like, foreign_key: :liked_id, references: :id}],
+  has_one:  [my_flag: {Flag, foreign_key: :flagged_id, references: :id}],
+  has_one:  [replied: {Replied, foreign_key: :id}],
+  has_many: [direct_replies: {Replied, foreign_key: :reply_to_id}],
+  has_one:  [profile:        {Profile,       foreign_key: :id}],
+  has_one:  [character:      {Character,     foreign_key: :id}],
+  has_one:  [actor:          {Actor,         foreign_key: :id}],
+  # add references of tags to any tagged Pointer
+  many_to_many: [
+    tags: {
+      Bonfire.Tag,
+      join_through: "bonfire_tagged",
+      unique: true,
+      join_keys: [pointer_id: :id, tag_id: :id],
+      on_replace: :delete
+    }
+  ]
+
 config :pointers, Table, []

 # now let's weave everything else together for convenience
@ -303,17 +327,7 @@ config :bonfire_tag, Bonfire.Tag,
  # for locations
  has_one: [geolocation:  {Bonfire.Geolocate.Geolocation,      references: :id, foreign_key: :id}]

-# add references of tags to any tagged Pointer
-config :pointers, Pointers.Pointer,
-  many_to_many: [
-    tags: {
-      Bonfire.Tag,
-      join_through: "bonfire_tagged",
-      unique: true,
-      join_keys: [pointer_id: :id, tag_id: :id],
-      on_replace: :delete
-    }
-  ]
+

 # add references of tagged objects to any Category
 config :bonfire_classify, Bonfire.Classify.Category,
--- a/mix.lock
+++ b/mix.lock
@ -27,10 +27,10 @@
  "bonfire_search": {:git, "https://github.com/bonfire-networks/bonfire_search", "8ff913fb0248b591d364e21d81b83171fccf8a2d", [branch: "main"]},
  "bonfire_social": {:git, "https://github.com/bonfire-networks/bonfire_social", "8a6180b0b205315985036b54430a446f66508bdc", [branch: "main"]},
  "bonfire_tag": {:git, "https://github.com/bonfire-networks/bonfire_tag", "bd7d97ad2d6e41e196f4e2afb232fe07f24b4fd3", [branch: "main"]},
-  "bonfire_ui_coordination": {:git, "https://github.com/bonfire-networks/bonfire_ui_coordination", "bb928613c868c505eb6bdb69c66155aa7dd92e6a", [branch: "main"]},
+  "bonfire_ui_coordination": {:git, "https://github.com/bonfire-networks/bonfire_ui_coordination", "35597f74589102b8c45b786d3e3829c85c021ff8", [branch: "main"]},
  "bonfire_ui_reflow": {:git, "https://github.com/bonfire-networks/bonfire_ui_reflow", "6400d1858e1686ac9c2e21471df3876839ec18e0", [branch: "main"]},
-  "bonfire_ui_social": {:git, "https://github.com/bonfire-networks/bonfire_ui_social", "32085859421dcb49c790cc72a0678629ba210d51", [branch: "main"]},
-  "bonfire_ui_valueflows": {:git, "https://github.com/bonfire-networks/bonfire_ui_valueflows", "8a153fd601190f063f131158732d283b7c09eab1", [branch: "main"]},
+  "bonfire_ui_social": {:git, "https://github.com/bonfire-networks/bonfire_ui_social", "4d8d1d4d9b87c0bc0b89cd5c39b6ab5e0b443c47", [branch: "main"]},
+  "bonfire_ui_valueflows": {:git, "https://github.com/bonfire-networks/bonfire_ui_valueflows", "db41ee71f544299f5caf13cb4ec8afa8ef2e4741", [branch: "main"]},
  "bonfire_valueflows": {:git, "https://github.com/bonfire-networks/bonfire_valueflows", "1c5d84c497b6ec168ce9f75281fe5e051a96be72", [branch: "main"]},
  "bonfire_valueflows_observe": {:git, "https://github.com/bonfire-networks/bonfire_valueflows_observe", "f90358836370fb8821b7d4020ce6cc01702ecb07", []},
  "bonfire_website": {:git, "https://github.com/bonfire-networks/bonfire_website", "47dd383630a5ee829ffde227bb22ae46eab4832c", [branch: "main"]},