Hack: Allow for me specifically to migrate to GoToSocial

* add automatic cache max size generation based on ratios of a singular fixed memory target

Signed-off-by: kim <grufwub@gmail.com>

* remove now-unused cache max-size config variables

Signed-off-by: kim <grufwub@gmail.com>

* slight ratio tweak

Signed-off-by: kim <grufwub@gmail.com>

* remove unused visibility config var

Signed-off-by: kim <grufwub@gmail.com>

* add secret little ratio config trick

Signed-off-by: kim <grufwub@gmail.com>

* fixed a word

Signed-off-by: kim <grufwub@gmail.com>

* update cache library to remove use of TTL in result caches + slice cache

Signed-off-by: kim <grufwub@gmail.com>

* update other cache usages to use correct interface

Signed-off-by: kim <grufwub@gmail.com>

* update example config to explain the cache memory target

Signed-off-by: kim <grufwub@gmail.com>

* update env parsing test with new config values

Signed-off-by: kim <grufwub@gmail.com>

* do some ratio twiddling

Signed-off-by: kim <grufwub@gmail.com>

* add missing header

* update envparsing with latest defaults

Signed-off-by: kim <grufwub@gmail.com>

* update size calculations to take into account result cache, simple cache and extra map overheads

Signed-off-by: kim <grufwub@gmail.com>

* tweak the ratios some more

Signed-off-by: kim <grufwub@gmail.com>

* more nan rampaging

Signed-off-by: kim <grufwub@gmail.com>

* fix envparsing script

Signed-off-by: kim <grufwub@gmail.com>

* update cache library, add sweep function to keep caches trim

Signed-off-by: kim <grufwub@gmail.com>

* sweep caches once a minute

Signed-off-by: kim <grufwub@gmail.com>

* add a regular job to sweep caches and keep under 80% utilisation

Signed-off-by: kim <grufwub@gmail.com>

* remove dead code

Signed-off-by: kim <grufwub@gmail.com>

* add new size library used to libraries section of readme

Signed-off-by: kim <grufwub@gmail.com>

* add better explanations for the mem-ratio numbers

Signed-off-by: kim <grufwub@gmail.com>

* update go-cache

Signed-off-by: kim <grufwub@gmail.com>

* library version bump

Signed-off-by: kim <grufwub@gmail.com>

* update cache.result{} size model estimation

Signed-off-by: kim <grufwub@gmail.com>

---------

Signed-off-by: kim <grufwub@gmail.com>

.drone.yml

@@ -12,7 +12,7 @@ steps:
   # We use golangci-lint for linting.
   # See: https://golangci-lint.run/
   - name: lint
-    image: golangci/golangci-lint:v1.50.1
+    image: golangci/golangci-lint:v1.53.1
     volumes:
       - name: go-build-cache
         path: /root/.cache/go-build
@@ -28,7 +28,7 @@ steps:
           - pull_request
 
   - name: test
-    image: golang:1.19.3-alpine
+    image: golang:1.20.4-alpine
     volumes:
       - name: go-build-cache
         path: /root/.cache/go-build
@@ -44,7 +44,7 @@ steps:
           - pull_request
 
   - name: web-setup
-    image: node:14-alpine
+    image: node:18-alpine
     when:
       event:
         include:
@@ -57,7 +57,7 @@ steps:
       - yarn --frozen-lockfile --cache-folder /tmp/cache
 
   - name: web-lint
-    image: node:14-alpine
+    image: node:18-alpine
     when:
       event:
         include:
@@ -69,7 +69,7 @@ steps:
       - yarn run lint
 
   - name: web-build
-    image: node:14-alpine
+    image: node:18-alpine
     when:
       event:
         include:
@@ -81,7 +81,7 @@ steps:
       - yarn run build
 
   - name: snapshot
-    image: superseriousbusiness/gotosocial-drone-build:0.0.7 # https://github.com/superseriousbusiness/gotosocial-drone-build
+    image: superseriousbusiness/gotosocial-drone-build:0.2.0 # https://github.com/superseriousbusiness/gotosocial-drone-build
     volumes:
       - name: go-build-cache
         path: /root/.cache/go-build
@@ -110,7 +110,7 @@ steps:
           - main
 
   - name: release
-    image: superseriousbusiness/gotosocial-drone-build:0.0.7 # https://github.com/superseriousbusiness/gotosocial-drone-build
+    image: superseriousbusiness/gotosocial-drone-build:0.2.0 # https://github.com/superseriousbusiness/gotosocial-drone-build
     volumes:
       - name: go-build-cache
         path: /root/.cache/go-build
@@ -169,7 +169,7 @@ clone:
 
 steps:
   - name: mirror
-    image: superseriousbusiness/gotosocial-drone-build:0.0.7
+    image: superseriousbusiness/gotosocial-drone-build:0.2.0
     environment:
       ORIGIN_REPO: https://github.com/superseriousbusiness/gotosocial
       TARGET_REPO: https://codeberg.org/superseriousbusiness/gotosocial
@@ -182,6 +182,6 @@ steps:
 
 ---
 kind: signature
-hmac: 46dd4ce5b5dab76bc64c68f4730663ed0ada81471ffcbfa1bcf935d8e7f6a155
+hmac: 946c2ffd4e79de07a767ec06ebac0a8ca70a03ce5666aae093c9b0af455041d1
 
 ...

.gitattributes

@@ -0,0 +1 @@
+/vendor/ linguist-generated

.github/ISSUE_TEMPLATE/bug_frontend.yaml

@@ -0,0 +1,67 @@
+name: Frontend Bug Report
+description: Report an issue related to the web frontend
+title: "[bug] Issue Title"
+labels: ["bug", "frontend"]
+assignees: []
+
+body:
+  - type: markdown
+    attributes:
+      value: |
+        Thanks for taking the time to fill out this bug report! Please be cautious with the sensitive information/logs while filing the issue.
+
+  - type: textarea
+    id: desc
+    attributes:
+      label: Describe the bug with a clear and concise description of what the bug is. Please include screenshots of any visual issues.
+    validations:
+      required: true
+
+  - type: input
+    id: GoToSocial-Version
+    attributes:
+      label: What's your GoToSocial Version?
+      description: Enter the Version of your GoToSocial Installation
+      placeholder: e.g. v0.3.4
+    validations:
+      required: true
+
+  - type: input
+    id: Browser-Version
+    attributes:
+      label: Browser version
+      description: What browser(s) and versions are you using that show this bug?
+      placeholder: Firefox 103.0b9 (64-bit)
+    validations:
+      required: true
+
+  - type: textarea
+    id: what-happened
+    attributes:
+      label: What happened?
+      description: Enter exactly what happened.
+    validations:
+      required: false
+
+  - type: textarea
+    id: what-expected
+    attributes:
+      label: What you expected to happen?
+      description: Enter what you expected to happen.
+    validations:
+      required: false
+
+  - type: textarea
+    id: how-to-reproduce
+    attributes:
+      label: How to reproduce it?
+      description: As minimally and precisely as possible.
+    validations:
+      required: false
+
+  - type: textarea
+    id: anything-else
+    attributes:
+      label: Anything else we need to know?
+    validations:
+      required: false

.github/ISSUE_TEMPLATE/bug_report.yaml

@@ -2,13 +2,23 @@ name: Bug Report
 description: Create a report to help us improve
 title: "[bug] Issue Title"
 labels: [bug]
-assignees:
-  - 
+assignees: []
+
 body:
   - type: markdown
     attributes:
       value: |
         Thanks for taking the time to fill out this bug report! Please be cautious with the sensitive information/logs while filing the issue.
+  - type: markdown
+    attributes:
+      value: |
+        ⚠️ If you're reporting an interoperability issue with a **closed source**
+        ActivityPub/Mastodon client or server, please report the issue to the
+        developers of that client or server instead. Without access to their
+        source debugging the issue is difficult for us. Since GoToSocial is
+        open source, developers of closed source implementations can run a copy
+        and autonomously debug the issue. We'll gladly address any bugs they
+        raise with us.
   - type: textarea
     id: desc
     attributes:
@@ -34,15 +44,6 @@ body:
     validations:
       required: false
 
-  - type: input
-    id: Browser-Version
-    attributes:
-      label: Browser version
-      description: If this is a frontend bug, what browser(s) and versions are you using that show this bug?
-      placeholder: Firefox 103.0b9 (64-bit)
-    validations:
-      required: false
-
   - type: textarea
     id: what-happened
     attributes:

.github/pull_request_template.md

@@ -0,0 +1,26 @@
+# Description
+
+> If this is a code change, please include a summary of what you've coded, and link to the issue(s) it closes/implements.
+>
+> If this is a documentation change, please briefly describe what you've changed and why.
+
+This pull request implements xyz or fixes abc.
+
+closes #(issue)
+closes #(another issue)
+
+## Checklist
+
+Please put an x inside each checkbox to indicate that you've read and followed it: `[ ]` -> `[x]`
+
+If this is a documentation change, only the first checkbox must be filled (you can delete the others if you want).
+
+- [ ] I/we have read the [GoToSocial contribution guidelines](https://github.com/superseriousbusiness/gotosocial/blob/main/CONTRIBUTING.md).
+- [ ] I/we have discussed the proposed changes already, either in an issue on the repository, or in the Matrix chat.
+- [ ] I/we have performed a self-review of added code.
+- [ ] I/we have written code that is legible and maintainable by others.
+- [ ] I/we have commented the added code, particularly in hard-to-understand areas.
+- [ ] I/we have made any necessary changes to documentation.
+- [ ] I/we have added tests that cover new code.
+- [ ] I/we have run tests and they pass locally with the changes.
+- [ ] I/we have run `go fmt ./...` and `golangci-lint run`.

.gitignore

@@ -36,3 +36,6 @@ shell.nix
 # ignore config dirs from IDEs
 /.idea/
 /.fleet/
+
+# ignore cache dir from mkdocs serve
+/.cache

.golangci.yml

@@ -14,18 +14,74 @@ run:
 linters:
   # enable some extra linters, see here for the list: https://golangci-lint.run/usage/linters/
   enable:
-    - contextcheck
     - forcetypeassert
     - goconst
     - gocritic
     - gofmt
+    - goheader
     - gosec
-    - ifshort
     - nilerr
     - revive
-    - wastedassign
 
+# https://golangci-lint.run/usage/linters/#linters-configuration
 linters-settings:
+  # https://golangci-lint.run/usage/linters/#goheader
+  goheader:
+    template: |-
+      GoToSocial
+      Copyright (C) GoToSocial Authors admin@gotosocial.org
+      SPDX-License-Identifier: AGPL-3.0-or-later
+      
+      This program is free software: you can redistribute it and/or modify
+      it under the terms of the GNU Affero General Public License as published by
+      the Free Software Foundation, either version 3 of the License, or
+      (at your option) any later version.
+
+      This program is distributed in the hope that it will be useful,
+      but WITHOUT ANY WARRANTY; without even the implied warranty of
+      MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+      GNU Affero General Public License for more details.
+
+      You should have received a copy of the GNU Affero General Public License
+      along with this program.  If not, see <http://www.gnu.org/licenses/>.
+  # https://golangci-lint.run/usage/linters/#govet
   govet:
     disable:
-      - composites
+      - composites
+  # https://golangci-lint.run/usage/linters/#revive
+  revive:
+    rules:
+      # Enable most default rules.
+      # See: https://github.com/mgechev/revive/blob/master/defaults.toml
+      - name: blank-imports
+      - name: context-as-argument
+      - name: context-keys-type
+      - name: dot-imports
+      - name: error-naming
+      - name: error-return
+      - name: error-strings
+      - name: exported
+      - name: if-return
+      - name: increment-decrement
+      - name: var-naming
+      - name: var-declaration
+      - name: package-comments
+      - name: range
+      - name: receiver-naming
+      - name: time-naming
+      - name: unexported-return
+      - name: indent-error-flow
+      - name: errorf
+      - name: empty-block
+      - name: superfluous-else
+      - name: unreachable-code
+      # Disable below rules.
+      - name: redefines-builtin-id 
+        disabled: true # This one is just annoying.
+      - name: unused-parameter     
+        disabled: true # We often pass parameters to fulfil interfaces.
+  # https://golangci-lint.run/usage/linters/#staticcheck
+  staticcheck:
+    # Enable all checks, but disable SA1012: nil context passing.
+    # See: https://staticcheck.io/docs/configuration/options/#checks
+    checks: ["all", "-SA1012"]

.goreleaser.yml

@@ -24,6 +24,8 @@ builds:
       - netgo
       - osusergo
       - static_build
+      - kvformat
+      - timetzdata
     env:
       - CGO_ENABLED=0
     goos:

.readthedocs.yaml

@@ -5,9 +5,13 @@
 # Required
 version: 2
 
-mkdocs:
-  configuration: mkdocs.yml
+build:
+  os: "ubuntu-20.04"
+  tools:
+    python: "mambaforge-4.10" # https://docs.readthedocs.io/en/stable/guides/conda.html#making-builds-faster-with-mamba
 
-python:
-   install:
-   - requirements: docs/requirements.txt
+mkdocs:
+  configuration: "mkdocs.yml"
+
+conda:
+  environment: "docs/environment.yml"

.vscode/launch.json

@@ -2,14 +2,18 @@
 	"version": "0.2.0",
 	"configurations": [
 		{
-			"name": "Launch Package",
+			"name": "Debug Package",
 			"type": "go",
 			"request": "launch",
-			"mode": "auto",
+			"mode": "debug",
 			"program": "${workspaceFolder}/cmd/gotosocial",
 			"args": [
 				"testrig", "start"
 			],
+			"buildFlags": "-tags='netgo osusergo static_build kvformat debugenv'",
+			"env": {
+				"DEBUG": "1",
+			},
 			"cwd": "${workspaceFolder}"
 		}
 	]

.vscode/settings.json

@@ -2,5 +2,11 @@
 	"go.lintTool":"golangci-lint",
 	"go.lintFlags": [
 	  "--fast"
-	]
+	],
+	"go.vetFlags": [
+		"-composites=false ."
+	],
+	"eslint.workingDirectories": ["web/source"],
+	"eslint.lintTask.enable": true,
+	"eslint.lintTask.options": "${workspaceFolder}/web/source"
 }

CODE_OF_CONDUCT.md

@@ -0,0 +1,9 @@
+# Code of Conduct
+
+This document is currently a stub.
+
+In lieu of a fuller code of conduct, here are a few ground rules.
+
+1. We *DO NOT ACCEPT* PRs from right-wingers, Nazis, transphobes, homophobes, racists, harassers, abusers, white-supremacists, misogynists, tech-bros of questionable ethics. If that's you, politely fuck off somewhere else.
+2. Any PR that moves GoToSocial in the direction of surveillance capitalism or other bad fediverse behavior will be rejected.
+3. Don't spam the general chat too hard.

CONTRIBUTING.md

@@ -1,92 +1,116 @@
-# Contributing <!-- omit in toc -->
+# Contribution Guidelines <!-- omit in toc -->
 
 Hey! Welcome to the CONTRIBUTING.md for GoToSocial :) Thanks for taking a look, that kicks ass.
 
-This document will expand as the project expands, so for now, this is basically a stub.
+These contribution guidelines were adapted from / inspired by those of Gitea (https://github.com/go-gitea/gitea/blob/main/CONTRIBUTING.md). Thanks Gitea!
 
-Contributions are welcome at this point, since the API is fairly stable now and the structure is somewhat coherent.
+## Table of Contents  <!-- omit in toc -->
 
-Check the [issues](https://github.com/superseriousbusiness/gotosocial/issues) to see if there's anything you fancy jumping in on.
-
-## Table Of Contents  <!-- omit in toc -->
-
-- [Communications](#communications)
-- [Code of Conduct](#code-of-conduct)
-- [Setting up your development environment](#setting-up-your-development-environment)
-  - [Stylesheet / Web dev](#stylesheet--web-dev)
+- [Introduction](#introduction)
+- [Bug reports and feature requests](#bug-reports-and-feature-requests)
+- [Pull requests](#pull-requests)
+  - [Code](#code)
+  - [Documentation](#documentation)
+- [Development](#development)
   - [Golang forking quirks](#golang-forking-quirks)
-- [Setting up your test environment](#setting-up-your-test-environment)
-  - [Standalone Testrig with Pinafore](#standalone-testrig-with-pinafore)
-  - [Running automated tests](#running-automated-tests)
-    - [SQLite](#sqlite)
-    - [Postgres](#postgres)
-    - [Both](#both)
-  - [CLI Tests](#cli-tests)
-- [Project Structure](#project-structure)
-- [Style](#style)
-- [Linting and Formatting](#linting-and-formatting)
-- [Updating Swagger docs](#updating-swagger-docs)
-- [CI/CD configuration](#cicd-configuration)
-- [Release Checklist](#release-checklist)
-  - [What if something goes wrong?](#what-if-something-goes-wrong)
-- [Building Docker containers](#building-docker-containers)
-  - [With GoReleaser](#with-goreleaser)
-  - [Manually](#manually)
-- [Financial Compensation](#financial-compensation)
+  - [Building GoToSocial](#building-gotosocial)
+    - [Binary](#binary)
+    - [Docker](#docker)
+      - [With GoReleaser](#with-goreleaser)
+      - [Manually](#manually)
+  - [Stylesheet / Web dev](#stylesheet--web-dev)
+    - [Live Reloading](#live-reloading)
+  - [Project Structure](#project-structure)
+    - [Finding your way around the code](#finding-your-way-around-the-code)
+  - [Style / Linting / Formatting](#style--linting--formatting)
+  - [Testing](#testing)
+    - [Standalone Testrig with Semaphore](#standalone-testrig-with-semaphore)
+    - [Running automated tests](#running-automated-tests)
+      - [SQLite](#sqlite)
+      - [Postgres](#postgres)
+    - [CLI Tests](#cli-tests)
+    - [Federation](#federation)
+  - [Updating Swagger docs](#updating-swagger-docs)
+  - [CI/CD configuration](#cicd-configuration)
+  - [Release Checklist](#release-checklist)
+    - [What if something goes wrong?](#what-if-something-goes-wrong)
 
-## Communications
+## Introduction
 
-Before starting on something, please comment on an issue to say that you're working on it, and/or drop into the GoToSocial Matrix room [here](https://matrix.to/#/#gotosocial:superseriousbusiness.org).
+This document contains important information that will help you to write a successful contribution to GoToSocial. Please read it carefully before opening a pull request!
 
-This is the recommended way of keeping in touch with other developers, asking direct questions about code, and letting everyone know what you're up to.
+## Bug reports and feature requests
 
-## Code of Conduct
+Currently, we use Github's issue system for tracking bug reports and feature requests.
 
-In lieu of a fuller code of conduct, here are a few ground rules.
+You can view all open issues [here](https://github.com/superseriousbusiness/gotosocial/issues "The Github Issues page for GoToSocial").
 
-1. We *DO NOT ACCEPT* PRs from right-wingers, Nazis, transphobes, homophobes, racists, harassers, abusers, white-supremacists, misogynists, tech-bros of questionable ethics. If that's you, politely fuck off somewhere else.
-2. Any PR that moves GoToSocial in the direction of surveillance capitalism or other bad fediverse behavior will be rejected.
-3. Don't spam the general chat too hard.
+Before opening a new issue, whether bug or feature request, **please search carefully through both open and closed issues to make sure it hasn't been addressed already**. You can use Github's keyword issue search for this. If your issue is a duplicate of an existing issue, it will be closed.
 
-## Setting up your development environment
+Before you open a feature request issue, please consider the following:
 
-To get started, you first need to have Go installed. GtS is currently using Go 1.19, so you should take that too. See [here](https://golang.org/doc/install).
+- Does this feature fit within the scope of GoToSocial? Since we are a small team, we are wary of [feature creep](https://en.wikipedia.org/wiki/Feature_creep "Wikipedia article on Feature Creep"), which can cause maintenance issues.
+- Will this feature be generally useful for many users of the software, or is it handy for only a very specific use case?
+- Will this feature have a negative impact on the performance of the software? If so, is the tradeoff worth it?
+- Does this feature require loosening API security restrictions in some way? If so, it will need a good justification.
+- Does this feature belong in GoToSocial's server backend, or is it something that a client could implement instead?
 
-Once you've got go installed, clone this repository into your Go path. Normally, this should be `~/go/src/github.com/superseriousbusiness/gotosocial`.
+We tend to prioritize feature requests related to accessibility, fedi interoperability, and client compatibility.
 
-Once that's done, you can try building the project: `./scripts/build.sh`. This will build the `gotosocial` binary.
+## Pull requests
 
-If there are no errors, great, you're good to go!
+We welcome pull requests from new and existing contributors, with the following provisos:
 
-For automatic re-compiling during development, you can use [nodemon](https://www.npmjs.com/package/nodemon):
+- You have read and agree to our Code of Conduct.
+- The pull request addresses an existing issue or bug (please link to the relevant issue in your pull request), or is related to documentation.
+- If your pull request introduces significant new code paths, you are willing to do some maintenance work on those code paths, and address bugs. We do not appreciate drive-by pull requests that introduce a significant maintenance burden!
+- The pull request is of decent quality. We are a small team and unfortunately we don't have a lot of time to help shepherd pull requests, or help with basic coding questions. If you're unsure, don't bite off more than you can chew: start with a small feature or bugfix for your first PR, and work your way up.
 
-```bash
-nodemon -e go --signal SIGTERM --exec "go run ./cmd/gotosocial --host localhost testrig start || exit 1"
-```
+If you have small questions or comments before/during the pull request process, you can [join our Matrix space](https://matrix.to/#/#gotosocial-space:superseriousbusiness.org "GoToSocial Matrix space") at `#gotosocial-space:superseriousbusiness.org`.
 
-### Stylesheet / Web dev
+Please read the appropriate section below for the kind of pull request you plan to open.
 
-To work with the stylesheet for templates, you need [Node.js](https://nodejs.org/en/download/) and [Yarn](https://classic.yarnpkg.com/en/docs/install).
+### Code
 
-To install Yarn dependencies:
+To keep things manageable for maintainers, the process for opening pull requests against the GoToSocial repository works roughly as follows:
 
-```bash
-yarn install --cwd web/source
-```
+1. Open an issue for the feature, bug, or issue your pull request will address, or comment on an existing issue to let everyone know you want to work on it.
+2. Use the open issue to discuss your design with us, gather feedback, and resolve any concerns about the implementation.
+3. Write your code! Make sure all existing tests pass. Add tests where appropriate. Run linters and formatters. Update documentation.
+4. Open your pull request. You can do this as a draft, if you want to gather more feedback on code-in-progress.
+5. Let us know that your pull request is ready to be reviewed.
+6. Wait for review.
+7. Address review comments, make changes to the code where appropriate. It's OK to push back on review comments if you have a sensible reason--we're all learning, after all--but please do so with patience and grace.
+8. Get your code merged, rejoice!
 
-To recompile bundles:
+To make your code easier to review, try to split your pull request into sensibly-sized commits, but don't worry too much about making it totally perfect: we always squash merge anyways.
 
-```bash
-BUDO_BUILD=1 node web/source 
-```
+If your pull request ends up being massive, consider splitting it into smaller discrete pull requests to make it easier to review and reason about.
 
-Or you can run live reloading in development. It will start a webserver (ip/port printed in console, default localhost:8081), while also keeping the bundles
-up-to-date on disk. You can access the user/admin panels at localhost:8080/user, localhost:8080/admin, or run in tandem with the GoToSocial testrig, which will also
-serve the updated bundles from disk.
+Make sure your pull request only contains code that's relevant to the feature you're trying to implement, or the bug you're trying to address. Don't include refactors of unrelated parts of the code in your pull request: make a separate PR for that!
 
-``` bash
-NODE_ENV=development node web/source
-```
+If you open a code pull request without following the above process, we may close it and ask you to follow the process instead.
+
+### Documentation
+
+The process for documentation pull requests is a bit looser than the process for code.
+
+If you see something in the documentation that's missing, wrong, or unclear, feel free to open a pull request addressing it; you don't necessarily need to open an issue first, but please explain why you're opening the PR in the PR comment.
+
+We support a [Conda](https://docs.conda.io/en/latest/)-based workflow for hacking, building & publishing the documentation. Here's how you can get started locally:
+
+* Install [`miniconda`](https://docs.conda.io/en/latest/miniconda.html)
+* Create your conda environment: `conda env create -f ./docs/environment.yml`
+* Activate the environment: `conda activate gotosocial-docs`
+* Serve locally: `mkdocs serve`
+
+Then you can visit [localhost:8000](http://127.0.0.1:8000/) in your browser to view.
+
+When adding a new page, you need to include it in the [`mkdocs.yml`](mkdocs.yml) so it shows in the sidebar in the right section.
+
+If you don't use Conda, you can read the `docs/environment.yml` to see which dependencies are required and `pip install` them manually. It's advisable to do this in a virtual environment, which you can create with something like `python3 -m venv /path-to/store-the-venv`. You can then call `/path-to/store-the-venv/bin/pip`, `/path-to/store-the-venv/bin/mkdocs` etc.
+
+## Development
 
 ### Golang forking quirks
 
@@ -120,193 +144,29 @@ In case this post disappears, here are the steps (slightly modified):
 > `git remote add origin git@github.com/yourgithubname/gotosocial`
 >
 
-## Setting up your test environment
+### Building GoToSocial
 
-GoToSocial provides a [testrig](https://github.com/superseriousbusiness/gotosocial/tree/main/testrig) with a number of mock packages you can use in integration tests.
+#### Binary
 
-One thing that *isn't* mocked is the Database interface because it's just easier to use an in-memory SQLite database than to mock everything out.
+To get started, you first need to have Go installed. GtS is currently using Go 1.20, so you should take that too. See [here](https://golang.org/doc/install) for installation instructions.
 
-### Standalone Testrig with Pinafore
+Once you've got go installed, clone this repository into your Go path. Normally, this should be `~/go/src/github.com/superseriousbusiness/gotosocial`.
 
-You can launch a testrig as a standalone server running at localhost, which you can connect to using something like [Pinafore](https://github.com/nolanlawson/pinafore).
+Once you've installed the prerequisites, you can try building the project: `./scripts/build.sh`. This will build the `gotosocial` binary.
 
-To do this, first build the gotosocial binary with `./scripts/build.sh`.
+If there are no errors, great, you're good to go!
 
-Then, launch the testrig by invoking the binary as follows:
+For automatic re-compiling during development, you can use [nodemon](https://www.npmjs.com/package/nodemon):
 
 ```bash
-GTS_DB_TYPE="sqlite" GTS_DB_ADDRESS=":memory:" ./gotosocial --host localhost:8080 testrig start
+nodemon -e go --signal SIGTERM --exec "go run ./cmd/gotosocial --host localhost testrig start || exit 1"
 ```
 
-To run Pinafore locally in dev mode, first clone the [Pinafore](https://github.com/nolanlawson/pinafore) repository, and then run the following command in the cloned directory:
-
-```bash
-yarn run dev
-```
-
-The Pinafore instance will start running on `localhost:4002`.
-
-To connect to the testrig, navigate to `http://localhost:4002` and enter your instance name as `localhost:8080`.
-
-At the login screen, enter the email address `zork@example.org` and password `password`. You will get a confirmation prompt. Accept, and you are logged in as Zork.
-
-Note the following constraints:
-
-- Since the testrig uses an in-memory database, the database will be destroyed when the testrig is stopped.
-- If you stop the testrig and start it again, any tokens or applications you created during your tests will also be removed. As such, you need to log out and in again every time you stop/start the rig.
-- The testrig does not make any actual external HTTP calls, so federation will not work from a testrig.
-
-### Running automated tests
-
-There are a few different ways of running tests. Each requires the use of the `-p 1` flag, to indicate that they should not be run in parallel.
-
-#### SQLite
-
-If you would like to run tests as quickly as possible, using an SQLite in-memory database, use:
-
-```bash
-GTS_DB_TYPE="sqlite" GTS_DB_ADDRESS=":memory:" go test ./...
-```
-
-#### Postgres
-
-If you want to run tests against a Postgres database running on localhost, run:
-
-```bash
-GTS_DB_TYPE="postgres" GTS_DB_ADDRESS="localhost" go test -p 1 ./...
-```
-
-In the above command, it is assumed you are using the default Postgres password of `postgres`.
-
-#### Both
-
-Finally, to run tests against both database types one after the other, use:
-
-```bash
-./scripts/test.sh
-```
-
-### CLI Tests
-
-In [./test/envparsing.sh](./test/envparsing.sh) there's a test for making sure that CLI flags, config, and environment variables get parsed as expected.
-
-Although this test *is* part of the CI/CD testing process, you probably won't need to worry too much about running it yourself. That is, unless you're messing about with code inside the `main` package in `cmd/gotosocial`, or inside the `config` package in `internal/config`.
-
-## Project Structure
-
-For project structure, GoToSocial follows a standard and widely accepted project layout [defined here](https://github.com/golang-standards/project-layout). As the author writes:
-
-> This is a basic layout for Go application projects. It's not an official standard defined by the core Go dev team; however, it is a set of common historical and emerging project layout patterns in the Go ecosystem.
-
-Most of the crucial business logic of the application is found inside the various packages and subpackages of the `internal` directory.
-
-Where possible, we prefer more files and packages of shorter length that very clearly pertain to definable chunks of application logic, rather than fewer but longer files: if one `.go` file is pushing 1,000 lines of code, it's probably too long.
-
-## Style
-
-It is a good idea to read the short official [Effective Go](https://golang.org/doc/effective_go) page before submitting code: this document is the foundation of many a style guide, for good reason, and GoToSocial more or less follows its advice.
-
-Another useful style guide that we try to follow: [this one](https://github.com/bahlo/go-styleguide).
-
-In addition, here are some specific highlights from Uber's Go style guide which agree with what we try to do in GtS:
-
-- [Group Similar Declarations](https://github.com/uber-go/guide/blob/master/style.md#group-similar-declarations).
-- [Reduce Nesting](https://github.com/uber-go/guide/blob/master/style.md#reduce-nesting).
-- [Unnecessary Else](https://github.com/uber-go/guide/blob/master/style.md#unnecessary-else).
-- [Local Variable Declarations](https://github.com/uber-go/guide/blob/master/style.md#local-variable-declarations).
-- [Reduce Scope of Variables](https://github.com/uber-go/guide/blob/master/style.md#reduce-scope-of-variables).
-- [Initializing Structs](https://github.com/uber-go/guide/blob/master/style.md#initializing-structs).
-
-## Linting and Formatting
-
-Before you submit any code, make sure to run `go fmt ./...` to update whitespace and other opinionated formatting.
-
-We use [golangci-lint](https://golangci-lint.run/) for linting, which allows us to catch style inconsistencies and potential bugs or security issues, using static code analysis.
-
-If you make a PR that doesn't pass the linter, it will be rejected. As such, it's good practice to run the linter locally before pushing or opening a PR.
-
-To do this, first install the linter following the instructions [here](https://golangci-lint.run/usage/install/#local-installation).
-
-Then, you can run the linter with:
-
-```bash
-golangci-lint run
-```
-
-If there's no output, great! It passed :).
-
-## Updating Swagger docs
-
-GoToSocial uses [go-swagger](https://goswagger.io) to generate Swagger API documentation from code annotations.
-
-You can install go-swagger following the instructions [here](https://goswagger.io/install.html).
-
-If you change Swagger annotations on any of the API paths, you can generate a new Swagger file at `./docs/api/swagger.yaml` by running:
-
-```bash
-swagger generate spec --scan-models --exclude-deps -o docs/api/swagger.yaml
-```
-
-## CI/CD configuration
-
-GoToSocial uses [Drone](https://www.drone.io/) for CI/CD tasks like running tests, linting, and building Docker containers.
-
-These runs are integrated with GitHub, and will be run on opening a pull request or merging into main.
-
-The Drone instance for GoToSocial is [here](https://drone.superseriousbusiness.org/superseriousbusiness/gotosocial).
-
-The `drone.yml` file is [here](./.drone.yml) — this defines how and when Drone should run. Documentation for Drone is [here](https://docs.drone.io/).
-
-It is worth noting that the `drone.yml` file must be signed by the Drone admin account to be considered valid. This must be done every time the file is changed. This is to prevent tampering and hijacking of the Drone instance. See [here](https://docs.drone.io/signature/).
-
-To sign the file, first install and setup the [drone cli tool](https://docs.drone.io/cli/install/). Then, run:
-
-```bash
-drone -t PUT_YOUR_DRONE_ADMIN_TOKEN_HERE -s https://drone.superseriousbusiness.org sign superseriousbusiness/gotosocial --save
-```
-
-## Release Checklist
-
-First things first: If this is a security hot-fix, we'll probably rush through this list, and make a prettier release a few days later.
-
-Now, with that out of the way, here's our Checklist.
-
-GoToSocial follows [Semantic Versioning](https://semver.org/).
-So our first concern on the Checklist is:
-
-- What version are we releasing?
-
-Next we need to check:
-
-- Do the assets have to be rebuilt and committed to the repository.
-- Do the swagger docs have to be rebuilt?
-
-On the project management side:
-
-- Are there any issues that have to be moved to a different milestone?
-- Are there any things on the [Roadmap](./ROADMAP.md) that can be ticked off?
-
-Once we're happy with our Checklist, we can create the tag, and push it.
-And the rest [is automation](./.drone.yml).
-
-We can now head to GitHub, and add some personality to the release notes.
-Finally, we make announcements on the all our channels that the release is out!
-
-### What if something goes wrong?
-
-Sometimes things go awry.
-We release a buggy release, we forgot something ­ something important.
-
-If the release is so bad that it's unusable ­ or dangerous! ­ to a great part of our user-base, we can pull.
-That is: Delete the tag.
-
-Either way, once we've resolved the issue, we just start from the top of this list again. Version numbers are cheap. It's cheap to burn them.
-
-## Building Docker containers
+#### Docker
 
 For both of the below methods, you need to have [Docker buildx](https://docs.docker.com/buildx/working-with-buildx/) installed.
 
-### With GoReleaser
+##### With GoReleaser
 
 GoToSocial uses the release tooling [GoReleaser](https://goreleaser.com/intro/) to make multiple-architecture + Docker builds simple.
 
@@ -328,7 +188,7 @@ goreleaser --rm-dist --snapshot
 
 If all goes according to plan, you should now have a number of multiple-architecture binaries and tars inside the `./dist` folder, and snapshot Docker images should be built (check your terminal output for version).
 
-### Manually
+##### Manually
 
 If you prefer a simple approach to building a Docker container, with fewer dependencies (go-swagger, Node, Yarn), you can also just build in the following way:
 
@@ -358,8 +218,304 @@ See also: [exhaustive list of GOOS and GOARCH values](https://gist.github.com/li
 
 And: [exhaustive list of possible values for docker's `--platform`](https://github.com/tonistiigi/binfmt/#build-test-image)
 
-## Financial Compensation
+### Stylesheet / Web dev
 
-Right now, there's no structure in place for financial compensation for pull requests and code. This is simply because there's no money being made on the project, apart from the very small weekly Liberapay donations.
+GoToSocial uses Gin templates in the `web/template` folder. Static assets are stored in `web/assets`. Source files for stylesheets and JS bundles (for frontend enhancement, and the settings interface) are stored in `web/source`, and bundled from there to the `web/assets/dist` folder (gitignored).
 
-If money starts coming in, I'll start looking at proper financial structures, but for now, code is considered to be a donation in itself.
+To bundle changes, you need [Node.js](https://nodejs.org/en/download/) and [Yarn](https://classic.yarnpkg.com/en/docs/install).
+
+Using [NVM](https://github.com/nvm-sh/nvm) is one convenient way to install them which also supports managing different Node versions.
+
+To install Yarn dependencies:
+
+```bash
+yarn install --cwd web/source
+```
+
+To recompile bundles:
+
+```bash
+node web/source
+```
+
+#### Live Reloading
+
+For a more convenient development environment, you can run a livereloading version of the bundler alongside the [testrig](#testing).
+
+Open two terminals, first start the testrig on port 8081:
+``` bash
+GTS_PORT=8081 go run ./cmd/gotosocial testrig start
+```
+Then start the bundler, which will run on port 8080, and proxy requests to the testrig instance where needed.
+``` bash
+NODE_ENV=development node web/source
+```
+
+The livereloading bundler *will not* change the bundled assets in `dist/`, so once you are finished with changes and want to deploy it somewhere, you have to run `node web/source` to generate production-ready bundles.
+
+### Project Structure
+
+For project structure, GoToSocial follows a standard and widely accepted project layout [defined here](https://github.com/golang-standards/project-layout). As the author writes:
+
+> This is a basic layout for Go application projects. It's not an official standard defined by the core Go dev team; however, it is a set of common historical and emerging project layout patterns in the Go ecosystem.
+
+Where possible, we prefer more files and packages of shorter length that very clearly pertain to definable chunks of application logic, rather than fewer but longer files: if one `.go` file is pushing 1,000 lines of code, it's probably too long.
+
+#### Finding your way around the code
+
+Most of the crucial business logic of the application is found inside the various packages and subpackages of the `internal` directory. Here's a brief summary of each of these:
+
+`internal/ap` - ActivityPub utility functions and interfaces.
+
+`internal/api` - Models, routers, and utilities for the client and federated (ActivityPub) APIs. This is where routes are attached to the router, and where you want to be if you're adding a route.
+
+`internal/concurrency` - Worker models used by the processor and other queues.
+
+`internal/config` - Code for configuration flags, CLI flag parsing, and getting/setting config.
+
+`internal/db` - DB interfaces for interacting with sqlite/postgres databases. Database migration code is in `internal/db/bundb/migrations`.
+
+`internal/email` - Email functionality, email sending via SMTP.
+
+`internal/federation` - ActivityPub federation code; implements `go-fed` interfaces.
+
+`internal/federation/federatingdb` - Implementation of `go-fed`'s Database interface.
+
+`internal/federation/dereferencing` - Code for making http calls to fetch resources from remote instances.
+
+`internal/gotosocial` - GoToSocial server startup/shutdown logic.
+
+`internal/gtserror` - Error models.
+
+`internal/gtsmodel` - Database and internal models. This is where `bundb` annotations live.
+
+`internal/httpclient` - The HTTP client used by GoToSocial for making requests to remote resources.
+
+`internal/id` - Code for generating IDs (ULIDs) for database models.
+
+`internal/log` - Our logging implementation.
+
+`internal/media` - Code related to managing + processing media attachments; images, video, emojis, etc.
+
+`internal/messages` - Models for wrapping worker messages.
+
+`internal/middleware` - Gin Gonic router middlewares: http signature checking, cache control, token checks, etc.
+
+`internal/netutil` - HTTP / net request validation code.
+
+`internal/oauth` - Wrapper code/interfaces for OAuth server implementation.
+
+`internal/oidc` - Wrapper code/interfaces for OIDC claims and callbacks.
+
+`internal/processing` - Logic for processing messages produced by the federation or client APIs. Much of the core business logic of GoToSocial is contained here.
+
+`internal/regexes` - Regular expressions used for text parsing and matching of URLs, hashtags, mentions, etc.
+
+`internal/router` - Wrapper for Gin HTTP router. Core HTTP logic is contained here. The router exposes functions for attaching routes, which are used by the code in `internal/api` handlers.
+
+`internal/storage` - Wrapper for `codeberg.org/gruf/go-store` implementations. Local file storage and s3 logic goes here.
+
+`internal/stream` - Websockets streaming logic.
+
+`internal/text` - Text parsing and transformation. Status parsing logic is contained here -- both plain and markdown.
+
+`internal/timeline` - Status timeline management code.
+
+`internal/trans` - Code for exporting models to json backup files from the database, and importing backup json files into the database.
+
+`internal/transport` - HTTP transport code and utilities.
+
+`internal/typeutils` - Code for converting from internal database models to json, and back, or from ActivityPub format to internal database model format and vice versa. Basically, serdes.
+
+`internal/uris` - Utilities for generating URIs used throughout GoToSocial.
+
+`internal/util` - Odds and ends; small utility functions used by more than one package.
+
+`internal/validate` - Model validation code -- currently not really used.
+
+`internal/visibility` - Status visibility checking and filtering.
+
+`internal/web` - Web UI handlers, specifically for serving web pages, the landing page, settings panels.
+
+### Style / Linting / Formatting
+
+It is a good idea to read the short official [Effective Go](https://golang.org/doc/effective_go) page before submitting code: this document is the foundation of many a style guide, for good reason, and GoToSocial more or less follows its advice.
+
+Another useful style guide that we try to follow: [this one](https://github.com/bahlo/go-styleguide).
+
+In addition, here are some specific highlights from Uber's Go style guide which agree with what we try to do in GtS:
+
+- [Group Similar Declarations](https://github.com/uber-go/guide/blob/master/style.md#group-similar-declarations).
+- [Reduce Nesting](https://github.com/uber-go/guide/blob/master/style.md#reduce-nesting).
+- [Unnecessary Else](https://github.com/uber-go/guide/blob/master/style.md#unnecessary-else).
+- [Local Variable Declarations](https://github.com/uber-go/guide/blob/master/style.md#local-variable-declarations).
+- [Reduce Scope of Variables](https://github.com/uber-go/guide/blob/master/style.md#reduce-scope-of-variables).
+- [Initializing Structs](https://github.com/uber-go/guide/blob/master/style.md#initializing-structs).
+
+Before you submit any code, make sure to run `go fmt ./...` to update whitespace and other opinionated formatting.
+
+We use [golangci-lint](https://golangci-lint.run/) for linting, which allows us to catch style inconsistencies and potential bugs or security issues, using static code analysis.
+
+If you make a PR that doesn't pass the linter, it will be rejected. As such, it's good practice to run the linter locally before pushing or opening a PR.
+
+To do this, first install the linter following the instructions [here](https://golangci-lint.run/usage/install/#local-installation).
+
+Then, you can run the linter with:
+
+```bash
+golangci-lint run
+```
+
+If there's no output, great! It passed :)
+
+### Testing
+
+GoToSocial provides a [testrig](https://github.com/superseriousbusiness/gotosocial/tree/main/testrig) with a number of mock packages you can use in integration tests.
+
+One thing that *isn't* mocked is the Database interface because it's just easier to use an in-memory SQLite database than to mock everything out.
+
+#### Standalone Testrig with Semaphore
+
+You can launch a testrig as a standalone server running at localhost, which you can connect to using something like [Semaphore](https://github.com/NickColley/semaphore/).
+
+To do this, first build the gotosocial binary with `DEBUG=1 ./scripts/build.sh`.
+
+Then, launch the testrig with the `DEBUG` environment variable set by invoking the binary as follows:
+
+```bash
+DEBUG=1 ./gotosocial testrig start
+```
+
+To run Semaphore locally in dev mode, first clone the [Semaphore](https://github.com/NickColley/semaphore/) repository, and then run the following commands in the cloned directory:
+
+```bash
+yarn # install dependencies
+yarn run dev
+```
+
+The Semaphore instance will start running on `localhost:4002`.
+
+To connect to the testrig, navigate to `http://localhost:4002` and enter your instance name as `localhost:8080`.
+
+At the login screen, enter the email address `zork@example.org` and password `password`. You will get a confirmation prompt. Accept, and you are logged in as Zork.
+
+Note the following constraints:
+
+- Since the testrig uses an in-memory database, the database will be destroyed when the testrig is stopped.
+- If you stop the testrig and start it again, any tokens or applications you created during your tests will also be removed. As such, you need to log out and in again every time you stop/start the rig.
+- The testrig does not make any actual external HTTP calls, so federation will not work from a testrig.
+
+#### Running automated tests
+
+Tests can be run against both SQLite and Postgres.
+
+##### SQLite
+
+If you would like to run tests as quickly as possible, using an SQLite in-memory database, use:
+
+```bash
+go test ./...
+```
+
+##### Postgres
+
+If you want to run tests against a Postgres database on localhost, run:
+
+```bash
+GTS_DB_TYPE="postgres" GTS_DB_ADDRESS="localhost" go test -p 1 ./...
+```
+
+In the above command, it is assumed you are using the default Postgres password of `postgres`.
+
+We set `-p 1` when running against Postgres because it requires tests to run in serial, not in parallel.
+
+#### CLI Tests
+
+In [./test/envparsing.sh](./test/envparsing.sh) there's a test for making sure that CLI flags, config, and environment variables get parsed as expected.
+
+Although this test *is* part of the CI/CD testing process, you probably won't need to worry too much about running it yourself. That is, unless you're messing about with code inside the `main` package in `cmd/gotosocial`, or inside the `config` package in `internal/config`.
+
+#### Federation
+
+By using the support for loading TLS files from disk it is possible to have two or more local instances with TLS to allow for (manually) testing federation.
+
+You'll need to set the following configuration options:
+
+- `GTS_TLS_CERTIFICATE_CHAIN`: poiting to a PEM-encoded certificate chain including the public certificate.
+- `GTS_TLS_CERTIFICATE_KEY`: pointing to a PEM-encoded private key.
+
+Additionally, for the Go HTTP client to recognise certificates issued by a custom CA as valid, you'll need to set one of:
+
+- `SSL_CERT_FILE`: pointing to the public key of your custom CA.
+- `SSL_CERT_DIR`: a `:`-separated list of directories to load CA certificates from.
+
+The above `SSL_CERT` variables work on Unix-like systems only, excluding Mac. See https://pkg.go.dev/crypto/x509#SystemCertPool. If you are running your tests on an architecture that doesn't support setting the above variables, you can instead disable TLS certificate verification for the HTTP client entirely by setting `http-client.tls-insecure-skip-verify` to `true` in the config.yaml file.
+
+You'll additionally need functioning DNS for your two instance names, which you can achieve through entries in `/etc/hosts` or by running a local DNS server like [dnsmasq](https://thekelleys.org.uk/dnsmasq/doc.html).
+
+### Updating Swagger docs
+
+GoToSocial uses [go-swagger](https://goswagger.io) to generate Swagger API documentation from code annotations.
+
+You can install go-swagger following the instructions [here](https://goswagger.io/install.html).
+
+If you change Swagger annotations on any of the API paths, you can generate a new Swagger file at `./docs/api/swagger.yaml` by running:
+
+```bash
+swagger generate spec --scan-models --exclude-deps -o docs/api/swagger.yaml
+```
+
+### CI/CD configuration
+
+GoToSocial uses [Drone](https://www.drone.io/) for CI/CD tasks like running tests, linting, and building Docker containers.
+
+These runs are integrated with GitHub, and will be run on opening a pull request or merging into main.
+
+The Drone instance for GoToSocial is [here](https://drone.superseriousbusiness.org/superseriousbusiness/gotosocial).
+
+The `drone.yml` file is [here](./.drone.yml) — this defines how and when Drone should run. Documentation for Drone is [here](https://docs.drone.io/).
+
+It is worth noting that the `drone.yml` file must be signed by the Drone admin account to be considered valid. This must be done every time the file is changed. This is to prevent tampering and hijacking of the Drone instance. See [here](https://docs.drone.io/signature/).
+
+To sign the file, first install and setup the [drone cli tool](https://docs.drone.io/cli/install/). Then, run:
+
+```bash
+drone -t PUT_YOUR_DRONE_ADMIN_TOKEN_HERE -s https://drone.superseriousbusiness.org sign superseriousbusiness/gotosocial --save
+```
+
+### Release Checklist
+
+First things first: If this is a security hot-fix, we'll probably rush through this list, and make a prettier release a few days later.
+
+Now, with that out of the way, here's our Checklist.
+
+GoToSocial follows [Semantic Versioning](https://semver.org/).
+So our first concern on the Checklist is:
+
+- What version are we releasing?
+
+Next we need to check:
+
+- Do the assets have to be rebuilt and committed to the repository.
+- Do the swagger docs have to be rebuilt?
+
+On the project management side:
+
+- Are there any issues that have to be moved to a different milestone?
+- Are there any things on the [Roadmap](./ROADMAP.md) that can be ticked off?
+
+Once we're happy with our Checklist, we can create the tag, and push it.
+And the rest [is automation](./.drone.yml).
+
+We can now head to GitHub, and add some personality to the release notes.
+Finally, we make announcements on the all our channels that the release is out!
+
+#### What if something goes wrong?
+
+Sometimes things go awry.
+We release a buggy release, we forgot something ­ something important.
+
+If the release is so bad that it's unusable ­ or dangerous! ­ to a great part of our user-base, we can pull.
+That is: Delete the tag.
+
+Either way, once we've resolved the issue, we just start from the top of this list again. Version numbers are cheap. It's cheap to burn them.

Dockerfile

@@ -2,7 +2,7 @@
 # Dockerfile reference: https://docs.docker.com/engine/reference/builder/
 
 # stage 1: generate up-to-date swagger.yaml to put in the final container
-FROM --platform=${BUILDPLATFORM} quay.io/goswagger/swagger:v0.30.0 AS swagger
+FROM --platform=${BUILDPLATFORM} quay.io/goswagger/swagger:v0.30.4 AS swagger
 
 COPY go.mod /go/src/github.com/superseriousbusiness/gotosocial/go.mod
 COPY go.sum /go/src/github.com/superseriousbusiness/gotosocial/go.sum
@@ -12,7 +12,7 @@ WORKDIR /go/src/github.com/superseriousbusiness/gotosocial
 RUN swagger generate spec -o /go/src/github.com/superseriousbusiness/gotosocial/swagger.yaml --scan-models
 
 # stage 2: generate the web/assets/dist bundles
-FROM --platform=${BUILDPLATFORM} node:16.15.1-alpine3.15 AS bundler
+FROM --platform=${BUILDPLATFORM} node:16.19.1-alpine3.17 AS bundler
 
 COPY web web
 RUN yarn install --cwd web/source && \
@@ -20,7 +20,7 @@ RUN yarn install --cwd web/source && \
     rm -r web/source
 
 # stage 3: build the executor container
-FROM --platform=${TARGETPLATFORM} alpine:3.15.4 as executor
+FROM --platform=${TARGETPLATFORM} alpine:3.17.2 as executor
 
 # switch to non-root user:group for GtS
 USER 1000:1000

README.md

@@ -8,9 +8,9 @@ With GoToSocial, you can keep in touch with your friends, post, read, and share
   <img src="./docs/assets/sloth.png" width="300"/>
 </p>
 
-**GoToSocial is still [ALPHA SOFTWARE](https://en.wikipedia.org/wiki/Software_release_life_cycle#Alpha)**. It is already deployable and useable, and it federates cleanly with many other Fediverse servers (not yet all). However, many things are not yet implemented, and there are plenty of bugs! We foresee entering beta somewhere in 2023.
+**GoToSocial is still [ALPHA SOFTWARE](https://en.wikipedia.org/wiki/Software_release_life_cycle#Alpha)**. It is already deployable and useable, and it federates cleanly with many other Fediverse servers (not yet all). However, many things are not yet implemented, and there are plenty of bugs! We foresee entering beta around the beginning of 2024.
 
-Documentation is at [docs.gotosocial.org](https://docs.gotosocial.org). You can skip straight to the API documentation [here](https://docs.gotosocial.org/en/latest/api/swagger/).
+Documentation is at [docs.gotosocial.org](https://docs.gotosocial.org). You can skip straight to the API documentation [here](https://docs.gotosocial.org/en/latest/api/swagger/). To build from source, check the [CONTRIBUTING.md](./CONTRIBUTING.md) file.
 
 Here's a screenshot of the instance landing page!
 
@@ -33,20 +33,22 @@ Here's a screenshot of the instance landing page!
 - [Wishlist](#wishlist)
 - [Getting Started](#getting-started)
 - [Third-Party Packaging](#third-party-packaging)
+  - [Distribution packaging](#distribution-packaging)
+  - [Self-hosting](#self-hosting)
 - [Known Issues](#known-issues)
   - [Client App Issues](#client-app-issues)
   - [Federation Issues](#federation-issues)
 - [Contributing](#contributing)
+- [Building](#building)
 - [Contact](#contact)
 - [Credits](#credits)
   - [Libraries](#libraries)
-  - [Image Attribution](#image-attribution)
-  - [Developers](#developers)
+  - [Image Attribution and Licensing](#image-attribution-and-licensing)
+  - [Team](#team)
   - [Special Thanks](#special-thanks)
 - [Sponsorship + Funding](#sponsorship--funding)
-  - [OpenCollective](#opencollective)
-  - [LiberaPay](#liberapay)
-  - [Provisos](#provisos)
+  - [Crowdfunding](#crowdfunding)
+  - [NLnet](#nlnet)
 - [License](#license)
 
 ## What is GoToSocial?
@@ -89,13 +91,13 @@ For a detailed view on what's implemented and what's not, and progress made towa
 
 The Mastodon API has become the de facto standard for client communication with federated servers, so GoToSocial has implemented and extended the API with custom functionality.
 
-In short, this means full support for modern, beautiful apps like [Tusky](https://tusky.app/) and [Pinafore](https://pinafore.social/).
+In short, this means full support for modern, beautiful apps like [Tusky](https://tusky.app/) and [Semaphore](https://semaphore.social/).
 
-Tusky                                                        |  Pinafore
+Tusky                                                        |  Semaphore
 :-----------------------------------------------------------:|:------------------------------------------------------------------:
-![An image of GoToSocial in Tusky](./docs/assets/tusky.png)  | ![An image of GoToSocial in Pinafore](./docs/assets/pinafore.png)
+![An image of GoToSocial in Tusky](./docs/assets/tusky.png)  | ![An image of GoToSocial in Semaphore](./docs/assets/semaphore.png)
 
-If you're used to using Mastodon with Tusky or Pinafore, you'll find using GoToSocial a breeze.
+If you're used to using Mastodon with Tusky or Semaphore, you'll find using GoToSocial a breeze.
 
 ### Granular post settings
 
@@ -135,8 +137,8 @@ GoToSocial plays nice with lower-powered machines like Raspberry Pi, old laptops
 GoToSocial doesn't apply a one-size-fits-all approach to federation. Who your server federates with should be up to you.
 
 - 'Normal' federation; discover new servers.
-- *Allow list*-only federation; choose which servers you talk to.
-- Zero federation; keep your server private.
+- *Allow list*-only federation; choose which servers you talk to (not yet implemented).
+- Zero federation; keep your server private (not yet implemented).
 
 ### OIDC integration
 
@@ -169,21 +171,30 @@ All docs for installation and configuration are hosted at [docs.gotosocial.org](
 
 ## Third-Party Packaging
 
-Thank you so much to the cool people who have put time and energy into packaging GoToSocial! Known third-party packaging projects are listed below:
+Thank you so much to the cool people who have put time and energy into packaging GoToSocial! 
 
-- [YunoHost GoToSocial Packaging](https://github.com/YunoHost-Apps/gotosocial_ynh) by [OniriCorpe](https://github.com/OniriCorpe).
-- GoToSocial Helm Charts:
-  - [GoToSocial Helm Chart](https://github.com/Maxxblow/charts/tree/main/charts/gotosocial) by [0hlov3](https://github.com/0hlov3).
+### Distribution packaging
 
 These packages are not maintained by GoToSocial, so please direct questions and issues to the repository maintainers (and donate to them!).
 
+[![Packaging status](https://repology.org/badge/vertical-allrepos/gotosocial.svg)](https://repology.org/project/gotosocial/versions)
+
+### Self-hosting
+
+You can deploy your own instance of GoToSocial with the help of:
+
+- [YunoHost GoToSocial Packaging](https://github.com/YunoHost-Apps/gotosocial_ynh) by [OniriCorpe](https://github.com/OniriCorpe).
+- [Ansible Playbook (MASH)](https://github.com/mother-of-all-self-hosting/mash-playbook): The playbook supports a many services, including GoToSocial. [Documentation](https://github.com/mother-of-all-self-hosting/mash-playbook/blob/main/docs/services/gotosocial.md)
+- GoToSocial Helm Charts:
+  - [GoToSocial Helm Chart](https://github.com/fSocietySocial/charts/tree/main/charts/gotosocial) by [0hlov3](https://github.com/0hlov3).
+
 ## Known Issues
 
 Since GoToSocial is still in alpha, there are plenty of bugs. We use [GitHub issues](https://github.com/superseriousbusiness/gotosocial/issues?q=is%3Aissue+is%3Aopen+label%3Abug) to track these. The [FAQ](docs/faq.md) also describes some of the features that haven't been implemented yet.
 
 ### Client App Issues
 
-GoToSocial works great with Tusky and Pinafore, but some other client applications still need work or have issues connecting to GoToSocial. We're tracking them [right here](https://github.com/superseriousbusiness/gotosocial/projects/5). It's our goal to make any app that's compatible with the Mastodon API work seamlessly with GoToSocial.
+GoToSocial works great with Tusky and Semaphore, but some other client applications still need work or have issues connecting to GoToSocial. We're tracking them [right here](https://github.com/superseriousbusiness/gotosocial/projects/5). It's our goal to make any app that's compatible with the Mastodon API work seamlessly with GoToSocial.
 
 ### Federation Issues
 
@@ -193,6 +204,10 @@ Since every ActivityPub server implementation has a slightly different interpret
 
 You would like to contribute to GtS? Great! ❤️❤️❤️ Check out the issues page to see if there's anything you intend to jump in on, and read the [CONTRIBUTING.md](./CONTRIBUTING.md) file for guidelines and setting up your dev environment.
 
+## Building
+
+Instructions for building GoToSocial from source are in the [CONTRIBUTING.md](./CONTRIBUTING.md) file.
+
 ## Contact
 
 For questions and comments, you can [join our Matrix space](https://matrix.to/#/#gotosocial-space:superseriousbusiness.org) at `#gotosocial-space:superseriousbusiness.org`. This is the quickest way to reach the devs. You can also mail [admin@gotosocial.org](mailto:admin@gotosocial.org).
@@ -203,67 +218,93 @@ For bugs and feature requests, please check to see if there's [already an issue]
 
 ### Libraries
 
-The following libraries and frameworks are used by GoToSocial, with gratitude 💕
+The following open source libraries, frameworks, and tools are used by GoToSocial, with gratitude 💕
 
+- [abema/go-mp4](https://github.com/abema/go-mp4); mp4 parsing. [MIT License](https://spdx.org/licenses/MIT.html).
 - [buckket/go-blurhash](https://github.com/buckket/go-blurhash); used for generating image blurhashes. [GPL-3.0 License](https://spdx.org/licenses/GPL-3.0-only.html).
 - [coreos/go-oidc](https://github.com/coreos/go-oidc); OIDC client library. [Apache-2.0 License](https://spdx.org/licenses/Apache-2.0.html).
 - [disintegration/imaging](https://github.com/disintegration/imaging); image resizing. [MIT License](https://spdx.org/licenses/MIT.html).
-- [gin-gonic/gin](https://github.com/gin-gonic/gin); speedy router engine. [MIT License](https://spdx.org/licenses/MIT.html).
+- [DmitriyVTitov/size](https://github.com/DmitriyVTitov/size); runtime model memory size calculations. [MIT License](https://spdx.org/licenses/MIT.html).
+- Gin:
   - [gin-contrib/cors](https://github.com/gin-contrib/cors); Gin CORS middleware. [MIT License](https://spdx.org/licenses/MIT.html).
   - [gin-contrib/gzip](https://github.com/gin-contrib/gzip); Gin gzip middleware. [MIT License](https://spdx.org/licenses/MIT.html).
   - [gin-contrib/sessions](https://github.com/gin-contrib/sessions); Gin sessions middleware. [MIT License](https://spdx.org/licenses/MIT.html).
-  - [gin-contrib/static](https://github.com/gin-contrib/static); Gin static page middleware. [MIT License](https://spdx.org/licenses/MIT.html).
+  - [gin-gonic/gin](https://github.com/gin-gonic/gin); speedy router engine. [MIT License](https://spdx.org/licenses/MIT.html).
 - [go-fed/httpsig](https://github.com/go-fed/httpsig); secure HTTP signature library. [BSD-3-Clause License](https://spdx.org/licenses/BSD-3-Clause.html).
 - [google/uuid](https://github.com/google/uuid); UUID generation. [BSD-3-Clause License](https://spdx.org/licenses/BSD-3-Clause.html).
 - [google/wuffs](https://github.com/google/wuffs); png-stripping code. [Apache-2.0 License](https://spdx.org/licenses/Apache-2.0.html).
-- [go-playground/validator](https://github.com/go-playground/validator); struct validation. [MIT License](https://spdx.org/licenses/MIT.html).
-- [gorilla/feeds](https://github.com/gorilla/feeds); RSS + Atom feed generation. [BSD-2-Clause License](https://spdx.org/licenses/BSD-2-Clause.html).
-- [gorilla/websocket](https://github.com/gorilla/websocket); Websocket connectivity. [BSD-2-Clause License](https://spdx.org/licenses/BSD-2-Clause.html).
-- [gruf/go-debug](https://codeberg.org/gruf/go-debug); profiling support in debug builds. [MIT License](https://spdx.org/licenses/MIT.html).
-- [gruf/go-bytesize](https://codeberg.org/gruf/go-bytesize); byte size parsing / formatting. [MIT License](https://spdx.org/licenses/MIT.html).
-- [gruf/go-cache](https://codeberg.org/gruf/go-cache); object caching. [MIT License](https://spdx.org/licenses/MIT.html).
-- [gruf/go-kv](https://codeberg.org/gruf/go-kv); key-value field formatting. [MIT License](https://spdx.org/licenses/MIT.html).
-- [gruf/go-mutexes](https://codeberg.org/gruf/go-mutexes); mutex map. [MIT License](https://spdx.org/licenses/MIT.html).
-- [gruf/go-runners](https://codeberg.org/gruf/go-runners); worker pool library. [MIT License](https://spdx.org/licenses/MIT.html).
-- [gruf/go-store](https://codeberg.org/gruf/go-store); local media store. [MIT License](https://spdx.org/licenses/MIT.html).
+- Go-Playground:
+  - [go-playground/form](https://github.com/go-playground/form); funky form mapping support. [MIT License](https://spdx.org/licenses/MIT.html).
+  - [go-playground/validator](https://github.com/go-playground/validator); struct validation. [MIT License](https://spdx.org/licenses/MIT.html).
+- Gorilla:
+  - [gorilla/feeds](https://github.com/gorilla/feeds); RSS + Atom feed generation. [BSD-2-Clause License](https://spdx.org/licenses/BSD-2-Clause.html).
+  - [gorilla/websocket](https://github.com/gorilla/websocket); Websocket connectivity. [BSD-2-Clause License](https://spdx.org/licenses/BSD-2-Clause.html).
+- [go-swagger/go-swagger](https://github.com/go-swagger/go-swagger); Swagger OpenAPI spec generation. [Apache-2.0 License](https://spdx.org/licenses/Apache-2.0.html).
+- gruf:
+  - [gruf/go-bytesize](https://codeberg.org/gruf/go-bytesize); byte size parsing / formatting. [MIT License](https://spdx.org/licenses/MIT.html).
+  - [gruf/go-cache](https://codeberg.org/gruf/go-cache); object & result caching. [MIT License](https://spdx.org/licenses/MIT.html).
+  - [gruf/go-debug](https://codeberg.org/gruf/go-debug); debug build tag. [MIT License](https://spdx.org/licenses/MIT.html).
+  - [gruf/go-errors](https://codeberg.org/gruf/go-errors); performant multi-error checking [MIT License](https://spdx.org/licenses/MIT.html).
+  - [gruf/go-fastcopy](https://codeberg.org/gruf/go-fastcopy); performant pooled I/O copying [MIT License](https://spdx.org/licenses/MIT.html).
+  - [gruf/go-kv](https://codeberg.org/gruf/go-kv); log field formatting. [MIT License](https://spdx.org/licenses/MIT.html).
+  - [gruf/go-mutexes](https://codeberg.org/gruf/go-mutexes); safemutex & mutex map. [MIT License](https://spdx.org/licenses/MIT.html).
+  - [gruf/go-runners](https://codeberg.org/gruf/go-runners); workerpools and synchronization. [MIT License](https://spdx.org/licenses/MIT.html).
+  - [gruf/go-sched](https://codeberg.org/gruf/go-sched); task scheduler. [MIT License](https://spdx.org/licenses/MIT.html).
+  - [gruf/go-store](https://codeberg.org/gruf/go-store); file storage backend (local & s3). [MIT License](https://spdx.org/licenses/MIT.html).
 - [h2non/filetype](https://github.com/h2non/filetype); filetype checking. [MIT License](https://spdx.org/licenses/MIT.html).
-- [jackc/pgx](https://github.com/jackc/pgx); Postgres driver. [MIT License](https://spdx.org/licenses/MIT.html).
+- jackc:
+  - [jackc/pgx](https://github.com/jackc/pgconn); Postgres driver. [MIT License](https://spdx.org/licenses/MIT.html).
+  - [jackc/pgx](https://github.com/jackc/pgx); Postgres driver and toolkit. [MIT License](https://spdx.org/licenses/MIT.html).
+- [KimMachineGun/automemlimit](https://github.com/KimMachineGun/automemlimit); cgroups memory limit checking. [MIT License](https://spdx.org/licenses/MIT.html).
 - [mcuadros/go-syslog](https://github.com/mcuadros/go-syslog); Syslog server library. [MIT License](https://spdx.org/licenses/MIT.html).
 - [microcosm-cc/bluemonday](https://github.com/microcosm-cc/bluemonday); HTML user-input sanitization. [BSD-3-Clause License](https://spdx.org/licenses/BSD-3-Clause.html).
 - [miekg/dns](https://github.com/miekg/dns); DNS utilities. [Go License](https://go.dev/LICENSE).
+- [minio/minio-go](https://github.com/minio/minio-go); S3 client SDK. [Apache-2.0 License](https://spdx.org/licenses/Apache-2.0.html).
 - [mitchellh/mapstructure](https://github.com/mitchellh/mapstructure); Go interface => struct parsing. [MIT License](https://spdx.org/licenses/MIT.html).
 - [modernc.org/sqlite](https://gitlab.com/cznic/sqlite); cgo-free port of SQLite. [Other License](https://gitlab.com/cznic/sqlite/-/blob/master/LICENSE).
-  - [modernc.org/ccgo](https://gitlab.com/cznic/ccgo); c99 AST -> Go translater. [BSD-3-Clause License](https://spdx.org/licenses/BSD-3-Clause.html).
-  - [modernc.org/libc](https://gitlab.com/cznic/libc); C-runtime services. [BSD-3-Clause License](https://spdx.org/licenses/BSD-3-Clause.html).
-- [mvdan/xurls](https://github.com/mvdan/xurls); URL parsing regular expressions. [BSD-3-Clause License](https://spdx.org/licenses/BSD-3-Clause.html).
+- [mvdan.cc/xurls](https://github.com/mvdan/xurls); URL parsing regular expressions. [BSD-3-Clause License](https://spdx.org/licenses/BSD-3-Clause.html).
 - [oklog/ulid](https://github.com/oklog/ulid); sequential, database-friendly ID generation. [Apache-2.0 License](https://spdx.org/licenses/Apache-2.0.html).
-- [ReneKroon/ttlcache](https://github.com/ReneKroon/ttlcache); in-memory caching. [MIT License](https://spdx.org/licenses/MIT.html).
-- [robfig/cron](https://github.com/robfig/cron); cron job scheduling. [MIT License](https://spdx.org/licenses/MIT.html).
-- [russross/blackfriday](https://github.com/russross/blackfriday); markdown parsing for statuses. [Simplified BSD License](https://spdx.org/licenses/BSD-2-Clause.html).
-- [spf13/cobra](https://github.com/spf13/cobra); command-line tooling. [Apache-2.0 License](https://spdx.org/licenses/Apache-2.0.html).
-- [spf13/pflag](https://github.com/spf13/pflag); command-line flag utilities. [Apache-2.0 License](https://spdx.org/licenses/Apache-2.0.html).
-- [spf13/viper](https://github.com/spf13/viper); configuration management. [Apache-2.0 License](https://spdx.org/licenses/Apache-2.0.html).
+- [open-telemetry/opentelemetry-go](https://github.com/open-telemetry/opentelemetry-go); OpenTelemetry API + SDK. [Apache-2.0 License](https://spdx.org/licenses/Apache-2.0.html).
+- spf13:
+  - [spf13/cobra](https://github.com/spf13/cobra); command-line tooling. [Apache-2.0 License](https://spdx.org/licenses/Apache-2.0.html).
+  - [spf13/viper](https://github.com/spf13/viper); configuration management. [Apache-2.0 License](https://spdx.org/licenses/Apache-2.0.html).
 - [stretchr/testify](https://github.com/stretchr/testify); test framework. [MIT License](https://spdx.org/licenses/MIT.html).
-- [superseriousbusiness/exif-terminator](https://github.com/superseriousbusiness/exif-terminator); EXIF data removal. [GNU AGPL v3 LICENSE](https://spdx.org/licenses/AGPL-3.0-or-later.html).
-- [superseriousbusiness/activity](https://github.com/superseriousbusiness/activity) forked from [go-fed/activity](https://github.com/go-fed/activity); Golang ActivityPub/ActivityStreams library. [BSD-3-Clause License](https://spdx.org/licenses/BSD-3-Clause.html).
-- [superseriousbusiness/oauth2](https://github.com/superseriousbusiness/oauth2) forked from [go-oauth2/oauth2](https://github.com/go-oauth2/oauth2); OAuth server framework and token handling. [MIT License](https://spdx.org/licenses/MIT.html).
-- [go-swagger/go-swagger](https://github.com/go-swagger/go-swagger); Swagger OpenAPI spec generation. [Apache-2.0 License](https://spdx.org/licenses/Apache-2.0.html).
+- superseriousbusiness:
+  - [superseriousbusiness/activity](https://github.com/superseriousbusiness/activity) forked from [go-fed/activity](https://github.com/go-fed/activity); Golang ActivityPub/ActivityStreams library. [BSD-3-Clause License](https://spdx.org/licenses/BSD-3-Clause.html).
+  - [superseriousbusiness/exif-terminator](https://github.com/superseriousbusiness/exif-terminator); EXIF data removal. [GNU AGPL v3 LICENSE](https://spdx.org/licenses/AGPL-3.0-or-later.html).
+  - [superseriousbusiness/oauth2](https://github.com/superseriousbusiness/oauth2) forked from [go-oauth2/oauth2](https://github.com/go-oauth2/oauth2); OAuth server framework and token handling. [MIT License](https://spdx.org/licenses/MIT.html).
 - [tdewolff/minify](https://github.com/tdewolff/minify); HTML minification for Markdown-submitted posts. [MIT License](https://spdx.org/licenses/MIT.html).
+- [uber-go/automaxprocs](https://github.com/uber-go/automaxprocs); GOMAXPROCS automation. [MIT License](https://spdx.org/licenses/MIT.html).
+- [ulule/limiter](https://github.com/ulule/limiter); http rate limit middleware. [MIT License](https://spdx.org/licenses/MIT.html).
 - [uptrace/bun](https://github.com/uptrace/bun); database ORM. [BSD-2-Clause License](https://spdx.org/licenses/BSD-2-Clause.html).
 - [wagslane/go-password-validator](https://github.com/wagslane/go-password-validator); password strength validation. [MIT License](https://spdx.org/licenses/MIT.html).
-- [ulule/limiter](https://github.com/ulule/limiter); http rate limit middleware. [MIT License](https://spdx.org/licenses/MIT.html).
+- [yuin/goldmark](https://github.com/yuin/goldmark); markdown parser. [MIT License](https://spdx.org/licenses/MIT.html).
 
-### Image Attribution
+### Image Attribution and Licensing
 
-Sloth logo by [Anna Abramek](https://abramek.art/), Copyright (C) 2021-2022 the GoToSocial Authors.
+Sloth logo by [Anna Abramek](https://abramek.art/).
 
-### Developers
+<a rel="license" href="http://creativecommons.org/licenses/by-sa/4.0/"><img alt="Creative Commons License" style="border-width:0" src="https://i.creativecommons.org/l/by-sa/4.0/88x31.png" /></a><br />The GoToSocial sloth mascot is licensed under a <a rel="license" href="http://creativecommons.org/licenses/by-sa/4.0/">Creative Commons Attribution-ShareAlike 4.0 International License</a>.
+
+The Creative Commons Attribution-ShareAlike 4.0 International License license applies specifically to the following files and subdirectories of this repository:
+
+- [sloth logo png](./web/assets/logo.png)
+- [sloth logo svg](./web/assets/logo.svg)
+- [all default avatars](./web/assets/default_avatars)
+
+Under the terms of the license, you are free to:
+
+- Share — copy and redistribute the abovementioned material in any medium or format.
+- Adapt — remix, transform, and build upon the abovementioned material for any purpose, even commercially.
+
+### Team
 
 In alphabetical order (... and order of smell):
 
 - f0x \[[donate with liberapay](https://liberapay.com/f0x)\]
 - kim \[check out my code @ [codeberg](https://codeberg.org/gruf), or find me @ [@kim](https://k.iim.gay/@kim)\]
 - tobi \[[donate with liberapay](https://liberapay.com/GoToSocial/)\]
+- maloki \[[@maloki@goblin.technology](https://goblin.technology/@maloki)\]
 
 ### Special Thanks
 
@@ -273,23 +314,27 @@ Thanks to everyone who has used GtS, opened an issue, suggested something, given
 
 ## Sponsorship + Funding
 
-### OpenCollective
+**Please note: GoToSocial has NO CORPORATE SPONSORS and does not desire corporate sponsorship. In addition, we do not take donations from any of the following: adult websites, affiliate and review websites, casinos and gambling, insurance and financial products (credit), pharmacy products, SEO services and social media buying, VPN and proxy services, and essay writing services. Donations from such sources will be automatically rejected.**
+
+### Crowdfunding
 
 ![open collective Standard Sloth badge](https://opencollective.com/gotosocial/tiers/standard-sloth/badge.svg?label=Standard%20Sloth&color=brightgreen) ![open collective Stable Sloth badge](https://opencollective.com/gotosocial/tiers/stable-sloth/badge.svg?label=Stable%20Sloth&color=green) ![open collective Special Sloth badge](https://opencollective.com/gotosocial/tiers/special-sloth/badge.svg?label=Special%20Sloth&color=yellowgreen) ![open collective Sugar Sloth badge](https://opencollective.com/gotosocial/tiers/sugar-sloth/badge.svg?label=Sugar%20Sloth&color=blue)
 
-Currently, work on GoToSocial is funded through donations to our [OpenCollective](https://opencollective.com/gotosocial) page.
-
-If you would like to donate to GoToSocial to keep the lights on during development, [you can do so here](https://opencollective.com/gotosocial#support)! 💕 🦥 💕 Thank you!
-
-### LiberaPay
+If you would like to donate to GoToSocial to keep the lights on during development, [you can do so via our OpenCollective page](https://opencollective.com/gotosocial#support)!
 
 ![LiberaPay patrons](https://img.shields.io/liberapay/patrons/GoToSocial.svg?logo=liberapay) ![receives via LiberaPay](https://img.shields.io/liberapay/receives/GoToSocial.svg?logo=liberapay)
 
 If you prefer, we also have an account on LiberaPay! You can find that [right here](https://liberapay.com/GoToSocial/).
 
-### Provisos
+Crowdfunded donations to our OpenCollective and Liberapay accounts go towards paying the core team, paying server costs, and paying for GtS art, design, and other bits and bobs.
 
-**Please note: GoToSocial has NO CORPORATE SPONSORS and does not desire corporate sponsorship. In addition, we do not take donations from any of the following: adult websites, affiliate and review websites, casinos and gambling, insurance and financial products (credit), pharmacy products, SEO services and social media buying, VPN and proxy services, and essay writing services. Donations from such sources will be automatically rejected.**
+💕 🦥 💕 Thank you!
+
+### NLnet
+
+<img src="https://nlnet.nl/logo/NGI/NGIZero-green.hex.svg" width="75" alt="NGIZero logo"/>
+
+Combined with the above crowdfunding sources, 2023 Alpha development of GoToSocial is also funded by a 50,000 EUR grant from the [NGI0 Entrust Fund](https://nlnet.nl/entrust/), via [NLnet](https://nlnet.nl/). See [here](https://nlnet.nl/project/GoToSocial/#ack) for more details. The successful grant application is archived [here](https://github.com/superseriousbusiness/gotosocial/blob/main/archive/nlnet/2022-next-generation-internet-zero.md).
 
 ## License
 
@@ -303,4 +348,6 @@ If you modify the GoToSocial source code, and run that modified code in a way th
 
 > \[I\]f you modify the Program, your modified version must prominently offer all users interacting with it remotely through a computer network (if your version supports such interaction) an opportunity to receive the Corresponding Source of your version by providing access to the Corresponding Source from a network server at no charge, through some standard or customary means of facilitating copying of software.
 
-Copyright (C) 2021-2022 the GoToSocial Authors.
+Copyright (C) 2021-2023 GoToSocial Authors
+
+(I'm adding this here to take the crown of having the 1000th commit ~ kim)

ROADMAP.md

@@ -1,291 +1,69 @@
-# Roadmap <!-- omit in toc -->
+# Roadmap to Beta <!-- omit in toc -->
 
-This document contains the roadmap for GoToSocial to be considered eligible for its first [beta release](https://en.wikipedia.org/wiki/Software_release_life_cycle#Beta), and a summary of progress made towards that goal.
+This document contains the roadmap for GoToSocial to be considered eligible for its first [beta release](https://en.wikipedia.org/wiki/Software_release_life_cycle#Beta).
 
-All the dates contained in this document are best-guess only. It's useful to have a rough timeline, but things will undoubtedly change along the way.
+All the info contained in this document is best-guess only. It's useful to have a rough timeline we can direct people to, but things will undoubtedly change along the way; don't hold us to anything in this doc!
+
+Thank you to [NLnet](https://nlnet.nl) for helping to fund the alpha phase of GoToSocial development and get us moving towards beta!
+
+Big thank you to all of our [Open Collective](https://opencollective.com/gotosocial) and [Liberapay](https://liberapay.com/gotosocial) contributors, who've helped us keep the lights on! 💕 
 
 ## Table of Contents <!-- omit in toc -->
 
 - [Beta Aims](#beta-aims)
 - [Timeline](#timeline)
-  - [Q3 2022](#q3-2022)
-  - [Q4 2022](#q4-2022)
-  - [Q1 2023](#q1-2023)
-  - [Q2 2023](#q2-2023)
-  - [Q3 2023](#q3-2023)
-- [Detailed To-do List](#detailed-to-do-list)
-  - [API Groups + Endpoints](#api-groups--endpoints)
-  - [Non-API tasks](#non-api-tasks)
+  - [Mid 2023](#mid-2023)
+  - [Mid/late 2023](#midlate-2023)
+  - [Late 2023](#late-2023)
+  - [Early 2024](#early-2024)
+  - [And then...](#and-then)
 
 ## Beta Aims
 
-The milestone for beta in GoToSocial's case is to have a feature set that roughly compares to existing popular ActivityPub server implementations (minus features which we don't see as particularly important or useful). The beta milestone also includes GoToSocial-specific features which, we believe, are vital for user safety, such as local-only posting, block list subscriptions, and so on.
+Every software project has a different idea of what it means to be "in beta". In our case, we want the beta version of GoToSocial to provide a feature set that roughly compares to existing popular ActivityPub server implementations.
 
-Once feature parity is roughly in place, we will use beta time to start adding and polishing bonus features like slow federation, group moderation decisions, migration tooling, etc.
+In other words, you should be able to use the beta version of GoToSocial as your go-to(!) fedi instance for following people and making posts, without running into too many issues with things being missing or not working properly.
 
-We currently foresee entering beta phase around the middle of 2023, though this is only an estimate and is subject to change.
+Our target for beta also includes features which, we believe, are vital for user safety and wellbeing, such as non-replyable posting, block list subscription, allow-list support, and so on.
+
+Once we have the features we want in order to say "this is now a beta", we will use beta time to work on fixing bugs, tuning performance, and adding extra features which require a stable base to build on.
+
+Our hope for beta is also that, once we're there, the client API will stay relatively stable, and folks can confidently start building applications on top of GoToSocial without worrying that the API will change overnight and necessitate major rewrites.
+
+We currently foresee entering beta phase around the start of 2024, though this is only an estimate and is subject to change.
 
 ## Timeline
 
-What follows is a per-quarter timeline of features that will be implemented on the road to beta. The timeline is calculated on the following assumptions:
+What follows is a rough timeline of features that will be implemented on the road to beta. The timeline is calculated on the following assumptions:
 
-- We will continue to develop at a pace that is similar to what we've done over the previous year.
+- We will continue to develop at a pace that is similar to what we've done over the previous two years.
 - Our combined bandwidth is roughly equivalent to one person working full time on the project.
 - One distinct 'feature' takes one person 2-4 weeks to develop and test, depending on the size of the feature.
 - There will be other bugs to fix in between implementing things, so we shouldn't pack in features too tightly.
 
-Each quarter contains one 'big feature' which will probably take the longest amount of time that quarter.
+**This timeline is a best-guess about when things will be implemented. The order of feature releases is not fixed. Things may go faster or slower depending on the number of hurdles we run into, and the amount of help we receive from community contributions of code. The timeline also does not include background tasks like admin, polishing existing features, refactoring code, release management, and ensuring compatibility with other AP implementations.**
 
-**This timeline is a best-guess about when things will be implemented. The order of feature releases may change. It may go faster or slower depending on the number of hurdles we run into, and the amount of help we receive from community contributions of code. The timeline also does not include background tasks like admin, polishing existing features, refactoring code, and ensuring compatibility with other AP implementations.**
+### Mid 2023
 
-### Q3 2022
+- **Hashtags** -- implement federating hashtags and viewing hashtags to allow users to discover posts that they might be interested in.
+- **Block list subscriptions** -- allow instance admins to subscribe their instance to plaintext domain block lists (much of the work for this is already in place).
+- **Direct conversation view** -- allow users to easily page through all direct-message conversations they're a part of.
 
-- **Big Feature** -- User settings page: allow users to edit their profile page and settings through a web page served by the GtS instance, for cases where client apps don't implement their own settings page.
-- Block list subscription support: allow instance admins to subscribe their instance to plaintext domain block lists (much of the work for this is already in place).
-- Tag support: implement federating hashtags and viewing hashtags to allow users to discover posts that they might be interested in.
+### Mid/late 2023
 
-### Q4 2022
+- **Polls** -- implementing parsing, creating, and voting in polls.
+- **Mute posts/threads** -- opt-out of notifications for replies to a thread; no longer show a given post in your timeline.
+- **Limited peering/allowlists** -- allow instance admins to limit federation with other instances by default.
 
-- **Big Feature** — Video support: allow users to view + post videos (before we only implemented images and gifs).
-- Custom emoji support: allow users to use custom emojis in posts. Fetch custom emojis from remote instances and display them properly.
-- Pinned posts: allow users to 'feature' or 'pin' posts on their profile, and serve these featured posts via AP for other servers to see.
-- Profile fields: allow users to set 'fields' on their profile: short key/value items that can display pronouns, links to websites, etc.
+### Late 2023
 
-### Q1 2023
+- **Move activity** -- use the ActivityPub `Move` activity to support migration of a user's profile across servers.
+- **Sign-up flow** -- allow users to submit a sign-up request to an instance; allow admins to moderate sign-up requests.
 
-- **Big Feature** — Reports: allow users to file reports for abusive behavior etc. Expose the API for admins to view + act on reports. Handle federation of reports.
-- List support: allow users to create lists of other users, which they can view as separate timelines.
-- Polls support: allow users to create polls and vote in existing polls; federate the polls correctly via AP.
+### Early 2024
 
-### Q2 2023
+- **Non-replyable posts** -- design a non-replyable post path for GoToSocial based on https://github.com/mastodon/mastodon/issues/14762#issuecomment-1196889788; allow users to create non-replyable posts.
 
-- **Big Feature** — Sign-up flow. Allow users to submit a sign-up request to an instance. Allow admins to moderate sign-up requests.
-- Direct conversations: Allow users to see all direct-message conversations they're a part of.
-- Muting conversations: Allow users to mute notifications for conversations they're no longer interested in.
+### And then...
 
-### Q3 2023
-
-- **Big Feature** — Support the `Move` Activity, to allow users to move across instances and Fediverse implementations.
-- More to be confirmed.
-
-## Detailed To-do List
-
-### API Groups + Endpoints
-
-Unfilled check - not yet implemented.
-Filled check   - implemented.
-Crossed out    - will not be implemented / will be stubbed only.
-
-- [ ] Client-To-Server (Client REST API)
-  - [ ] Token and sign-in
-    - [x] /api/v1/apps POST                                 (Create an application)
-    - [ ] /api/v1/apps/verify_credentials GET               (Verify an application works)
-    - [x] /oauth/authorize GET                              (Show authorize page to user)
-    - [x] /oauth/authorize POST                             (Get an OAuth access code for an app/user)
-    - [x] /oauth/token POST                                 (Obtain a user-level access token)
-    - [ ] /oauth/revoke POST                                (Revoke a user-level access token)
-    - [x] /auth/sign_in GET                                 (Show form for user sign-in)
-    - [x] /auth/sign_in POST                                (Validate username and password and sign user in)
-  - [ ] Accounts
-    - [x] /api/v1/accounts POST                             (Register a new account)
-    - [x] /api/v1/accounts/verify_credentials GET           (Verify account credentials with a user token)
-    - [x] /api/v1/accounts/update_credentials PATCH         (Update user's display name/preferences)
-    - [x] /api/v1/accounts/:id GET                          (Get account information)
-    - [x] /api/v1/accounts/:id/statuses GET                 (Get an account's statuses)
-    - [x] /api/v1/accounts/:id/followers GET                (Get an account's followers)
-    - [x] /api/v1/accounts/:id/following GET                (Get an account's following)
-    - [ ] /api/v1/accounts/:id/featured_tags GET            (Get an account's featured tags)
-    - [ ] /api/v1/accounts/:id/lists GET                    (Get lists containing this account)
-    - [x] /api/v1/accounts/:id/follow POST                  (Follow this account)
-    - [x] /api/v1/accounts/:id/unfollow POST                (Unfollow this account)
-    - [x] /api/v1/accounts/:id/block POST                   (Block this account)
-    - [x] /api/v1/accounts/:id/unblock POST                 (Unblock this account)
-    - [ ] /api/v1/accounts/:id/mute POST                    (Mute this account)
-    - [ ] /api/v1/accounts/:id/unmute POST                  (Unmute this account)
-    - [ ] /api/v1/accounts/:id/pin POST                     (Feature this account on profile)
-    - [ ] /api/v1/accounts/:id/unpin POST                   (Remove this account from profile)
-    - [ ] /api/v1/accounts/:id/note POST                    (Make a personal note about this account)
-    - [x] /api/v1/accounts/relationships GET                (Check relationships with accounts)
-    - [ ] /api/v1/accounts/search GET                       (Search for an account)
-    - ~~/api/v1/accounts/:id/identity_proofs GET            (Get identity proofs for this account)~~
-  - [x] Favourites
-    - [x] /api/v1/favourites GET                            (See faved statuses)
-  - [ ] Mutes
-    - [ ] /api/v1/mutes GET                                 (See list of muted accounts)
-  - [x] Blocks
-    - [x] /api/v1/blocks GET                                (See list of blocked accounts)
-  - [x] Domain Blocks
-    - [x] /api/v1/domain_blocks GET                         (See list of domain blocks)
-    - [x] /api/v1/domain_blocks POST                        (Create a domain block)
-    - [x] /api/v1/domain_blocks DELETE                      (Remove a domain block)
-  - [ ] Filters
-    - [ ] /api/v1/filters GET                               (Get list of filters)
-    - [ ] /api/v1/filters/:id GET                           (View a filter)
-    - [ ] /api/v1/filters POST                              (Create a filter)
-    - [ ] /api/v1/filters/:id PUT                           (Update a filter)
-    - [ ] /api/v1/filters/:id DELETE                        (Remove a filter)
-  - [ ] Reports
-    - [ ] /api/v1/reports POST                              (File a report)
-  - [x] Follow Requests
-    - [x] /api/v1/follow_requests GET                       (View pending follow requests)
-    - [x] /api/v1/follow_requests/:id/authorize POST        (Accept a follow request)
-    - [x] /api/v1/follow_requests/:id/reject POST           (Reject a follow request)
-  - [ ] Featured Tags
-    - [ ] /api/v1/featured_tags GET                         (View featured tags)
-    - [ ] /api/v1/featured_tags POST                        (Feature a tag)
-    - [ ] /api/v1/featured_tags/:id DELETE                  (Unfeature a tag)
-    - ~~/api/v1/featured_tags/suggestions GET               (See most used tags)~~
-  - [ ] Preferences
-    - [ ] /api/v1/preferences GET                           (Get user preferences)
-  - [ ] Statuses
-    - [x] /api/v1/statuses POST                             (Create a new status)
-    - [x] /api/v1/statuses/:id GET                          (View an existing status)
-    - [x] /api/v1/statuses/:id DELETE                       (Delete a status)
-    - [x] /api/v1/statuses/:id/context GET                  (View statuses above and below status ID)
-    - [x] /api/v1/statuses/:id/reblogged_by GET             (See who has reblogged a status)
-    - [x] /api/v1/statuses/:id/favourited_by GET            (See who has faved a status)
-    - [x] /api/v1/statuses/:id/favourite POST               (Fave a status)
-    - [x] /api/v1/statuses/:id/unfavourite POST             (Unfave a status)
-    - [x] /api/v1/statuses/:id/reblog POST                  (Reblog a status)
-    - [x] /api/v1/statuses/:id/unreblog POST                (Undo a reblog)
-    - [ ] /api/v1/statuses/:id/mute POST                    (Mute notifications on a status)
-    - [ ] /api/v1/statuses/:id/unmute POST                  (Unmute notifications on a status)
-    - [ ] /api/v1/statuses/:id/pin POST                     (Pin a status to profile)
-    - [ ] /api/v1/statuses/:id/unpin POST                   (Unpin a status from profile)
-    - ~~/api/v1/statuses/:id/bookmark POST                  (Bookmark a status)~~
-    - ~~/api/v1/statuses/:id/unbookmark POST                (Undo a bookmark)~~
-  - [x] Media
-    - [x] /api/v1/media POST                                (Upload a media attachment)
-    - [x] /api/v1/media/:id GET                             (Get a media attachment)
-    - [x] /api/v1/media/:id PUT                             (Update an attachment)
-  - [ ] Polls
-    - [ ] /api/v1/polls/:id GET                             (Show a poll)
-    - [ ] /api/v1/polls/:id/votes POST                      (Vote on a poll)
-  - [ ] Timelines
-    - [x] /api/v1/timelines/public GET                      (See the public/federated timeline)
-    - [ ] /api/v1/timelines/tag/:hashtag GET                (Get public statuses that use hashtag)
-    - [x] /api/v1/timelines/home GET                        (View statuses from followed users)
-    - [ ] /api/v1/timelines/list/:list_id GET               (Get statuses in given list)
-  - [ ] Conversations
-    - [ ] /api/v1/conversations GET                         (Get a list of direct message convos)
-    - [ ] /api/v1/conversations/:id DELETE                  (Delete a direct message convo)
-    - [ ] /api/v1/conversations/:id POST                    (Mark a conversation as read)
-  - [ ] Lists
-    - [ ] /api/v1/lists GET                                 (Show a list of lists)
-    - [ ] /api/v1/lists/:id GET                             (Show a single list)
-    - [ ] /api/v1/lists POST                                (Create a new list)
-    - [ ] /api/v1/lists/:id PUT                             (Update a list)
-    - [ ] /api/v1/lists/:id DELETE                          (Delete a list)
-    - [ ] /api/v1/lists/:id/accounts GET                    (View which accounts are in a list)
-    - [ ] /api/v1/lists/:id/accounts POST                   (Add accounts to a list)
-    - [ ] /api/v1/lists/:id/accounts DELETE                 (Remove accounts from a list)
-  - [ ] Markers
-    - [ ] /api/v1/markers GET                               (Get saved timeline position)
-    - [ ] /api/v1/markers POST                              (Save timeline position)
-  - [x] Streaming
-    - [x] /api/v1/streaming WEBSOCKETS                      (Stream live events to user via websockets)
-  - [ ] Notifications
-    - [x] /api/v1/notifications GET                         (Get list of notifications)
-    - [x] /api/v1/notifications/:id GET                     (Get a single notification)
-    - [ ] /api/v1/notifications/clear POST                  (Clear all notifications)
-    - [ ] /api/v1/notifications/:id POST                    (Clear a single notification)
-  - [x] Search
-    - [x] /api/v2/search GET                                (Get search query results)
-  - [ ] Instance
-    - [x] /api/v1/instance GET                              (Get instance information)
-    - [x] /api/v1/instance PATCH                            (Update instance information)
-    - [x] /api/v1/instance/peers GET                        (Get list of federated servers)
-    - ~~ /api/v1/instance/activity GET                      (Instance activity over the last 3 months, binned weekly.)~~
-  - [x] Custom Emojis
-    - [x] /api/v1/custom_emojis GET                         (Show this server's custom emoji)
-  - [ ] Admin
-    - [x] /api/v1/admin/custom_emojis POST                  (Upload a custom emoji for instance-wide usage)
-    - [ ] /api/v1/admin/accounts GET                        (View accounts filtered by criteria)
-    - [ ] /api/v1/admin/accounts/:id GET                    (View admin level info about an account)
-    - [x] /api/v1/admin/accounts/:id/action POST            (Perform an admin action on account)
-    - [ ] /api/v1/admin/accounts/:id/approve POST           (Approve pending account)
-    - [ ] /api/v1/admin/accounts/:id/reject POST            (Deny pending account)
-    - [ ] /api/v1/admin/accounts/:id/enable POST            (Re-enable a disabled account)
-    - [ ] /api/v1/admin/accounts/:id/unsilence POST         (Unsilence a silenced account)
-    - [ ] /api/v1/admin/accounts/:id/unsuspend POST         (Unsuspend a suspended account)
-    - [ ] /api/v1/admin/reports GET                         (View all reports)
-    - [ ] /api/v1/admin/reports/:id GET                     (View a single report)
-    - [ ] /api/v1/admin/reports/:id/assign_to_self POST     (Assign a report to the current admin account)
-    - [ ] /api/v1/admin/reports/:id/unassign POST           (Unassign a report)
-    - [ ] /api/v1/admin/reports/:id/resolve POST            (Mark a report as resolved)
-    - [ ] /api/v1/admin/reports/:id/reopen POST             (Reopen a closed report)
-  - [ ] Announcements
-    - [ ] /api/v1/announcements GET                         (Show all current announcements)
-    - [ ] /api/v1/announcements/:id/dismiss POST            (Mark an announcement as read)
-    - [ ] /api/v1/announcements/:id/reactions/:name PUT     (Add a reaction to an announcement)
-    - [ ] /api/v1/announcements/:id/reactions/:name DELETE  (Remove a reaction from an announcement)
-  - [ ] Oembed
-    - [ ] /api/oembed GET                                   (Get oembed metadata for a status URL)
-  - ~~Proofs~~
-    - ~~/api/proofs GET                                     (View identity proofs)~~
-  - ~~Trends~~
-    - ~~/api/v1/trends GET                                  (Get a list of trending tags for the last week)~~
-  - ~~Directory~~
-    - ~~/api/v1/directory GET                               (Show profiles this server is aware of.)~~
-  - ~~Endorsements~~
-    - ~~/api/v1/endorsements GET                            (View existing endorsements)~~
-  - ~~Push~~
-    - ~~/api/v1/push/subscription POST                      (Subscribe to push notifications)~~
-    - ~~/api/v1/push/subscription GET                       (Get current subscription)~~
-    - ~~/api/v1/push/subscription PUT                       (Change notification types)~~
-    - ~~/api/v1/push/subscription DELETE                    (Delete current subscription)~~
-  - ~~Scheduled Statuses~~
-    - ~~/api/v1/scheduled_statuses GET                      (View scheduled statuses)~~
-    - ~~/api/v1/scheduled_statuses/:id GET                  (View a scheduled status)~~
-    - ~~/api/v1/scheduled_statuses/:id PUT                  (Schedule a status)~~
-    - ~~/api/v1/scheduled_statuses/:id DELETE               (Cancel a scheduled status)~~
-  - ~~Suggestions~~
-    - ~~/api/v1/suggestions GET                             (Get suggested accounts to follow)~~
-    - ~~/api/v1/suggestions/:account_id DELETE              (Delete a suggestion)~~
-  - ~~Bookmarks~~
-    - ~~/api/v1/bookmarks GET                               (See bookmarked statuses)~~
-
-### Non-API tasks
-
-- [ ] Server-To-Server (Federation protocol)
-  - [x] Mechanism to trigger side effects from client AP
-  - [x] Webfinger account lookups
-  - [ ] Federation modes
-    - [ ] 'Slow' federation
-      - [ ] Reputation scoring system for instances
-    - [x] 'Greedy' federation
-    - [ ] No federation (insulate this instance from the Fediverse)
-      - [ ] Allow list
-  - [x] Secure HTTP signatures (creation and validation)
-- [x] Storage
-  - [x] Internal/statuses/preferences etc
-    - [x] Postgres interface
-    - [x] SQLite interface
-  - [x] Media storage
-    - [x] Local storage interface
-    - [x] S3 storage interface
-- [x] Cache
-  - [x] In-memory cache
-- [ ] Security features
-  - [x] Authorization middleware
-  - [ ] Rate limiting middleware
-  - [ ] Scope middleware
-  - [x] Permissions/ACL middleware for admins+moderators
-- [ ] Documentation
-  - [x] Swagger API documentation
-  - [x] ReadTheDocs.io documentation
-  - [x] Deployment documentation
-  - [ ] App creation guide
-- [ ] Tooling
-  - [ ] Database migration tool
-  - [x] Admin CLI tool
-- [x] Build
-  - [x] Docker containerization
-    - [x] Dockerfile
-    - [x] docker-compose.yml
-- [ ] Tests
-  - [ ] Unit/integration
-    - [x] 25% coverage
-    - [ ] 50% coverage
-    - [ ] 90%+ coverage
-  - [ ] Benchmarking
+BETA TIME baby!

archive/design/reports.md

@@ -0,0 +1,109 @@
+# Reports / Flag Activity Federation
+
+This document contains design notes for GoToSocial's federated (s2s) Flag functionality.
+
+## What do existing implementations do?
+
+Information below is true as of Jan 2023. If you're reading this much later, the below things may no longer apply.
+
+### Mastodon
+
+Mastodon uses the Flag activity to federate reports to other AP servers.
+
+The Activity is wrapped inside a Create, which is addressed To the Inbox of the offending account.
+
+To preserve anonymity of the reporter, the instance Actor is used as the Actor of the Activity.
+
+Examples of the unwrapped Flag:
+
+```json
+{
+  "@context": "https://www.w3.org/ns/activitystreams",
+  "actor": "https://example.org/actor",
+  "content": "misinfo: it's not a good morning",
+  "id": "https://example.org/341e866f-93f8-4755-9cf4-f8fb17f434fd",
+  "object": [
+    "https://bad.instance/users/tobi",
+    "https://bad.instance/users/tobi/statuses/01GP388K19DGXSV3SW2RXWM533"
+  ],
+  "type": "Flag"
+}
+```
+
+```json
+{
+  "@context": "https://www.w3.org/ns/activitystreams",
+  "actor": "https://example.org/actor",
+  "content": "smellyyyyyyyyyyyyy",
+  "id": "https://example.org/3088184f-81b2-4545-8ce2-4cee4895449f",
+  "object": "https://bad.instance/users/tobi",
+  "type": "Flag"
+}
+```
+
+The `content` field contains the report description.
+
+The `object` field contains the reported Account URI and optionally one or more Note/Article/etc URIs. `object` value will be a string if just the Account URI is reported, or an array if the Account and one or more posts are being reported.
+
+The `id` field is a generic URI that doesn't reveal any metadata. Trying to GET this URI gives a 404 Not Found error, which is OK since all the info needed to process the report is included in the Activity already.
+
+### Misskey
+
+Misskey uses the Flag activity to federate reports to other AP servers.
+
+The Activity is wrapped inside a Create, which is addressed To the Inbox of the offending account.
+
+To preserve anonymity of the reporter, the instance Actor is used as the Actor of the Activity.
+
+Example of the unwrapped Flag:
+
+```json
+{
+  "@context": "https://www.w3.org/ns/activitystreams",
+  "actor": "https://example.org/users/909i45meeo",
+  "content": "Note: https://bad.instance/@tobi/statuses/01GPB56GPJ37JTK9HW308HQKBQ\n-----\nincites anti-police behaviour while being cute! ⛔",
+  "id": "https://example.org/db22128d-884e-4358-9935-6a7c3940535d",
+  "object": "https://bad.instance/users/tobi",
+  "type": "Flag"
+}
+```
+
+The `content` field contains the report description. Unlike with Mastodon, `content` also seems to include one or more statuses, as opposed to including statuses in the `object` field.
+
+The `object` field contains the reported Account URI.
+
+Trying to dereference the `id` field for a Misskey report with `Accept: application/activity+json` gives a 200 OK, but the returned content is some HTML unrelated to the report, so functionally equivalent to Mastodon's 404 behavior. Again, this is not really a problem.
+
+### Calckey
+
+Same as Misskey. Example:
+
+```json
+{
+  "@context": "https://www.w3.org/ns/activitystreams",
+  "actor": "https://example.org/users/97wsu4gkns",
+  "content": "Note: https://bad.instance/@tobi/statuses/01GPB56GPJ37JTK9HW308HQKBQ\n-----\nTest report from Calckey",
+  "id": "https://example.org/b9a02404-d007-4b31-8dd6-bfc53387ad85",
+  "object": "https://bad.instance/users/tobi",
+  "type": "Flag"
+}
+```
+
+### Pleroma / Akkoma
+
+todo
+
+### Friendica
+
+Unsure: Friendica and GoToSocial still don't federate properly with one another (https://github.com/superseriousbusiness/gotosocial/issues/169) so it's hard to test this.
+
+## What should GoToSocial do?
+
+Since the above implementations of Flag seem fairly consistent, GoToSocial should do more or less the same thing when federating reports outwards. So GtS ought to adopt the Mastodon behavior:
+
+- Wrap Flag Activity in a Create and deliver it to the offending account.
+- Use the GtS instance Actor as the Actor of the Flag.
+- Generate an ID that doesn't reveal who created the report.
+- Include Actor and one or more Note / Article / etc URIs in the `object` field
+
+For incoming reports, all the above fields should be handled in order to generate a report for admins to look at.

cmd/gotosocial/action/action.go

@@ -1,20 +1,19 @@
-/*
-   GoToSocial
-   Copyright (C) 2021-2022 GoToSocial Authors admin@gotosocial.org
-
-   This program is free software: you can redistribute it and/or modify
-   it under the terms of the GNU Affero General Public License as published by
-   the Free Software Foundation, either version 3 of the License, or
-   (at your option) any later version.
-
-   This program is distributed in the hope that it will be useful,
-   but WITHOUT ANY WARRANTY; without even the implied warranty of
-   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-   GNU Affero General Public License for more details.
-
-   You should have received a copy of the GNU Affero General Public License
-   along with this program.  If not, see <http://www.gnu.org/licenses/>.
-*/
+// GoToSocial
+// Copyright (C) GoToSocial Authors admin@gotosocial.org
+// SPDX-License-Identifier: AGPL-3.0-or-later
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
 package action
 

cmd/gotosocial/action/admin/account/account.go

@@ -1,273 +1,333 @@
-/*
-   GoToSocial
-   Copyright (C) 2021-2022 GoToSocial Authors admin@gotosocial.org
-
-   This program is free software: you can redistribute it and/or modify
-   it under the terms of the GNU Affero General Public License as published by
-   the Free Software Foundation, either version 3 of the License, or
-   (at your option) any later version.
-
-   This program is distributed in the hope that it will be useful,
-   but WITHOUT ANY WARRANTY; without even the implied warranty of
-   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-   GNU Affero General Public License for more details.
-
-   You should have received a copy of the GNU Affero General Public License
-   along with this program.  If not, see <http://www.gnu.org/licenses/>.
-*/
+// GoToSocial
+// Copyright (C) GoToSocial Authors admin@gotosocial.org
+// SPDX-License-Identifier: AGPL-3.0-or-later
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
 package account
 
 import (
 	"context"
-	"errors"
 	"fmt"
+	"os"
+	"text/tabwriter"
 	"time"
 
 	"github.com/superseriousbusiness/gotosocial/cmd/gotosocial/action"
 	"github.com/superseriousbusiness/gotosocial/internal/config"
 	"github.com/superseriousbusiness/gotosocial/internal/db/bundb"
+	"github.com/superseriousbusiness/gotosocial/internal/gtsmodel"
+	"github.com/superseriousbusiness/gotosocial/internal/state"
 	"github.com/superseriousbusiness/gotosocial/internal/validate"
 	"golang.org/x/crypto/bcrypt"
 )
 
-// Create creates a new account in the database using the provided flags.
-var Create action.GTSAction = func(ctx context.Context) error {
-	dbConn, err := bundb.NewBunDBService(ctx)
+func initState(ctx context.Context) (*state.State, error) {
+	var state state.State
+	state.Caches.Init()
+	state.Caches.Start()
+	state.Workers.Start()
+
+	// Set the state DB connection
+	dbConn, err := bundb.NewBunDBService(ctx, &state)
 	if err != nil {
-		return fmt.Errorf("error creating dbservice: %s", err)
+		return nil, fmt.Errorf("error creating dbConn: %w", err)
+	}
+	state.DB = dbConn
+
+	return &state, nil
+}
+
+func stopState(ctx context.Context, state *state.State) error {
+	if err := state.DB.Stop(ctx); err != nil {
+		return fmt.Errorf("error stopping dbConn: %w", err)
+	}
+
+	state.Workers.Stop()
+	state.Caches.Stop()
+
+	return nil
+}
+
+// Create creates a new account and user
+// in the database using the provided flags.
+var Create action.GTSAction = func(ctx context.Context) error {
+	state, err := initState(ctx)
+	if err != nil {
+		return err
 	}
 
 	username := config.GetAdminAccountUsername()
-	if username == "" {
-		return errors.New("no username set")
-	}
 	if err := validate.Username(username); err != nil {
 		return err
 	}
 
-	usernameAvailable, err := dbConn.IsUsernameAvailable(ctx, username)
+	usernameAvailable, err := state.DB.IsUsernameAvailable(ctx, username)
 	if err != nil {
 		return err
 	}
+
 	if !usernameAvailable {
 		return fmt.Errorf("username %s is already in use", username)
 	}
 
 	email := config.GetAdminAccountEmail()
-	if email == "" {
-		return errors.New("no email set")
-	}
 	if err := validate.Email(email); err != nil {
 		return err
 	}
 
-	emailAvailable, err := dbConn.IsEmailAvailable(ctx, email)
+	emailAvailable, err := state.DB.IsEmailAvailable(ctx, email)
 	if err != nil {
 		return err
 	}
+
 	if !emailAvailable {
 		return fmt.Errorf("email address %s is already in use", email)
 	}
 
 	password := config.GetAdminAccountPassword()
-	if password == "" {
-		return errors.New("no password set")
-	}
-	if err := validate.NewPassword(password); err != nil {
+	if err := validate.Password(password); err != nil {
 		return err
 	}
 
-	_, err = dbConn.NewSignup(ctx, username, "", false, email, password, nil, "", "", true, false)
+	if _, err := state.DB.NewSignup(ctx, gtsmodel.NewSignup{
+		Username:      username,
+		Email:         email,
+		Password:      password,
+		EmailVerified: true, // Assume cli user wants email marked as verified already.
+		PreApproved:   true, // Assume cli user wants account marked as approved already.
+	}); err != nil {
+		return err
+	}
+
+	return stopState(ctx, state)
+}
+
+// List returns all existing local accounts.
+var List action.GTSAction = func(ctx context.Context) error {
+	state, err := initState(ctx)
 	if err != nil {
 		return err
 	}
 
-	return dbConn.Stop(ctx)
+	users, err := state.DB.GetAllUsers(ctx)
+	if err != nil {
+		return err
+	}
+
+	fmtBool := func(b *bool) string {
+		if b == nil {
+			return "unknown"
+		}
+		if *b {
+			return "yes"
+		}
+		return "no"
+	}
+
+	fmtDate := func(t time.Time) string {
+		if t.Equal(time.Time{}) {
+			return "no"
+		}
+		return "yes"
+	}
+
+	w := tabwriter.NewWriter(os.Stdout, 0, 0, 1, ' ', 0)
+	fmt.Fprintln(w, "user\taccount\tapproved\tadmin\tmoderator\tsuspended\tconfirmed")
+	for _, u := range users {
+		fmt.Fprintf(w, "%s\t%s\t%s\t%s\t%s\t%s\t%s\n", u.Account.Username, u.AccountID, fmtBool(u.Approved), fmtBool(u.Admin), fmtBool(u.Moderator), fmtDate(u.Account.SuspendedAt), fmtDate(u.ConfirmedAt))
+	}
+	w.Flush()
+	return nil
 }
 
-// Confirm sets a user to Approved, sets Email to the current UnconfirmedEmail value, and sets ConfirmedAt to now.
+// Confirm sets a user to Approved, sets Email to the current
+// UnconfirmedEmail value, and sets ConfirmedAt to now.
 var Confirm action.GTSAction = func(ctx context.Context) error {
-	dbConn, err := bundb.NewBunDBService(ctx)
+	state, err := initState(ctx)
 	if err != nil {
-		return fmt.Errorf("error creating dbservice: %s", err)
+		return err
 	}
 
 	username := config.GetAdminAccountUsername()
-	if username == "" {
-		return errors.New("no username set")
-	}
 	if err := validate.Username(username); err != nil {
 		return err
 	}
 
-	a, err := dbConn.GetAccountByUsernameDomain(ctx, username, "")
+	account, err := state.DB.GetAccountByUsernameDomain(ctx, username, "")
 	if err != nil {
 		return err
 	}
 
-	u, err := dbConn.GetUserByAccountID(ctx, a.ID)
+	user, err := state.DB.GetUserByAccountID(ctx, account.ID)
 	if err != nil {
 		return err
 	}
 
-	updatingColumns := []string{"approved", "email", "confirmed_at", "updated_at"}
-	approved := true
-	u.Approved = &approved
-	u.Email = u.UnconfirmedEmail
-	u.ConfirmedAt = time.Now()
-	u.UpdatedAt = time.Now()
-	if err := dbConn.UpdateByID(ctx, u, u.ID, updatingColumns...); err != nil {
+	user.Approved = func() *bool { a := true; return &a }()
+	user.Email = user.UnconfirmedEmail
+	user.ConfirmedAt = time.Now()
+	if err := state.DB.UpdateUser(
+		ctx, user,
+		"approved", "email", "confirmed_at",
+	); err != nil {
 		return err
 	}
 
-	return dbConn.Stop(ctx)
+	return stopState(ctx, state)
 }
 
-// Promote sets a user to admin.
+// Promote sets admin + moderator flags on a user to true.
 var Promote action.GTSAction = func(ctx context.Context) error {
-	dbConn, err := bundb.NewBunDBService(ctx)
+	state, err := initState(ctx)
 	if err != nil {
-		return fmt.Errorf("error creating dbservice: %s", err)
+		return err
 	}
 
 	username := config.GetAdminAccountUsername()
-	if username == "" {
-		return errors.New("no username set")
-	}
 	if err := validate.Username(username); err != nil {
 		return err
 	}
 
-	a, err := dbConn.GetAccountByUsernameDomain(ctx, username, "")
+	account, err := state.DB.GetAccountByUsernameDomain(ctx, username, "")
 	if err != nil {
 		return err
 	}
 
-	u, err := dbConn.GetUserByAccountID(ctx, a.ID)
+	user, err := state.DB.GetUserByAccountID(ctx, account.ID)
 	if err != nil {
 		return err
 	}
 
-	admin := true
-	u.Admin = &admin
-	if err := dbConn.UpdateUser(ctx, u, "admin"); err != nil {
+	user.Admin = func() *bool { a := true; return &a }()
+	user.Moderator = func() *bool { a := true; return &a }()
+	if err := state.DB.UpdateUser(
+		ctx, user,
+		"admin", "moderator",
+	); err != nil {
 		return err
 	}
 
-	return dbConn.Stop(ctx)
+	return stopState(ctx, state)
 }
 
-// Demote sets admin on a user to false.
+// Demote sets admin + moderator flags on a user to false.
 var Demote action.GTSAction = func(ctx context.Context) error {
-	dbConn, err := bundb.NewBunDBService(ctx)
+	state, err := initState(ctx)
 	if err != nil {
-		return fmt.Errorf("error creating dbservice: %s", err)
+		return err
 	}
 
 	username := config.GetAdminAccountUsername()
-	if username == "" {
-		return errors.New("no username set")
-	}
 	if err := validate.Username(username); err != nil {
 		return err
 	}
 
-	a, err := dbConn.GetAccountByUsernameDomain(ctx, username, "")
+	a, err := state.DB.GetAccountByUsernameDomain(ctx, username, "")
 	if err != nil {
 		return err
 	}
 
-	u, err := dbConn.GetUserByAccountID(ctx, a.ID)
+	user, err := state.DB.GetUserByAccountID(ctx, a.ID)
 	if err != nil {
 		return err
 	}
 
-	admin := false
-	u.Admin = &admin
-	if err := dbConn.UpdateUser(ctx, u, "admin"); err != nil {
+	user.Admin = func() *bool { a := false; return &a }()
+	user.Moderator = func() *bool { a := false; return &a }()
+	if err := state.DB.UpdateUser(
+		ctx, user,
+		"admin", "moderator",
+	); err != nil {
 		return err
 	}
 
-	return dbConn.Stop(ctx)
+	return stopState(ctx, state)
 }
 
 // Disable sets Disabled to true on a user.
 var Disable action.GTSAction = func(ctx context.Context) error {
-	dbConn, err := bundb.NewBunDBService(ctx)
+	state, err := initState(ctx)
 	if err != nil {
-		return fmt.Errorf("error creating dbservice: %s", err)
+		return err
 	}
 
 	username := config.GetAdminAccountUsername()
-	if username == "" {
-		return errors.New("no username set")
-	}
 	if err := validate.Username(username); err != nil {
 		return err
 	}
 
-	a, err := dbConn.GetAccountByUsernameDomain(ctx, username, "")
+	account, err := state.DB.GetAccountByUsernameDomain(ctx, username, "")
 	if err != nil {
 		return err
 	}
 
-	u, err := dbConn.GetUserByAccountID(ctx, a.ID)
+	user, err := state.DB.GetUserByAccountID(ctx, account.ID)
 	if err != nil {
 		return err
 	}
 
-	disabled := true
-	u.Disabled = &disabled
-	if err := dbConn.UpdateUser(ctx, u, "disabled"); err != nil {
+	user.Disabled = func() *bool { d := true; return &d }()
+	if err := state.DB.UpdateUser(
+		ctx, user,
+		"disabled",
+	); err != nil {
 		return err
 	}
 
-	return dbConn.Stop(ctx)
+	return stopState(ctx, state)
 }
 
 // Password sets the password of target account.
 var Password action.GTSAction = func(ctx context.Context) error {
-	dbConn, err := bundb.NewBunDBService(ctx)
+	state, err := initState(ctx)
 	if err != nil {
-		return fmt.Errorf("error creating dbservice: %s", err)
+		return err
 	}
 
 	username := config.GetAdminAccountUsername()
-	if username == "" {
-		return errors.New("no username set")
-	}
 	if err := validate.Username(username); err != nil {
 		return err
 	}
 
 	password := config.GetAdminAccountPassword()
-	if password == "" {
-		return errors.New("no password set")
-	}
-	if err := validate.NewPassword(password); err != nil {
+	if err := validate.Password(password); err != nil {
 		return err
 	}
 
-	a, err := dbConn.GetAccountByUsernameDomain(ctx, username, "")
+	account, err := state.DB.GetAccountByUsernameDomain(ctx, username, "")
 	if err != nil {
 		return err
 	}
 
-	u, err := dbConn.GetUserByAccountID(ctx, a.ID)
+	user, err := state.DB.GetUserByAccountID(ctx, account.ID)
 	if err != nil {
 		return err
 	}
 
-	pw, err := bcrypt.GenerateFromPassword([]byte(password), bcrypt.DefaultCost)
+	encryptedPassword, err := bcrypt.GenerateFromPassword([]byte(password), bcrypt.DefaultCost)
 	if err != nil {
 		return fmt.Errorf("error hashing password: %s", err)
 	}
 
-	u.EncryptedPassword = string(pw)
-	if err := dbConn.UpdateUser(ctx, u, "encrypted_password"); err != nil {
+	user.EncryptedPassword = string(encryptedPassword)
+	if err := state.DB.UpdateUser(
+		ctx, user,
+		"encrypted_password",
+	); err != nil {
 		return err
 	}
 
-	return nil
+	return stopState(ctx, state)
 }

cmd/gotosocial/action/admin/media/list.go

@@ -0,0 +1,165 @@
+// GoToSocial
+// Copyright (C) GoToSocial Authors admin@gotosocial.org
+// SPDX-License-Identifier: AGPL-3.0-or-later
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+package media
+
+import (
+	"bufio"
+	"context"
+	"fmt"
+	"os"
+	"path"
+
+	"github.com/superseriousbusiness/gotosocial/cmd/gotosocial/action"
+	"github.com/superseriousbusiness/gotosocial/internal/config"
+	"github.com/superseriousbusiness/gotosocial/internal/db"
+	"github.com/superseriousbusiness/gotosocial/internal/db/bundb"
+	"github.com/superseriousbusiness/gotosocial/internal/gtsmodel"
+	"github.com/superseriousbusiness/gotosocial/internal/log"
+	"github.com/superseriousbusiness/gotosocial/internal/state"
+)
+
+type list struct {
+	dbService db.DB
+	state     *state.State
+	maxID     string
+	limit     int
+	out       *bufio.Writer
+}
+
+func (l *list) GetAllMediaPaths(ctx context.Context, filter func(*gtsmodel.MediaAttachment) string) ([]string, error) {
+	res := make([]string, 0, 100)
+	for {
+		attachments, err := l.dbService.GetAttachments(ctx, l.maxID, l.limit)
+		if err != nil {
+			return nil, fmt.Errorf("failed to retrieve media metadata from database: %w", err)
+		}
+
+		for _, a := range attachments {
+			v := filter(a)
+			if v != "" {
+				res = append(res, v)
+			}
+		}
+
+		// If we got less results than our limit, we've reached the
+		// last page to retrieve and we can break the loop. If the
+		// last batch happens to contain exactly the same amount of
+		// items as the limit we'll end up doing one extra query.
+		if len(attachments) < l.limit {
+			break
+		}
+
+		// Grab the last ID from the batch and set it as the maxID
+		// that'll be used in the next iteration so we don't get items
+		// we've already seen.
+		l.maxID = attachments[len(attachments)-1].ID
+	}
+	return res, nil
+}
+
+func setupList(ctx context.Context) (*list, error) {
+	var state state.State
+
+	state.Caches.Init()
+	state.Caches.Start()
+
+	state.Workers.Start()
+
+	dbService, err := bundb.NewBunDBService(ctx, &state)
+	if err != nil {
+		return nil, fmt.Errorf("error creating dbservice: %w", err)
+	}
+	state.DB = dbService
+
+	return &list{
+		dbService: dbService,
+		state:     &state,
+		limit:     200,
+		maxID:     "",
+		out:       bufio.NewWriter(os.Stdout),
+	}, nil
+}
+
+func (l *list) shutdown(ctx context.Context) error {
+	l.out.Flush()
+	err := l.dbService.Stop(ctx)
+	l.state.Workers.Stop()
+	l.state.Caches.Stop()
+
+	return err
+}
+
+var ListLocal action.GTSAction = func(ctx context.Context) error {
+	list, err := setupList(ctx)
+	if err != nil {
+		return err
+	}
+
+	defer func() {
+		// Ensure lister gets shutdown on exit.
+		if err := list.shutdown(ctx); err != nil {
+			log.Error(ctx, err)
+		}
+	}()
+
+	mediaPath := config.GetStorageLocalBasePath()
+	media, err := list.GetAllMediaPaths(
+		ctx,
+		func(m *gtsmodel.MediaAttachment) string {
+			if m.RemoteURL == "" {
+				return path.Join(mediaPath, m.File.Path)
+			}
+			return ""
+		})
+	if err != nil {
+		return err
+	}
+
+	for _, m := range media {
+		_, _ = list.out.WriteString(m + "\n")
+	}
+	return nil
+}
+
+var ListRemote action.GTSAction = func(ctx context.Context) error {
+	list, err := setupList(ctx)
+	if err != nil {
+		return err
+	}
+
+	defer func() {
+		// Ensure lister gets shutdown on exit.
+		if err := list.shutdown(ctx); err != nil {
+			log.Error(ctx, err)
+		}
+	}()
+
+	media, err := list.GetAllMediaPaths(
+		ctx,
+		func(m *gtsmodel.MediaAttachment) string {
+			return m.RemoteURL
+		})
+	if err != nil {
+		return err
+	}
+
+	for _, m := range media {
+		_, _ = list.out.WriteString(m + "\n")
+	}
+	return nil
+}

cmd/gotosocial/action/admin/media/prune/all.go

@@ -0,0 +1,61 @@
+// GoToSocial
+// Copyright (C) GoToSocial Authors admin@gotosocial.org
+// SPDX-License-Identifier: AGPL-3.0-or-later
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+package prune
+
+import (
+	"context"
+
+	"github.com/superseriousbusiness/gotosocial/cmd/gotosocial/action"
+	"github.com/superseriousbusiness/gotosocial/internal/config"
+	"github.com/superseriousbusiness/gotosocial/internal/gtscontext"
+	"github.com/superseriousbusiness/gotosocial/internal/log"
+)
+
+// All performs all media clean actions
+var All action.GTSAction = func(ctx context.Context) error {
+	// Setup pruning utilities.
+	prune, err := setupPrune(ctx)
+	if err != nil {
+		return err
+	}
+
+	defer func() {
+		// Ensure pruner gets shutdown on exit.
+		if err := prune.shutdown(ctx); err != nil {
+			log.Error(ctx, err)
+		}
+	}()
+
+	if config.GetAdminMediaPruneDryRun() {
+		log.Info(ctx, "prune DRY RUN")
+		ctx = gtscontext.SetDryRun(ctx)
+	}
+
+	days := config.GetMediaRemoteCacheDays()
+
+	// Perform the actual pruning with logging.
+	prune.cleaner.Media().All(ctx, days)
+	prune.cleaner.Emoji().All(ctx, days)
+
+	// Perform a cleanup of storage (for removed local dirs).
+	if err := prune.storage.Storage.Clean(ctx); err != nil {
+		log.Error(ctx, "error cleaning storage: %v", err)
+	}
+
+	return nil
+}

cmd/gotosocial/action/admin/media/prune/common.go

@@ -0,0 +1,92 @@
+// GoToSocial
+// Copyright (C) GoToSocial Authors admin@gotosocial.org
+// SPDX-License-Identifier: AGPL-3.0-or-later
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+package prune
+
+import (
+	"context"
+	"fmt"
+
+	"github.com/superseriousbusiness/gotosocial/internal/cleaner"
+	"github.com/superseriousbusiness/gotosocial/internal/db"
+	"github.com/superseriousbusiness/gotosocial/internal/db/bundb"
+	"github.com/superseriousbusiness/gotosocial/internal/gtserror"
+	"github.com/superseriousbusiness/gotosocial/internal/media"
+	"github.com/superseriousbusiness/gotosocial/internal/state"
+	gtsstorage "github.com/superseriousbusiness/gotosocial/internal/storage"
+)
+
+type prune struct {
+	dbService db.DB
+	storage   *gtsstorage.Driver
+	manager   *media.Manager
+	cleaner   *cleaner.Cleaner
+	state     *state.State
+}
+
+func setupPrune(ctx context.Context) (*prune, error) {
+	var state state.State
+
+	state.Caches.Init()
+	state.Caches.Start()
+
+	state.Workers.Start()
+
+	dbService, err := bundb.NewBunDBService(ctx, &state)
+	if err != nil {
+		return nil, fmt.Errorf("error creating dbservice: %w", err)
+	}
+	state.DB = dbService
+
+	//nolint:contextcheck
+	storage, err := gtsstorage.AutoConfig()
+	if err != nil {
+		return nil, fmt.Errorf("error creating storage backend: %w", err)
+	}
+	state.Storage = storage
+
+	//nolint:contextcheck
+	manager := media.NewManager(&state)
+
+	//nolint:contextcheck
+	cleaner := cleaner.New(&state)
+
+	return &prune{
+		dbService: dbService,
+		storage:   storage,
+		manager:   manager,
+		cleaner:   cleaner,
+		state:     &state,
+	}, nil
+}
+
+func (p *prune) shutdown(ctx context.Context) error {
+	errs := gtserror.NewMultiError(2)
+
+	if err := p.storage.Close(); err != nil {
+		errs.Appendf("error closing storage backend: %w", err)
+	}
+
+	if err := p.dbService.Stop(ctx); err != nil {
+		errs.Appendf("error stopping database: %w", err)
+	}
+
+	p.state.Workers.Stop()
+	p.state.Caches.Stop()
+
+	return errs.Combine()
+}

cmd/gotosocial/action/admin/media/prune/orphaned.go

@@ -1,71 +1,57 @@
-/*
-   GoToSocial
-   Copyright (C) 2021-2022 GoToSocial Authors admin@gotosocial.org
-
-   This program is free software: you can redistribute it and/or modify
-   it under the terms of the GNU Affero General Public License as published by
-   the Free Software Foundation, either version 3 of the License, or
-   (at your option) any later version.
-
-   This program is distributed in the hope that it will be useful,
-   but WITHOUT ANY WARRANTY; without even the implied warranty of
-   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-   GNU Affero General Public License for more details.
-
-   You should have received a copy of the GNU Affero General Public License
-   along with this program.  If not, see <http://www.gnu.org/licenses/>.
-*/
+// GoToSocial
+// Copyright (C) GoToSocial Authors admin@gotosocial.org
+// SPDX-License-Identifier: AGPL-3.0-or-later
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
 package prune
 
 import (
 	"context"
-	"fmt"
 
 	"github.com/superseriousbusiness/gotosocial/cmd/gotosocial/action"
 	"github.com/superseriousbusiness/gotosocial/internal/config"
-	"github.com/superseriousbusiness/gotosocial/internal/db/bundb"
+	"github.com/superseriousbusiness/gotosocial/internal/gtscontext"
 	"github.com/superseriousbusiness/gotosocial/internal/log"
-	"github.com/superseriousbusiness/gotosocial/internal/media"
-	gtsstorage "github.com/superseriousbusiness/gotosocial/internal/storage"
 )
 
 // Orphaned prunes orphaned media from storage.
 var Orphaned action.GTSAction = func(ctx context.Context) error {
-	dbService, err := bundb.NewBunDBService(ctx)
+	// Setup pruning utilities.
+	prune, err := setupPrune(ctx)
 	if err != nil {
-		return fmt.Errorf("error creating dbservice: %s", err)
+		return err
 	}
 
-	storage, err := gtsstorage.AutoConfig()
-	if err != nil {
-		return fmt.Errorf("error creating storage backend: %w", err)
+	defer func() {
+		// Ensure pruner gets shutdown on exit.
+		if err := prune.shutdown(ctx); err != nil {
+			log.Error(ctx, err)
+		}
+	}()
+
+	if config.GetAdminMediaPruneDryRun() {
+		log.Info(ctx, "prune DRY RUN")
+		ctx = gtscontext.SetDryRun(ctx)
 	}
 
-	manager, err := media.NewManager(dbService, storage)
-	if err != nil {
-		return fmt.Errorf("error instantiating mediamanager: %s", err)
-	}
+	// Perform the actual pruning with logging.
+	prune.cleaner.Media().LogPruneOrphaned(ctx)
 
-	dry := config.GetAdminMediaPruneDryRun()
-
-	pruned, err := manager.PruneOrphaned(ctx, dry)
-	if err != nil {
-		return fmt.Errorf("error pruning: %s", err)
-	}
-
-	if dry {
-		log.Infof("DRY RUN: %d stored items are orphaned and eligible to be pruned", pruned)
-	} else {
-		log.Infof("%d stored items were orphaned and pruned", pruned)
-	}
-
-	if err := storage.Close(); err != nil {
-		return fmt.Errorf("error closing storage backend: %w", err)
-	}
-
-	if err := dbService.Stop(ctx); err != nil {
-		return fmt.Errorf("error closing dbservice: %s", err)
+	// Perform a cleanup of storage (for removed local dirs).
+	if err := prune.storage.Storage.Clean(ctx); err != nil {
+		log.Error(ctx, "error cleaning storage: %v", err)
 	}
 
 	return nil

cmd/gotosocial/action/admin/media/prune/remote.go

@@ -0,0 +1,62 @@
+// GoToSocial
+// Copyright (C) GoToSocial Authors admin@gotosocial.org
+// SPDX-License-Identifier: AGPL-3.0-or-later
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+package prune
+
+import (
+	"context"
+	"time"
+
+	"github.com/superseriousbusiness/gotosocial/cmd/gotosocial/action"
+	"github.com/superseriousbusiness/gotosocial/internal/config"
+	"github.com/superseriousbusiness/gotosocial/internal/gtscontext"
+	"github.com/superseriousbusiness/gotosocial/internal/log"
+)
+
+// Remote prunes old and/or unused remote media.
+var Remote action.GTSAction = func(ctx context.Context) error {
+	// Setup pruning utilities.
+	prune, err := setupPrune(ctx)
+	if err != nil {
+		return err
+	}
+
+	defer func() {
+		// Ensure pruner gets shutdown on exit.
+		if err := prune.shutdown(ctx); err != nil {
+			log.Error(ctx, err)
+		}
+	}()
+
+	if config.GetAdminMediaPruneDryRun() {
+		log.Info(ctx, "prune DRY RUN")
+		ctx = gtscontext.SetDryRun(ctx)
+	}
+
+	t := time.Now().Add(-24 * time.Hour * time.Duration(config.GetMediaRemoteCacheDays()))
+
+	// Perform the actual pruning with logging.
+	prune.cleaner.Media().LogPruneUnused(ctx)
+	prune.cleaner.Media().LogUncacheRemote(ctx, t)
+
+	// Perform a cleanup of storage (for removed local dirs).
+	if err := prune.storage.Storage.Clean(ctx); err != nil {
+		log.Error(ctx, "error cleaning storage: %v", err)
+	}
+
+	return nil
+}

cmd/gotosocial/action/admin/trans/export.go

@@ -1,20 +1,19 @@
-/*
-   GoToSocial
-   Copyright (C) 2021-2022 GoToSocial Authors admin@gotosocial.org
-
-   This program is free software: you can redistribute it and/or modify
-   it under the terms of the GNU Affero General Public License as published by
-   the Free Software Foundation, either version 3 of the License, or
-   (at your option) any later version.
-
-   This program is distributed in the hope that it will be useful,
-   but WITHOUT ANY WARRANTY; without even the implied warranty of
-   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-   GNU Affero General Public License for more details.
-
-   You should have received a copy of the GNU Affero General Public License
-   along with this program.  If not, see <http://www.gnu.org/licenses/>.
-*/
+// GoToSocial
+// Copyright (C) GoToSocial Authors admin@gotosocial.org
+// SPDX-License-Identifier: AGPL-3.0-or-later
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
 package trans
 
@@ -26,16 +25,22 @@ import (
 	"github.com/superseriousbusiness/gotosocial/cmd/gotosocial/action"
 	"github.com/superseriousbusiness/gotosocial/internal/config"
 	"github.com/superseriousbusiness/gotosocial/internal/db/bundb"
+	"github.com/superseriousbusiness/gotosocial/internal/state"
 	"github.com/superseriousbusiness/gotosocial/internal/trans"
 )
 
 // Export exports info from the database into a file
 var Export action.GTSAction = func(ctx context.Context) error {
-	dbConn, err := bundb.NewBunDBService(ctx)
+	var state state.State
+
+	dbConn, err := bundb.NewBunDBService(ctx, &state)
 	if err != nil {
 		return fmt.Errorf("error creating dbservice: %s", err)
 	}
 
+	// Set the state DB connection
+	state.DB = dbConn
+
 	exporter := trans.NewExporter(dbConn)
 
 	path := config.GetAdminTransPath()

cmd/gotosocial/action/admin/trans/import.go

@@ -1,20 +1,19 @@
-/*
-   GoToSocial
-   Copyright (C) 2021-2022 GoToSocial Authors admin@gotosocial.org
-
-   This program is free software: you can redistribute it and/or modify
-   it under the terms of the GNU Affero General Public License as published by
-   the Free Software Foundation, either version 3 of the License, or
-   (at your option) any later version.
-
-   This program is distributed in the hope that it will be useful,
-   but WITHOUT ANY WARRANTY; without even the implied warranty of
-   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-   GNU Affero General Public License for more details.
-
-   You should have received a copy of the GNU Affero General Public License
-   along with this program.  If not, see <http://www.gnu.org/licenses/>.
-*/
+// GoToSocial
+// Copyright (C) GoToSocial Authors admin@gotosocial.org
+// SPDX-License-Identifier: AGPL-3.0-or-later
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
 package trans
 
@@ -26,16 +25,22 @@ import (
 	"github.com/superseriousbusiness/gotosocial/cmd/gotosocial/action"
 	"github.com/superseriousbusiness/gotosocial/internal/config"
 	"github.com/superseriousbusiness/gotosocial/internal/db/bundb"
+	"github.com/superseriousbusiness/gotosocial/internal/state"
 	"github.com/superseriousbusiness/gotosocial/internal/trans"
 )
 
 // Import imports info from a file into the database
 var Import action.GTSAction = func(ctx context.Context) error {
-	dbConn, err := bundb.NewBunDBService(ctx)
+	var state state.State
+
+	dbConn, err := bundb.NewBunDBService(ctx, &state)
 	if err != nil {
 		return fmt.Errorf("error creating dbservice: %s", err)
 	}
 
+	// Set the state DB connection
+	state.DB = dbConn
+
 	importer := trans.NewImporter(dbConn)
 
 	path := config.GetAdminTransPath()

cmd/gotosocial/action/debug/config/config.go

@@ -1,27 +1,26 @@
-/*
-   GoToSocial
-   Copyright (C) 2021-2022 GoToSocial Authors admin@gotosocial.org
-
-   This program is free software: you can redistribute it and/or modify
-   it under the terms of the GNU Affero General Public License as published by
-   the Free Software Foundation, either version 3 of the License, or
-   (at your option) any later version.
-
-   This program is distributed in the hope that it will be useful,
-   but WITHOUT ANY WARRANTY; without even the implied warranty of
-   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-   GNU Affero General Public License for more details.
-
-   You should have received a copy of the GNU Affero General Public License
-   along with this program.  If not, see <http://www.gnu.org/licenses/>.
-*/
+// GoToSocial
+// Copyright (C) GoToSocial Authors admin@gotosocial.org
+// SPDX-License-Identifier: AGPL-3.0-or-later
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
 package config
 
 import (
 	"context"
 	"encoding/json"
-	"fmt"
+	"os"
 
 	"github.com/superseriousbusiness/gotosocial/cmd/gotosocial/action"
 	"github.com/superseriousbusiness/gotosocial/internal/config"
@@ -39,13 +38,8 @@ var Config action.GTSAction = func(ctx context.Context) (err error) {
 		return err
 	}
 
-	// Marshal map to JSON
-	b, err := json.Marshal(raw)
-	if err != nil {
-		return err
-	}
-
-	// Print to stdout
-	fmt.Printf("%s\n", b)
-	return nil
+	enc := json.NewEncoder(os.Stdout)
+	enc.SetIndent("", "    ")
+	err = enc.Encode(raw)
+	return err
 }

cmd/gotosocial/action/server/server.go

@@ -1,56 +1,45 @@
-/*
-   GoToSocial
-   Copyright (C) 2021-2022 GoToSocial Authors admin@gotosocial.org
-
-   This program is free software: you can redistribute it and/or modify
-   it under the terms of the GNU Affero General Public License as published by
-   the Free Software Foundation, either version 3 of the License, or
-   (at your option) any later version.
-
-   This program is distributed in the hope that it will be useful,
-   but WITHOUT ANY WARRANTY; without even the implied warranty of
-   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-   GNU Affero General Public License for more details.
-
-   You should have received a copy of the GNU Affero General Public License
-   along with this program.  If not, see <http://www.gnu.org/licenses/>.
-*/
+// GoToSocial
+// Copyright (C) GoToSocial Authors admin@gotosocial.org
+// SPDX-License-Identifier: AGPL-3.0-or-later
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
 package server
 
 import (
 	"context"
+	"errors"
 	"fmt"
+	"net/http"
 	"os"
 	"os/signal"
 	"syscall"
+	"time"
 
+	"codeberg.org/gruf/go-sched"
+	"github.com/gin-gonic/gin"
 	"github.com/superseriousbusiness/gotosocial/cmd/gotosocial/action"
 	"github.com/superseriousbusiness/gotosocial/internal/api"
-	"github.com/superseriousbusiness/gotosocial/internal/api/client/account"
-	"github.com/superseriousbusiness/gotosocial/internal/api/client/admin"
-	"github.com/superseriousbusiness/gotosocial/internal/api/client/app"
-	"github.com/superseriousbusiness/gotosocial/internal/api/client/auth"
-	"github.com/superseriousbusiness/gotosocial/internal/api/client/blocks"
-	"github.com/superseriousbusiness/gotosocial/internal/api/client/emoji"
-	"github.com/superseriousbusiness/gotosocial/internal/api/client/favourites"
-	"github.com/superseriousbusiness/gotosocial/internal/api/client/fileserver"
-	"github.com/superseriousbusiness/gotosocial/internal/api/client/filter"
-	"github.com/superseriousbusiness/gotosocial/internal/api/client/followrequest"
-	"github.com/superseriousbusiness/gotosocial/internal/api/client/instance"
-	"github.com/superseriousbusiness/gotosocial/internal/api/client/list"
-	mediaModule "github.com/superseriousbusiness/gotosocial/internal/api/client/media"
-	"github.com/superseriousbusiness/gotosocial/internal/api/client/notification"
-	"github.com/superseriousbusiness/gotosocial/internal/api/client/search"
-	"github.com/superseriousbusiness/gotosocial/internal/api/client/status"
-	"github.com/superseriousbusiness/gotosocial/internal/api/client/streaming"
-	"github.com/superseriousbusiness/gotosocial/internal/api/client/timeline"
-	userClient "github.com/superseriousbusiness/gotosocial/internal/api/client/user"
-	"github.com/superseriousbusiness/gotosocial/internal/api/s2s/nodeinfo"
-	"github.com/superseriousbusiness/gotosocial/internal/api/s2s/user"
-	"github.com/superseriousbusiness/gotosocial/internal/api/s2s/webfinger"
-	"github.com/superseriousbusiness/gotosocial/internal/api/security"
-	"github.com/superseriousbusiness/gotosocial/internal/concurrency"
+	apiutil "github.com/superseriousbusiness/gotosocial/internal/api/util"
+	"github.com/superseriousbusiness/gotosocial/internal/gtserror"
+	"github.com/superseriousbusiness/gotosocial/internal/middleware"
+	tlprocessor "github.com/superseriousbusiness/gotosocial/internal/processing/timeline"
+	"github.com/superseriousbusiness/gotosocial/internal/timeline"
+	"github.com/superseriousbusiness/gotosocial/internal/tracing"
+	"github.com/superseriousbusiness/gotosocial/internal/visibility"
+	"go.uber.org/automaxprocs/maxprocs"
+
 	"github.com/superseriousbusiness/gotosocial/internal/config"
 	"github.com/superseriousbusiness/gotosocial/internal/db/bundb"
 	"github.com/superseriousbusiness/gotosocial/internal/email"
@@ -60,24 +49,47 @@ import (
 	"github.com/superseriousbusiness/gotosocial/internal/httpclient"
 	"github.com/superseriousbusiness/gotosocial/internal/log"
 	"github.com/superseriousbusiness/gotosocial/internal/media"
-	"github.com/superseriousbusiness/gotosocial/internal/messages"
 	"github.com/superseriousbusiness/gotosocial/internal/oauth"
 	"github.com/superseriousbusiness/gotosocial/internal/oidc"
 	"github.com/superseriousbusiness/gotosocial/internal/processing"
 	"github.com/superseriousbusiness/gotosocial/internal/router"
+	"github.com/superseriousbusiness/gotosocial/internal/state"
 	gtsstorage "github.com/superseriousbusiness/gotosocial/internal/storage"
 	"github.com/superseriousbusiness/gotosocial/internal/transport"
 	"github.com/superseriousbusiness/gotosocial/internal/typeutils"
 	"github.com/superseriousbusiness/gotosocial/internal/web"
+
+	// Inherit memory limit if set from cgroup
+	_ "github.com/KimMachineGun/automemlimit"
 )
 
 // Start creates and starts a gotosocial server
 var Start action.GTSAction = func(ctx context.Context) error {
-	dbService, err := bundb.NewBunDBService(ctx)
+	if _, err := maxprocs.Set(maxprocs.Logger(nil)); err != nil {
+		log.Infof(ctx, "could not set CPU limits from cgroup: %s", err)
+	}
+
+	var state state.State
+
+	// Initialize caches
+	state.Caches.Init()
+	state.Caches.Start()
+	defer state.Caches.Stop()
+
+	// Initialize Tracing
+	if err := tracing.Initialize(); err != nil {
+		return fmt.Errorf("error initializing tracing: %w", err)
+	}
+
+	// Open connection to the database
+	dbService, err := bundb.NewBunDBService(ctx, &state)
 	if err != nil {
 		return fmt.Errorf("error creating dbservice: %s", err)
 	}
 
+	// Set the state DB connection
+	state.DB = dbService
+
 	if err := dbService.CreateInstanceAccount(ctx); err != nil {
 		return fmt.Errorf("error creating instance account: %s", err)
 	}
@@ -86,134 +98,177 @@ var Start action.GTSAction = func(ctx context.Context) error {
 		return fmt.Errorf("error creating instance instance: %s", err)
 	}
 
-	// Create the client API and federator worker pools
-	// NOTE: these MUST NOT be used until they are passed to the
-	// processor and it is started. The reason being that the processor
-	// sets the Worker process functions and start the underlying pools
-	clientWorker := concurrency.NewWorkerPool[messages.FromClientAPI](-1, -1)
-	fedWorker := concurrency.NewWorkerPool[messages.FromFederator](-1, -1)
-
-	federatingDB := federatingdb.New(dbService, fedWorker)
-
-	router, err := router.New(ctx, dbService)
-	if err != nil {
-		return fmt.Errorf("error creating router: %s", err)
-	}
-
-	// build converters and util
-	typeConverter := typeutils.NewConverter(dbService)
-
 	// Open the storage backend
 	storage, err := gtsstorage.AutoConfig()
 	if err != nil {
 		return fmt.Errorf("error creating storage backend: %w", err)
 	}
 
-	// Build HTTP client (TODO: add configurables here)
-	client := httpclient.New(httpclient.Config{})
+	// Set the state storage driver
+	state.Storage = storage
 
-	// build backend handlers
-	mediaManager, err := media.NewManager(dbService, storage)
-	if err != nil {
-		return fmt.Errorf("error creating media manager: %s", err)
-	}
+	// Build HTTP client
+	client := httpclient.New(httpclient.Config{
+		AllowRanges:           config.MustParseIPPrefixes(config.GetHTTPClientAllowIPs()),
+		BlockRanges:           config.MustParseIPPrefixes(config.GetHTTPClientBlockIPs()),
+		Timeout:               config.GetHTTPClientTimeout(),
+		TLSInsecureSkipVerify: config.GetHTTPClientTLSInsecureSkipVerify(),
+	})
+
+	// Initialize workers.
+	state.Workers.Start()
+	defer state.Workers.Stop()
+
+	// Add a task to the scheduler to sweep caches.
+	// Frequency = 1 * minute
+	// Threshold = 80% capacity
+	sweep := func(time.Time) { state.Caches.Sweep(80) }
+	job := sched.NewJob(sweep).Every(time.Minute)
+	_ = state.Workers.Scheduler.Schedule(job)
+
+	// Build handlers used in later initializations.
+	mediaManager := media.NewManager(&state)
 	oauthServer := oauth.New(ctx, dbService)
-	transportController := transport.NewController(dbService, federatingDB, &federation.Clock{}, client)
-	federator := federation.NewFederator(dbService, federatingDB, transportController, typeConverter, mediaManager)
+	typeConverter := typeutils.NewConverter(dbService)
+	filter := visibility.NewFilter(&state)
+	federatingDB := federatingdb.New(&state, typeConverter)
+	transportController := transport.NewController(&state, federatingDB, &federation.Clock{}, client)
+	federator := federation.NewFederator(&state, federatingDB, transportController, typeConverter, mediaManager)
 
-	// decide whether to create a noop email sender (won't send emails) or a real one
+	// Decide whether to create a noop email
+	// sender (won't send emails) or a real one.
 	var emailSender email.Sender
 	if smtpHost := config.GetSMTPHost(); smtpHost != "" {
-		// host is defined so create a proper sender
+		// Host is defined; create a proper sender.
 		emailSender, err = email.NewSender()
 		if err != nil {
 			return fmt.Errorf("error creating email sender: %s", err)
 		}
 	} else {
-		// no host is defined so create a noop sender
+		// No host is defined; create a noop sender.
 		emailSender, err = email.NewNoopSender(nil)
 		if err != nil {
 			return fmt.Errorf("error creating noop email sender: %s", err)
 		}
 	}
 
-	// create and start the message processor using the other services we've created so far
-	processor := processing.NewProcessor(typeConverter, federator, oauthServer, mediaManager, storage, dbService, emailSender, clientWorker, fedWorker)
-	if err := processor.Start(); err != nil {
-		return fmt.Errorf("error starting processor: %s", err)
+	// Initialize timelines.
+	state.Timelines.Home = timeline.NewManager(
+		tlprocessor.HomeTimelineGrab(&state),
+		tlprocessor.HomeTimelineFilter(&state, filter),
+		tlprocessor.HomeTimelineStatusPrepare(&state, typeConverter),
+		tlprocessor.SkipInsert(),
+	)
+	if err := state.Timelines.Home.Start(); err != nil {
+		return fmt.Errorf("error starting home timeline: %s", err)
 	}
 
-	idp, err := oidc.NewIDP(ctx)
+	state.Timelines.List = timeline.NewManager(
+		tlprocessor.ListTimelineGrab(&state),
+		tlprocessor.ListTimelineFilter(&state, filter),
+		tlprocessor.ListTimelineStatusPrepare(&state, typeConverter),
+		tlprocessor.SkipInsert(),
+	)
+	if err := state.Timelines.List.Start(); err != nil {
+		return fmt.Errorf("error starting list timeline: %s", err)
+	}
+
+	// Create the processor using all the other services we've created so far.
+	processor := processing.NewProcessor(typeConverter, federator, oauthServer, mediaManager, &state, emailSender)
+
+	// Set state client / federator worker enqueue functions
+	state.Workers.EnqueueClientAPI = processor.EnqueueClientAPI
+	state.Workers.EnqueueFederator = processor.EnqueueFederator
+
+	/*
+		HTTP router initialization
+	*/
+
+	router, err := router.New(ctx)
 	if err != nil {
-		return fmt.Errorf("error creating oidc idp: %s", err)
+		return fmt.Errorf("error creating router: %s", err)
 	}
 
-	// build web module
-	webModule := web.New(processor)
-
-	// build client api modules
-	authModule := auth.New(dbService, idp, processor)
-	accountModule := account.New(processor)
-	instanceModule := instance.New(processor)
-	appsModule := app.New(processor)
-	followRequestsModule := followrequest.New(processor)
-	webfingerModule := webfinger.New(processor)
-	nodeInfoModule := nodeinfo.New(processor)
-	usersModule := user.New(processor)
-	timelineModule := timeline.New(processor)
-	notificationModule := notification.New(processor)
-	searchModule := search.New(processor)
-	filtersModule := filter.New(processor)
-	emojiModule := emoji.New(processor)
-	listsModule := list.New(processor)
-	mm := mediaModule.New(processor)
-	fileServerModule := fileserver.New(processor)
-	adminModule := admin.New(processor)
-	statusModule := status.New(processor)
-	securityModule := security.New(dbService, oauthServer)
-	streamingModule := streaming.New(processor)
-	favouritesModule := favourites.New(processor)
-	blocksModule := blocks.New(processor)
-	userClientModule := userClient.New(processor)
-
-	apis := []api.ClientModule{
-		// modules with middleware go first
-		securityModule,
-		authModule,
-
-		// now the web module
-		webModule,
-
-		// now everything else
-		accountModule,
-		instanceModule,
-		appsModule,
-		followRequestsModule,
-		mm,
-		fileServerModule,
-		adminModule,
-		statusModule,
-		webfingerModule,
-		nodeInfoModule,
-		usersModule,
-		timelineModule,
-		notificationModule,
-		searchModule,
-		filtersModule,
-		emojiModule,
-		listsModule,
-		streamingModule,
-		favouritesModule,
-		blocksModule,
-		userClientModule,
+	middlewares := []gin.HandlerFunc{
+		middleware.AddRequestID(config.GetRequestIDHeader()), // requestID middleware must run before tracing
 	}
+	if config.GetTracingEnabled() {
+		middlewares = append(middlewares, tracing.InstrumentGin())
+	}
+	middlewares = append(middlewares, []gin.HandlerFunc{
+		// note: hooks adding ctx fields must be ABOVE
+		// the logger, otherwise won't be accessible.
+		middleware.Logger(config.GetLogClientIP()),
+		middleware.UserAgent(),
+		middleware.CORS(),
+		middleware.ExtraHeaders(),
+	}...)
 
-	for _, m := range apis {
-		if err := m.Route(router); err != nil {
-			return fmt.Errorf("routing error: %s", err)
+	// attach global middlewares which are used for every request
+	router.AttachGlobalMiddleware(middlewares...)
+
+	// attach global no route / 404 handler to the router
+	router.AttachNoRouteHandler(func(c *gin.Context) {
+		apiutil.ErrorHandler(c, gtserror.NewErrorNotFound(errors.New(http.StatusText(http.StatusNotFound))), processor.InstanceGetV1)
+	})
+
+	// build router modules
+	var idp oidc.IDP
+	if config.GetOIDCEnabled() {
+		idp, err = oidc.NewIDP(ctx)
+		if err != nil {
+			return fmt.Errorf("error creating oidc idp: %w", err)
 		}
 	}
 
+	routerSession, err := dbService.GetSession(ctx)
+	if err != nil {
+		return fmt.Errorf("error retrieving router session for session middleware: %w", err)
+	}
+
+	sessionName, err := middleware.SessionName()
+	if err != nil {
+		return fmt.Errorf("error generating session name for session middleware: %w", err)
+	}
+
+	var (
+		authModule        = api.NewAuth(dbService, processor, idp, routerSession, sessionName) // auth/oauth paths
+		clientModule      = api.NewClient(dbService, processor)                                // api client endpoints
+		fileserverModule  = api.NewFileserver(processor)                                       // fileserver endpoints
+		wellKnownModule   = api.NewWellKnown(processor)                                        // .well-known endpoints
+		nodeInfoModule    = api.NewNodeInfo(processor)                                         // nodeinfo endpoint
+		activityPubModule = api.NewActivityPub(dbService, processor)                           // ActivityPub endpoints
+		webModule         = web.New(dbService, processor)                                      // web pages + user profiles + settings panels etc
+	)
+
+	// create required middleware
+	// rate limiting
+	limit := config.GetAdvancedRateLimitRequests()
+	clLimit := middleware.RateLimit(limit)  // client api
+	s2sLimit := middleware.RateLimit(limit) // server-to-server (AP)
+	fsLimit := middleware.RateLimit(limit)  // fileserver / web templates
+
+	// throttling
+	cpuMultiplier := config.GetAdvancedThrottlingMultiplier()
+	retryAfter := config.GetAdvancedThrottlingRetryAfter()
+	clThrottle := middleware.Throttle(cpuMultiplier, retryAfter)  // client api
+	s2sThrottle := middleware.Throttle(cpuMultiplier, retryAfter) // server-to-server (AP)
+	fsThrottle := middleware.Throttle(cpuMultiplier, retryAfter)  // fileserver / web templates
+	pkThrottle := middleware.Throttle(cpuMultiplier, retryAfter)  // throttle public key endpoint separately
+
+	gzip := middleware.Gzip() // applied to all except fileserver
+
+	// these should be routed in order;
+	// apply throttling *after* rate limiting
+	authModule.Route(router, clLimit, clThrottle, gzip)
+	clientModule.Route(router, clLimit, clThrottle, gzip)
+	fileserverModule.Route(router, fsLimit, fsThrottle)
+	wellKnownModule.Route(router, gzip, s2sLimit, s2sThrottle)
+	nodeInfoModule.Route(router, s2sLimit, s2sThrottle, gzip)
+	activityPubModule.Route(router, s2sLimit, s2sThrottle, gzip)
+	activityPubModule.RoutePublicKey(router, s2sLimit, pkThrottle, gzip)
+	webModule.Route(router, fsLimit, fsThrottle, gzip)
+
 	gts, err := gotosocial.NewServer(dbService, router, federator, mediaManager)
 	if err != nil {
 		return fmt.Errorf("error creating gotosocial service: %s", err)
@@ -223,22 +278,17 @@ var Start action.GTSAction = func(ctx context.Context) error {
 		return fmt.Errorf("error starting gotosocial service: %s", err)
 	}
 
-	// perform initial media prune in case value of MediaRemoteCacheDays changed
-	if err := processor.AdminMediaPrune(ctx, config.GetMediaRemoteCacheDays()); err != nil {
-		return fmt.Errorf("error during initial media prune: %s", err)
-	}
-
 	// catch shutdown signals from the operating system
 	sigs := make(chan os.Signal, 1)
-	signal.Notify(sigs, os.Interrupt, syscall.SIGTERM)
-	sig := <-sigs
-	log.Infof("received signal %s, shutting down", sig)
+	signal.Notify(sigs, syscall.SIGINT, syscall.SIGTERM)
+	sig := <-sigs // block until signal received
+	log.Infof(ctx, "received signal %s, shutting down", sig)
 
 	// close down all running services in order
 	if err := gts.Stop(ctx); err != nil {
 		return fmt.Errorf("error closing gotosocial service: %s", err)
 	}
 
-	log.Info("done! exiting...")
+	log.Info(ctx, "done! exiting...")
 	return nil
 }

cmd/gotosocial/action/testrig/testrig.go

@@ -1,26 +1,26 @@
-/*
-   GoToSocial
-   Copyright (C) 2021-2022 GoToSocial Authors admin@gotosocial.org
-
-   This program is free software: you can redistribute it and/or modify
-   it under the terms of the GNU Affero General Public License as published by
-   the Free Software Foundation, either version 3 of the License, or
-   (at your option) any later version.
-
-   This program is distributed in the hope that it will be useful,
-   but WITHOUT ANY WARRANTY; without even the implied warranty of
-   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-   GNU Affero General Public License for more details.
-
-   You should have received a copy of the GNU Affero General Public License
-   along with this program.  If not, see <http://www.gnu.org/licenses/>.
-*/
+// GoToSocial
+// Copyright (C) GoToSocial Authors admin@gotosocial.org
+// SPDX-License-Identifier: AGPL-3.0-or-later
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
 package testrig
 
 import (
 	"bytes"
 	"context"
+	"errors"
 	"fmt"
 	"io"
 	"net/http"
@@ -28,152 +28,163 @@ import (
 	"os/signal"
 	"syscall"
 
+	"github.com/gin-gonic/gin"
 	"github.com/superseriousbusiness/gotosocial/cmd/gotosocial/action"
 	"github.com/superseriousbusiness/gotosocial/internal/api"
-	"github.com/superseriousbusiness/gotosocial/internal/api/client/account"
-	"github.com/superseriousbusiness/gotosocial/internal/api/client/admin"
-	"github.com/superseriousbusiness/gotosocial/internal/api/client/app"
-	"github.com/superseriousbusiness/gotosocial/internal/api/client/auth"
-	"github.com/superseriousbusiness/gotosocial/internal/api/client/blocks"
-	"github.com/superseriousbusiness/gotosocial/internal/api/client/emoji"
-	"github.com/superseriousbusiness/gotosocial/internal/api/client/favourites"
-	"github.com/superseriousbusiness/gotosocial/internal/api/client/fileserver"
-	"github.com/superseriousbusiness/gotosocial/internal/api/client/filter"
-	"github.com/superseriousbusiness/gotosocial/internal/api/client/followrequest"
-	"github.com/superseriousbusiness/gotosocial/internal/api/client/instance"
-	"github.com/superseriousbusiness/gotosocial/internal/api/client/list"
-	mediaModule "github.com/superseriousbusiness/gotosocial/internal/api/client/media"
-	"github.com/superseriousbusiness/gotosocial/internal/api/client/notification"
-	"github.com/superseriousbusiness/gotosocial/internal/api/client/search"
-	"github.com/superseriousbusiness/gotosocial/internal/api/client/status"
-	"github.com/superseriousbusiness/gotosocial/internal/api/client/streaming"
-	"github.com/superseriousbusiness/gotosocial/internal/api/client/timeline"
-	userClient "github.com/superseriousbusiness/gotosocial/internal/api/client/user"
-	"github.com/superseriousbusiness/gotosocial/internal/api/s2s/nodeinfo"
-	"github.com/superseriousbusiness/gotosocial/internal/api/s2s/user"
-	"github.com/superseriousbusiness/gotosocial/internal/api/s2s/webfinger"
-	"github.com/superseriousbusiness/gotosocial/internal/api/security"
-	"github.com/superseriousbusiness/gotosocial/internal/concurrency"
+	apiutil "github.com/superseriousbusiness/gotosocial/internal/api/util"
+	"github.com/superseriousbusiness/gotosocial/internal/config"
 	"github.com/superseriousbusiness/gotosocial/internal/gotosocial"
+	"github.com/superseriousbusiness/gotosocial/internal/gtserror"
 	"github.com/superseriousbusiness/gotosocial/internal/log"
-	"github.com/superseriousbusiness/gotosocial/internal/messages"
+	"github.com/superseriousbusiness/gotosocial/internal/middleware"
 	"github.com/superseriousbusiness/gotosocial/internal/oidc"
+	tlprocessor "github.com/superseriousbusiness/gotosocial/internal/processing/timeline"
+	"github.com/superseriousbusiness/gotosocial/internal/state"
 	"github.com/superseriousbusiness/gotosocial/internal/storage"
+	"github.com/superseriousbusiness/gotosocial/internal/timeline"
+	"github.com/superseriousbusiness/gotosocial/internal/tracing"
+	"github.com/superseriousbusiness/gotosocial/internal/visibility"
 	"github.com/superseriousbusiness/gotosocial/internal/web"
 	"github.com/superseriousbusiness/gotosocial/testrig"
 )
 
 // Start creates and starts a gotosocial testrig server
 var Start action.GTSAction = func(ctx context.Context) error {
+	var state state.State
+
 	testrig.InitTestConfig()
 	testrig.InitTestLog()
 
-	dbService := testrig.NewTestDB()
-	testrig.StandardDBSetup(dbService, nil)
-	router := testrig.NewTestRouter(dbService)
-	var storageBackend *storage.Driver
-	if os.Getenv("GTS_STORAGE_BACKEND") == "s3" {
-		storageBackend = testrig.NewS3Storage()
-	} else {
-		storageBackend = testrig.NewInMemoryStorage()
+	if err := tracing.Initialize(); err != nil {
+		return fmt.Errorf("error initializing tracing: %w", err)
 	}
-	testrig.StandardStorageSetup(storageBackend, "./testrig/media")
 
-	// Create client API and federator worker pools
-	clientWorker := concurrency.NewWorkerPool[messages.FromClientAPI](-1, -1)
-	fedWorker := concurrency.NewWorkerPool[messages.FromFederator](-1, -1)
+	// Initialize caches and database
+	state.DB = testrig.NewTestDB(&state)
+
+	// New test db inits caches so we don't need to do
+	// that twice, we can just start the initialized caches.
+	state.Caches.Start()
+	defer state.Caches.Stop()
+
+	testrig.StandardDBSetup(state.DB, nil)
+
+	if os.Getenv("GTS_STORAGE_BACKEND") == "s3" {
+		state.Storage, _ = storage.NewS3Storage()
+	} else {
+		state.Storage = testrig.NewInMemoryStorage()
+	}
+	testrig.StandardStorageSetup(state.Storage, "./testrig/media")
+
+	// Initialize workers.
+	state.Workers.Start()
+	defer state.Workers.Stop()
 
 	// build backend handlers
-	oauthServer := testrig.NewTestOauthServer(dbService)
-	transportController := testrig.NewTestTransportController(testrig.NewMockHTTPClient(func(req *http.Request) (*http.Response, error) {
+	transportController := testrig.NewTestTransportController(&state, testrig.NewMockHTTPClient(func(req *http.Request) (*http.Response, error) {
 		r := io.NopCloser(bytes.NewReader([]byte{}))
 		return &http.Response{
 			StatusCode: 200,
 			Body:       r,
 		}, nil
-	}, ""), dbService, fedWorker)
-	mediaManager := testrig.NewTestMediaManager(dbService, storageBackend)
-	federator := testrig.NewTestFederator(dbService, transportController, storageBackend, mediaManager, fedWorker)
+	}, ""))
+	mediaManager := testrig.NewTestMediaManager(&state)
+	federator := testrig.NewTestFederator(&state, transportController, mediaManager)
 
 	emailSender := testrig.NewEmailSender("./web/template/", nil)
+	typeConverter := testrig.NewTestTypeConverter(state.DB)
+	filter := visibility.NewFilter(&state)
 
-	processor := testrig.NewTestProcessor(dbService, storageBackend, federator, emailSender, mediaManager, clientWorker, fedWorker)
-	if err := processor.Start(); err != nil {
-		return fmt.Errorf("error starting processor: %s", err)
+	// Initialize timelines.
+	state.Timelines.Home = timeline.NewManager(
+		tlprocessor.HomeTimelineGrab(&state),
+		tlprocessor.HomeTimelineFilter(&state, filter),
+		tlprocessor.HomeTimelineStatusPrepare(&state, typeConverter),
+		tlprocessor.SkipInsert(),
+	)
+	if err := state.Timelines.Home.Start(); err != nil {
+		return fmt.Errorf("error starting home timeline: %s", err)
 	}
 
-	idp, err := oidc.NewIDP(ctx)
-	if err != nil {
-		return fmt.Errorf("error creating oidc idp: %s", err)
+	state.Timelines.List = timeline.NewManager(
+		tlprocessor.ListTimelineGrab(&state),
+		tlprocessor.ListTimelineFilter(&state, filter),
+		tlprocessor.ListTimelineStatusPrepare(&state, typeConverter),
+		tlprocessor.SkipInsert(),
+	)
+	if err := state.Timelines.List.Start(); err != nil {
+		return fmt.Errorf("error starting list timeline: %s", err)
 	}
 
-	// build web module
-	webModule := web.New(processor)
+	processor := testrig.NewTestProcessor(&state, federator, emailSender, mediaManager)
 
-	// build client api modules
-	authModule := auth.New(dbService, idp, processor)
-	accountModule := account.New(processor)
-	instanceModule := instance.New(processor)
-	appsModule := app.New(processor)
-	followRequestsModule := followrequest.New(processor)
-	webfingerModule := webfinger.New(processor)
-	nodeInfoModule := nodeinfo.New(processor)
-	usersModule := user.New(processor)
-	timelineModule := timeline.New(processor)
-	notificationModule := notification.New(processor)
-	searchModule := search.New(processor)
-	filtersModule := filter.New(processor)
-	emojiModule := emoji.New(processor)
-	listsModule := list.New(processor)
-	mm := mediaModule.New(processor)
-	fileServerModule := fileserver.New(processor)
-	adminModule := admin.New(processor)
-	statusModule := status.New(processor)
-	securityModule := security.New(dbService, oauthServer)
-	streamingModule := streaming.New(processor)
-	favouritesModule := favourites.New(processor)
-	blocksModule := blocks.New(processor)
-	userClientModule := userClient.New(processor)
+	/*
+		HTTP router initialization
+	*/
 
-	apis := []api.ClientModule{
-		// modules with middleware go first
-		securityModule,
-		authModule,
-
-		// now the web module
-		webModule,
-
-		// now everything else
-		accountModule,
-		instanceModule,
-		appsModule,
-		followRequestsModule,
-		mm,
-		fileServerModule,
-		adminModule,
-		statusModule,
-		webfingerModule,
-		nodeInfoModule,
-		usersModule,
-		timelineModule,
-		notificationModule,
-		searchModule,
-		filtersModule,
-		emojiModule,
-		listsModule,
-		streamingModule,
-		favouritesModule,
-		blocksModule,
-		userClientModule,
+	router := testrig.NewTestRouter(state.DB)
+	middlewares := []gin.HandlerFunc{
+		middleware.AddRequestID(config.GetRequestIDHeader()), // requestID middleware must run before tracing
 	}
+	if config.GetTracingEnabled() {
+		middlewares = append(middlewares, tracing.InstrumentGin())
+	}
+	middlewares = append(middlewares, []gin.HandlerFunc{
+		middleware.Logger(config.GetLogClientIP()),
+		middleware.UserAgent(),
+		middleware.CORS(),
+		middleware.ExtraHeaders(),
+	}...)
 
-	for _, m := range apis {
-		if err := m.Route(router); err != nil {
-			return fmt.Errorf("routing error: %s", err)
+	// attach global middlewares which are used for every request
+	router.AttachGlobalMiddleware(middlewares...)
+
+	// attach global no route / 404 handler to the router
+	router.AttachNoRouteHandler(func(c *gin.Context) {
+		apiutil.ErrorHandler(c, gtserror.NewErrorNotFound(errors.New(http.StatusText(http.StatusNotFound))), processor.InstanceGetV1)
+	})
+
+	// build router modules
+	var idp oidc.IDP
+	var err error
+	if config.GetOIDCEnabled() {
+		idp, err = oidc.NewIDP(ctx)
+		if err != nil {
+			return fmt.Errorf("error creating oidc idp: %w", err)
 		}
 	}
 
-	gts, err := gotosocial.NewServer(dbService, router, federator, mediaManager)
+	routerSession, err := state.DB.GetSession(ctx)
+	if err != nil {
+		return fmt.Errorf("error retrieving router session for session middleware: %w", err)
+	}
+
+	sessionName, err := middleware.SessionName()
+	if err != nil {
+		return fmt.Errorf("error generating session name for session middleware: %w", err)
+	}
+
+	var (
+		authModule        = api.NewAuth(state.DB, processor, idp, routerSession, sessionName) // auth/oauth paths
+		clientModule      = api.NewClient(state.DB, processor)                                // api client endpoints
+		fileserverModule  = api.NewFileserver(processor)                                      // fileserver endpoints
+		wellKnownModule   = api.NewWellKnown(processor)                                       // .well-known endpoints
+		nodeInfoModule    = api.NewNodeInfo(processor)                                        // nodeinfo endpoint
+		activityPubModule = api.NewActivityPub(state.DB, processor)                           // ActivityPub endpoints
+		webModule         = web.New(state.DB, processor)                                      // web pages + user profiles + settings panels etc
+	)
+
+	// these should be routed in order
+	authModule.Route(router)
+	clientModule.Route(router)
+	fileserverModule.Route(router)
+	wellKnownModule.Route(router)
+	nodeInfoModule.Route(router)
+	activityPubModule.Route(router)
+	activityPubModule.RoutePublicKey(router)
+	webModule.Route(router)
+
+	gts, err := gotosocial.NewServer(state.DB, router, federator, mediaManager)
 	if err != nil {
 		return fmt.Errorf("error creating gotosocial service: %s", err)
 	}
@@ -186,16 +197,16 @@ var Start action.GTSAction = func(ctx context.Context) error {
 	sigs := make(chan os.Signal, 1)
 	signal.Notify(sigs, os.Interrupt, syscall.SIGTERM)
 	sig := <-sigs
-	log.Infof("received signal %s, shutting down", sig)
+	log.Infof(ctx, "received signal %s, shutting down", sig)
 
-	testrig.StandardDBTeardown(dbService)
-	testrig.StandardStorageTeardown(storageBackend)
+	testrig.StandardDBTeardown(state.DB)
+	testrig.StandardStorageTeardown(state.Storage)
 
 	// close down all running services in order
 	if err := gts.Stop(ctx); err != nil {
 		return fmt.Errorf("error closing gotosocial service: %s", err)
 	}
 
-	log.Info("done! exiting...")
+	log.Info(ctx, "done! exiting...")
 	return nil
 }

cmd/gotosocial/admin.go

@@ -1,26 +1,26 @@
-/*
-   GoToSocial
-   Copyright (C) 2021-2022 GoToSocial Authors admin@gotosocial.org
-
-   This program is free software: you can redistribute it and/or modify
-   it under the terms of the GNU Affero General Public License as published by
-   the Free Software Foundation, either version 3 of the License, or
-   (at your option) any later version.
-
-   This program is distributed in the hope that it will be useful,
-   but WITHOUT ANY WARRANTY; without even the implied warranty of
-   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-   GNU Affero General Public License for more details.
-
-   You should have received a copy of the GNU Affero General Public License
-   along with this program.  If not, see <http://www.gnu.org/licenses/>.
-*/
+// GoToSocial
+// Copyright (C) GoToSocial Authors admin@gotosocial.org
+// SPDX-License-Identifier: AGPL-3.0-or-later
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
 package main
 
 import (
 	"github.com/spf13/cobra"
 	"github.com/superseriousbusiness/gotosocial/cmd/gotosocial/action/admin/account"
+	"github.com/superseriousbusiness/gotosocial/cmd/gotosocial/action/admin/media"
 	"github.com/superseriousbusiness/gotosocial/cmd/gotosocial/action/admin/media/prune"
 	"github.com/superseriousbusiness/gotosocial/cmd/gotosocial/action/admin/trans"
 	"github.com/superseriousbusiness/gotosocial/internal/config"
@@ -55,6 +55,18 @@ func adminCommands() *cobra.Command {
 	config.AddAdminAccountCreate(adminAccountCreateCmd)
 	adminAccountCmd.AddCommand(adminAccountCreateCmd)
 
+	adminAccountListCmd := &cobra.Command{
+		Use:   "list",
+		Short: "list all existing local accounts",
+		PreRunE: func(cmd *cobra.Command, args []string) error {
+			return preRun(preRunArgs{cmd: cmd})
+		},
+		RunE: func(cmd *cobra.Command, args []string) error {
+			return run(cmd.Context(), account.List)
+		},
+	}
+	adminAccountCmd.AddCommand(adminAccountListCmd)
+
 	adminAccountConfirmCmd := &cobra.Command{
 		Use:   "confirm",
 		Short: "confirm an existing local account manually, thereby skipping email confirmation",
@@ -159,17 +171,44 @@ func adminCommands() *cobra.Command {
 
 	adminMediaCmd := &cobra.Command{
 		Use:   "media",
-		Short: "admin commands related stored media attachments/emojis",
+		Short: "admin commands related to stored media / emojis",
 	}
 
+	/*
+		ADMIN MEDIA LIST COMMANDS
+	*/
+
+	adminMediaListLocalCmd := &cobra.Command{
+		Use:   "list-local",
+		Short: "admin command to list media on local storage",
+		PreRunE: func(cmd *cobra.Command, args []string) error {
+			return preRun(preRunArgs{cmd: cmd})
+		},
+		RunE: func(cmd *cobra.Command, args []string) error {
+			return run(cmd.Context(), media.ListLocal)
+		},
+	}
+
+	adminMediaListRemoteCmd := &cobra.Command{
+		Use:   "list-remote",
+		Short: "admin command to list remote media cached on this instance",
+		PreRunE: func(cmd *cobra.Command, args []string) error {
+			return preRun(preRunArgs{cmd: cmd})
+		},
+		RunE: func(cmd *cobra.Command, args []string) error {
+			return run(cmd.Context(), media.ListRemote)
+		},
+	}
+
+	adminMediaCmd.AddCommand(adminMediaListLocalCmd, adminMediaListRemoteCmd)
+
 	/*
 		ADMIN MEDIA PRUNE COMMANDS
 	*/
 	adminMediaPruneCmd := &cobra.Command{
 		Use:   "prune",
-		Short: "admin commands for pruning unused/orphaned media from storage",
+		Short: "admin commands for pruning media from storage",
 	}
-	config.AddAdminMediaPrune(adminMediaPruneCmd)
 
 	adminMediaPruneOrphanedCmd := &cobra.Command{
 		Use:   "orphaned",
@@ -184,6 +223,32 @@ func adminCommands() *cobra.Command {
 	config.AddAdminMediaPrune(adminMediaPruneOrphanedCmd)
 	adminMediaPruneCmd.AddCommand(adminMediaPruneOrphanedCmd)
 
+	adminMediaPruneRemoteCmd := &cobra.Command{
+		Use:   "remote",
+		Short: "prune unused / stale media from storage, older than given number of days",
+		PreRunE: func(cmd *cobra.Command, args []string) error {
+			return preRun(preRunArgs{cmd: cmd})
+		},
+		RunE: func(cmd *cobra.Command, args []string) error {
+			return run(cmd.Context(), prune.Remote)
+		},
+	}
+	config.AddAdminMediaPrune(adminMediaPruneRemoteCmd)
+	adminMediaPruneCmd.AddCommand(adminMediaPruneRemoteCmd)
+
+	adminMediaPruneAllCmd := &cobra.Command{
+		Use:   "all",
+		Short: "perform all media and emoji prune / cleaning commands",
+		PreRunE: func(cmd *cobra.Command, args []string) error {
+			return preRun(preRunArgs{cmd: cmd})
+		},
+		RunE: func(cmd *cobra.Command, args []string) error {
+			return run(cmd.Context(), prune.All)
+		},
+	}
+	config.AddAdminMediaPrune(adminMediaPruneAllCmd)
+	adminMediaPruneCmd.AddCommand(adminMediaPruneAllCmd)
+
 	adminMediaCmd.AddCommand(adminMediaPruneCmd)
 
 	adminCmd.AddCommand(adminMediaCmd)

cmd/gotosocial/common.go

@@ -1,20 +1,19 @@
-/*
-   GoToSocial
-   Copyright (C) 2021-2022 GoToSocial Authors admin@gotosocial.org
-
-   This program is free software: you can redistribute it and/or modify
-   it under the terms of the GNU Affero General Public License as published by
-   the Free Software Foundation, either version 3 of the License, or
-   (at your option) any later version.
-
-   This program is distributed in the hope that it will be useful,
-   but WITHOUT ANY WARRANTY; without even the implied warranty of
-   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-   GNU Affero General Public License for more details.
-
-   You should have received a copy of the GNU Affero General Public License
-   along with this program.  If not, see <http://www.gnu.org/licenses/>.
-*/
+// GoToSocial
+// Copyright (C) GoToSocial Authors admin@gotosocial.org
+// SPDX-License-Identifier: AGPL-3.0-or-later
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
 package main
 

cmd/gotosocial/debug.go

@@ -1,20 +1,19 @@
-/*
-   GoToSocial
-   Copyright (C) 2021-2022 GoToSocial Authors admin@gotosocial.org
-
-   This program is free software: you can redistribute it and/or modify
-   it under the terms of the GNU Affero General Public License as published by
-   the Free Software Foundation, either version 3 of the License, or
-   (at your option) any later version.
-
-   This program is distributed in the hope that it will be useful,
-   but WITHOUT ANY WARRANTY; without even the implied warranty of
-   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-   GNU Affero General Public License for more details.
-
-   You should have received a copy of the GNU Affero General Public License
-   along with this program.  If not, see <http://www.gnu.org/licenses/>.
-*/
+// GoToSocial
+// Copyright (C) GoToSocial Authors admin@gotosocial.org
+// SPDX-License-Identifier: AGPL-3.0-or-later
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
 package main
 

cmd/gotosocial/main.go

@@ -1,28 +1,29 @@
-/*
-   GoToSocial
-   Copyright (C) 2021-2022 GoToSocial Authors admin@gotosocial.org
-
-   This program is free software: you can redistribute it and/or modify
-   it under the terms of the GNU Affero General Public License as published by
-   the Free Software Foundation, either version 3 of the License, or
-   (at your option) any later version.
-
-   This program is distributed in the hope that it will be useful,
-   but WITHOUT ANY WARRANTY; without even the implied warranty of
-   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-   GNU Affero General Public License for more details.
-
-   You should have received a copy of the GNU Affero General Public License
-   along with this program.  If not, see <http://www.gnu.org/licenses/>.
-*/
+// GoToSocial
+// Copyright (C) GoToSocial Authors admin@gotosocial.org
+// SPDX-License-Identifier: AGPL-3.0-or-later
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
 package main
 
 import (
 	"log"
-	"runtime/debug"
+	"os"
+	godebug "runtime/debug"
 	"strings"
 
+	"codeberg.org/gruf/go-debug"
 	"github.com/spf13/cobra"
 
 	_ "github.com/superseriousbusiness/gotosocial/docs"
@@ -60,9 +61,14 @@ func main() {
 
 	// add subcommands
 	rootCmd.AddCommand(serverCommands())
-	rootCmd.AddCommand(testrigCommands())
 	rootCmd.AddCommand(debugCommands())
 	rootCmd.AddCommand(adminCommands())
+	if debug.DEBUG {
+		// only add testrig if debug enabled.
+		rootCmd.AddCommand(testrigCommands())
+	} else if len(os.Args) > 1 && os.Args[1] == "testrig" {
+		log.Fatalln("gotosocial must be built and run with the DEBUG enviroment variable set to enable and access testrig")
+	}
 
 	// run
 	if err := rootCmd.Execute(); err != nil {
@@ -73,7 +79,7 @@ func main() {
 // version will build a version string from binary's stored build information.
 func version() string {
 	// Read build information from binary
-	build, ok := debug.ReadBuildInfo()
+	build, ok := godebug.ReadBuildInfo()
 	if !ok {
 		return ""
 	}

cmd/gotosocial/server.go

@@ -1,20 +1,19 @@
-/*
-   GoToSocial
-   Copyright (C) 2021-2022 GoToSocial Authors admin@gotosocial.org
-
-   This program is free software: you can redistribute it and/or modify
-   it under the terms of the GNU Affero General Public License as published by
-   the Free Software Foundation, either version 3 of the License, or
-   (at your option) any later version.
-
-   This program is distributed in the hope that it will be useful,
-   but WITHOUT ANY WARRANTY; without even the implied warranty of
-   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-   GNU Affero General Public License for more details.
-
-   You should have received a copy of the GNU Affero General Public License
-   along with this program.  If not, see <http://www.gnu.org/licenses/>.
-*/
+// GoToSocial
+// Copyright (C) GoToSocial Authors admin@gotosocial.org
+// SPDX-License-Identifier: AGPL-3.0-or-later
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
 package main
 

cmd/gotosocial/testrig.go

@@ -1,20 +1,19 @@
-/*
-   GoToSocial
-   Copyright (C) 2021-2022 GoToSocial Authors admin@gotosocial.org
-
-   This program is free software: you can redistribute it and/or modify
-   it under the terms of the GNU Affero General Public License as published by
-   the Free Software Foundation, either version 3 of the License, or
-   (at your option) any later version.
-
-   This program is distributed in the hope that it will be useful,
-   but WITHOUT ANY WARRANTY; without even the implied warranty of
-   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-   GNU Affero General Public License for more details.
-
-   You should have received a copy of the GNU Affero General Public License
-   along with this program.  If not, see <http://www.gnu.org/licenses/>.
-*/
+// GoToSocial
+// Copyright (C) GoToSocial Authors admin@gotosocial.org
+// SPDX-License-Identifier: AGPL-3.0-or-later
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
 package main
 

docs/admin/admin_panel.md

@@ -1,26 +0,0 @@
-# Admin Control Panel
-
-The GoToSocial admin panel is a simple webclient that uses the [admin api routes](https://docs.gotosocial.org/en/latest/api/swagger/#operations-tag-admin) to manage your instance. It uses the same OAUTH mechanism as normal clients (with scope: admin), and as such can be hosted anywhere, separately from your instance, or run locally. A public installation is available here: [https://gts.superseriousbusiness.org/admin](https://gts.superseriousbusiness.org/admin).
-
-## Using the panel
-To use the Admin API your account has to be promoted as such:
-```
-./gotosocial --config-path ./config.yaml admin account promote --username YOUR_USERNAME
-```
-After this, you can enter your instance domain in the login field (auto-filled if you run GoToSocial on the same domain), and login like you would with
-any other client.
-
-<p align="middle">
-	<img src="../../assets/admin-panel.png">Screenshot of the GoToSocial admin panel, showing the fields to change an instance's settings</img>
-</p>
-
-You can change the instance's settings like the title and descriptions, and add/remove/change domain blocks including a bulk import/export.
-
-## Building the panel
-Build requirements: some version of [Node.js](https://nodejs.org) and yarn.
-```
-yarn install --cwd web/source
-BUDO_BUILD=1 node web/source 
-```
-
-See also: [Contributing.md Stylesheet / Web dev](https://github.com/superseriousbusiness/gotosocial/blob/main/CONTRIBUTING.md#stylesheet--web-dev)

docs/admin/backup_and_restore.md

@@ -1,18 +1,37 @@
 # Backup and Restore
 
-In certain conditions, it may be desirable to be able to back up a GoToSocial instance, and then to restore it again later, or just save the backup somewhere.
+As the GoToSocial database contains the instance as well as all user signing keys it is vital to back it up. If you lose these keys you'll never be able to federate from this domain again. Don't forget to also encrypt your backups in order to keep the data safe at rest.
 
-Some potential scenarios:
+Aside from disaster recovery, there are other good reasons to keep backups. Some potential scenarios for you to consider:
 
 * You want to close down your instance but you might create it again later and you don't want to break federation.
 * You need to migrate to a different database for some reason (Postgres => SQLite or vice versa).
-* You want to keep regular backups of your data just in case something happens.
-* You want to migrate from GoToSocial to a different Fediverse server, or from a different Fediverse server to GoToSocial.
 * You're about to hack around on your instance and you want to make a quick backup so you don't lose everything if you mess up.
 
-There are a few different ways of doing this, most of which require some technical knowledge.
+## What to backup
 
-## Image your disk
+### Database
+
+Most backup tools have built-in support for common databases like PostgreSQL and SQLite. Ensure you review their documentation first as they often spell out certain considerations and conditions that need to be met for backups to complete and restore successfully.
+
+### Media
+
+Local media should be backed up. You can use the [GoToSocial CLI](cli.md#gotosocial-admin-media-list-local) to list all media files that belong to your instance and its users.
+
+Remote media does not have to be backed up. This can be a good way to keep the size of your backups down. Remote media will be fetched from the origin instance, much like how it'll be fetched again if it got pruned due to media retention.
+
+## How to backup
+
+You can go about this a few different ways:
+
+* Imaging the VMs/machines your instance and database runs on
+* Dumping GoToSocial's state with the CLI
+* Backing up database and media files
+* Backup software
+
+Though setting up backup software can be a bit more work, it's by far the best option. It ensures consistent and encrypted backups and can protect you against filesystem corruption in a way that taking disk snapshots and copying the raw database and media files won't.
+
+### Image your disk
 
 If you're running GoToSocial on a VPS (a remote machine in the cloud), arguably the easiest way to preserve all of your database entries and media is to image the disk attached to the VPS. This will preserve the whole disk. Many VPS providers offer the option of automatically creating backups on a timer, so you'll always be able to restore if your data is lost.
 
@@ -28,24 +47,7 @@ Disadvantages:
 * Will probably also preserve stuff you don't need, from other programs running on the same machine.
 * Vendor lock-in, difficult to move the data around.
 
-## Back up your database files
-
-Regardless of whether you're using Postgres or SQLite as your GoToSocial database, it's possible to simply back up the database files directly by using something like [rclone](https://rclone.org/), or following best practices for [backing up Postgres data](https://www.postgresql.org/docs/9.1/backup.html) or [SQLite data](https://sqlite.org/backup.html).
-
-Advantages:
-
-* Backups are relatively portable - you can move data from one machine to another.
-* Well-documented procedure with a lot of guides and tooling available.
-* Lots of different ways of doing your backups, depending on what you need.
-
-Disadvantages:
-
-* Can be a bit fiddly to set up initially.
-* You need to figure out where to keep your backups.
-* Restoring from backups can be a pain.
-* Unless you back up media as well, references to media attachments in your db will be broken.
-
-## Use the GoToSocial CLI
+### Use the GoToSocial CLI
 
 The GoToSocial CLI tool also provides commands for backing up and restoring data from your instance, which will preserve the *bare-minimum* necessary data to backup and restore your instance, without breaking federation with other instances.
 
@@ -88,7 +90,7 @@ The backup file produced will be in the form of a line-separated series of JSON
 {"type":"instance","id":"01BZDDRPAB8J645ABY31HHF68Y","createdAt":"2021-09-08T10:00:54.763912Z","domain":"localhost:8080","title":"localhost:8080","uri":"http://localhost:8080","reputation":0}
 ```
 
-For information on how to use the commands to import/export, see [here](cli.md#gotosocial-admin-export).
+For information on how to use the commands to import/export, see [here](cli.md#gotosocial-admin-export). Though the `export` command won't backup media, you can use the [`media list-local`](cli.md#gotosocial-admin-media-list-local) command to figure out which media files you should keep.
 
 Advantages:
 
@@ -98,5 +100,123 @@ Advantages:
 
 Disadvantages:
 
-* Loss of statuses/media/etc: don't do a backup/restore this way unless you're willing to drop stuff.
+* Loss of statuses/faves/etc: don't do a backup/restore this way unless you're willing to drop stuff.
 * You need to use the GtS CLI tool to insert data back into a database, unless you write custom tooling for it.
+
+
+### Back up your database files and media
+
+Regardless of whether you're using PostgreSQL or SQLite as your GoToSocial database, it's possible to simply back up the database files directly by using something like [rclone](https://rclone.org/), or following best practices for [backing up Postgres data](https://www.postgresql.org/docs/15/backup.html) or [SQLite data](https://sqlite.org/backup.html).
+
+Use the [GoToSocial CLI](cli.md#gotosocial-admin-media-list-local) to get a list of media files you need to safeguard.
+
+Advantages:
+
+* Backups are relatively portable - you can move data from one machine to another.
+* Well-documented procedure with a lot of guides and tooling available.
+* Lots of different ways of doing your backups, depending on what you need.
+
+Disadvantages:
+
+* Can be a bit fiddly to set up initially.
+* You need to figure out where to keep your backups.
+* Restoring from backups can be a pain.
+* Unless you back up media as well, references to media attachments in your db will be broken.
+
+### Backup software
+
+Backup software is created with the specific purpose of helping you create, manage and restore your backups. It typically knows how to safely backup your database so you don't have to be an expert on how to do PostgreSQL or SQLite backups. It can backup from the filesystem too.
+
+Though the same advantages and disadvantages roughly apply as with backing up the database files directly, this approach does have some nice extras:
+
+* Backups are highly portable and can be used to restore the database from 0
+* Backups happen on a regular schedule and with configurable retention policies
+* Backups are incremental and compressed to save on storage and bandwidth
+* Backups are encrypted
+* Built-in tooling to list your snapshots and restore from them
+
+!!! tip
+    [Rsync.net](https://rsync.net/), [BorgBase](https://www.borgbase.com/) and [Hetzner Storage](https://www.hetzner.com/storage/storage-box) provide affordable storage that you can use as a backup target. Rsync.net has a special Borg-only backup product that is much cheaper than their regular storage product. If you only want to use them for backups managed with Borg, [sign up here instead](https://www.rsync.net/products/borg.html).
+
+#### Borgmatic
+
+[Borgmatic](https://torsion.org/borgmatic/) is a utility to help perform backups using [Borg](https://www.borgbackup.org/). It's driven by a declarative configuration file using YAML. BorgBase, Rsync.net and Hetzner all support Borg.
+
+!!! warning
+    When initialising the Borg repository, ensure you set it up with a strong encryption key and store that key somewhere safely. Without it you won't be able to decrypt your backups in the future. The ArchWiki entry on Borgmatic explains how to safely pass your encryption key to Borgmatic without storing it plain text in its configuration file.
+
+How to backup databases with Borgmatic has its own [documentation page](https://torsion.org/borgmatic/docs/how-to/backup-your-databases/) that you should review. A simple `config.yaml` for Borgmatic with GoToSocial using SQLite looks like this:
+
+```yaml
+location:
+  repositories:
+    - path: ssh://<find it in your provider control panel>
+      label: <anything but typically the provider, for example borgbase>
+  patterns_from:
+  - /etc/borgmatic/gotosocial_patterns
+
+storage:
+  compression: auto,zstd
+  archive_name_format: '{hostname}-{now:%Y-%m-%d-%H%M%S}'
+  retries: 5
+  retry_wait: 30
+
+retention:
+    keep_daily: 7
+    keep_weekly: 6
+    keep_monthly: 12
+
+hooks:
+  before_backup:
+      - /usr/bin/systemctl stop gotosocial
+  after_backup:
+      - /usr/bin/systemctl start gotosocial
+  sqlite_databases:
+    - name: gotosocial
+      path: /path/to/sqlite.db
+```
+
+For PostgreSQL, you'll want to use `postgresql_databases` instead.
+
+The file mentioned in `patterns_from` can be created by transforming the output from the [GoToSocial CLI](cli.md#gotosocial-admin-media-list-local). In order to generate the right patterns you can use the [`media-to-borg-patterns.py`](https://github.com/superseriousbusiness/gotosocial/tree/main/example/borgmatic/media-to-borg-patterns.py) script. How Borg patterns work is explained in [their documentation](https://man.archlinux.org/man/borg-patterns.1).
+
+You'll need to put that file on your GoToSocial instance and make sure the file is executable. It requires Python 3 which you will already have if you have Borg and Borgmatic installed. It only depends on the Python standard library.
+
+!!! note
+    For this to work reliably, you should ensure that the [storage-local-base-path](../configuration/storage.md) in your GoToSocial configuration uses an absolute path. Otherwise you'll have to tweak the paths yourself.
+
+```sh
+$ gotosocial admin media list-local | \
+    /path/to/media-to-borg-patterns.py \
+    <storage-local-base-path>
+```
+
+This will output a pattern set looking roughly like this to your console:
+
+```
+R <storage-local-base-path>
++ pp:<storage-local-base-path>/<account ID>
+- <storage-local-base-path>/*
+```
+
+!!! tip
+    You can view the help by passing `--help` to `media-to-borg-patterns.py`. It can write the output to a file directly by passing the location of a file as the last argument to the script.
+
+Given this set of patterns, Borg will start looking for files starting from `<storage-local-base-path>`. Anything that matches the path prefix, `pp:` will be included. Everything else will match the last pattern, excluding it from the archive.
+
+On a single-user instance, you can run this command once and inline the patterns directly in your Borgmatic configuration [using the `patterns` key](https://torsion.org/borgmatic/docs/reference/configuration/). On multi-user instances you should run this after a user signs up. Alternatively, you can run it every time before you do a backup.
+
+If you're running Borgmatic as a systemd service, you can [create a drop-in](https://wiki.archlinux.org/title/systemd#Drop-in_files) for `borgmatic.service` and run the pattern generation before the backup is started with:
+
+```ini
+[Service]
+ExecStartPre=/path/to/gotosocial admin media list-local | /path/to/media-to-borg-patterns.py <storage-local-base-path> /etc/borgmatic/gotosocial_patterns
+```
+
+Documentation that's good to review:
+
+* Borgmatic [configuration reference](https://torsion.org/borgmatic/docs/reference/configuration/)
+* ArchWiki entry [on Borgmatic](https://wiki.archlinux.org/title/Borgmatic)
+* ArchWiki entry [on Borg](https://wiki.archlinux.org/title/Borg_backup)
+* BorgBase [documentation](https://docs.borgbase.com/)
+* Hetzner community guide on [setting up Borgmatic](https://community.hetzner.com/tutorials/install-and-configure-borgmatic)

docs/admin/cli.md

@@ -179,7 +179,7 @@ Flags:
 Example:
 
 ```bash
-gotosocial admin account password --username some_username --pasword some_really_good_password --config-path config.yaml
+gotosocial admin account password --username some_username --password some_really_good_password --config-path config.yaml
 ```
 
 ### gotosocial admin export
@@ -255,6 +255,18 @@ Example:
 gotosocial admin import --path example.json --config-path config.yaml
 ```
 
+### gotosocial admin media list-local
+
+This command can be used to list local media. Local media is media that belongs to posts by users with an account on the instance.
+
+The output will be a list of files. The list can be used to drive your backups.
+
+### gotosocial admin media list-remote
+
+This is the corollary to list-local, but instead lists media from remote instances. Remote media belongs to other instances, but was attached to a post we received over federation and have potentially cached locally.
+
+The output will be a list of URLs to retrieve the original content from. GoToSocial automatically retrieves remote media when it needs it, so you should never need to do so yourself.
+
 ### gotosocial admin media prune orphaned
 
 This command can be used to prune orphaned media from your GoToSocial.
@@ -287,3 +299,40 @@ Example (for real):
 ```bash
 gotosocial admin media prune orphaned --dry-run=false
 ```
+
+### gotosocial admin media prune remote
+
+This command can be used to prune unused/stale remote media from your GoToSocial.
+
+Stale media means avatars/headers/status attachments from remote instances that are older than `media-remote-cache-days`.
+
+These items will be refetched later on demand, if necessary.
+
+Unused media means avatars/headers/status attachments which are not currently in use by an account or status.
+
+**This command only works when GoToSocial is not running, since it acquires an exclusive lock on storage. Stop GoToSocial first before running this command!**
+
+```text
+prune unused/stale remote media from storage, older than given number of days
+
+Usage:
+  gotosocial admin media prune remote [flags]
+
+Flags:
+      --dry-run   perform a dry run and only log number of items eligible for pruning (default true)
+  -h, --help      help for remote
+```
+
+By default, this command performs a dry run, which will log how many items can be pruned. To do it for real, add `--dry-run=false` to the command.
+
+Example (dry run):
+
+```bash
+gotosocial admin media prune remote
+```
+
+Example (for real):
+
+```bash
+gotosocial admin media prune remote --dry-run=false
+```

docs/admin/settings.md

@@ -0,0 +1,57 @@
+# Admin Settings
+
+The GoToSocial Settings interface uses the [admin api routes](https://docs.gotosocial.org/en/latest/api/swagger/#operations-tag-admin) to manage your instance. It's combined with the [User settings](../user_guide/settings.md) and uses the same OAUTH mechanism as normal clients (with scope: admin).
+
+## Account permissions
+To use the Admin API your account has to be promoted as such:
+```
+./gotosocial --config-path ./config.yaml admin account promote --username YOUR_USERNAME
+```
+
+After this, you can enter your instance domain in the login field (auto-filled if you run GoToSocial on the same domain), and login like you would with any other client.
+
+
+## Instance Settings
+![Screenshot of the GoToSocial admin panel, showing the fields to change an instance's settings](../assets/admin-settings.png)
+
+Here you can set various metadata for your instance, like the displayed name, thumbnail image, description texts (HTML), and contact username and email.
+
+## Actions
+You can use media cleanup to remove remote media older than the specified number of days. This also removes unused headers and avatars.
+
+## Federation
+![List of suspended instances, with a field to filter/add new blocks. Below is a link to the bulk import/export interface](../assets/admin-settings-federation.png)
+
+In the federation section you can influence which instances you federate with, through adding domain blocks. You can enter a domain to suspend in the search field, which will filter the list to show you if you already have a block for it. Clicking 'suspend' gives you a form to add a public and/or private comment, and submit to add the block. Adding a suspension will suspend all the currently known accounts on the instance, and prevent any new interactions with any user on the blocked instance.
+
+### Bulk import/export
+Through the link at the bottom of the Federation section (or going to `/settings/admin/federation/import-export`) you can do bulk import/export of your domain blocklist. 
+
+![List of domains included in an import, providing ways to select some or all of them, change their domains, and update the use of subdomains.](../assets/admin-settings-federation-import-export.png)
+
+Upon importing a list, either through the input field or from a file, you can review the entries in the list before importing a subset. You'll also be warned for entries that use subdomains, providing an easy way to change them to the main domain.
+
+## Reports
+![List of reports for testing, one resolved and one open.](../assets/admin-settings-reports.png)
+
+The reports section shows a list of reports, originating from your local users, or remote instances (shown anonymously as just the name of the instance, without specific username).
+
+Clicking a report shows if it was resolved (with the reasoning if available), more information, and a list of reported toots if selected by the reporting user.
+
+## Custom Emoji
+Custom Emoji will be automatically fetched when included in remote toots, but to use them in your own posts they have to be enabled on your instance.
+
+### Local
+![Local custom emoji section, showing an overview of custom emoji sorted by category. There are a lot of garfields.](../assets/admin-settings-emoji-local.png)
+
+This section shows an overview of all the custom emoji enabled on your instance, sorted by their category. Clicking an emoji shows it's details, and provides options to change the category or image, or delete it completely. The shortcode cannot be updated here, you would have to upload it with the new shortcode yourself (and optionally delete the old one).
+
+Below the overview you can upload your own custom emoji, after previewing how they look in a toot. PNG and (animated) GIF's are supported.
+
+### Remote
+![Remote custom emoji section, showing a list of 3 emoji parsed from the entered toot, garfield, blobfoxbox and blobhajmlem. They can be selected, their shortcode can be tweaked, and they can be assigned to a category, before submitting as a copy or delete operation](../assets/admin-settings-emoji-remote.png)
+
+Through the 'remote' section, you can look up a link to any remote toots (provided the instance isn't suspended). If they use any custom emoji they will be listed, providing an easy way to copy them to the local emoji (for use in your own toots), or disable them ( hiding them from toots).
+
+**Note:** as the testrig server does not federate, this feature can't be used in development (500: Internal Server Error).
+

docs/advanced/caching/api.md

@@ -0,0 +1,86 @@
+# Caching API responses
+
+It is possible to cache certain API responses to offload the GoToSocial process from having to handle all requests. We don't recommend caching responses to requests under `/api`.
+
+When using a [split domain](../host-account-domain.md) deployment style, you need to ensure you configure caching on the host domain. The account domain should only be issuing redirects to the host domain which clients will automatically remember.
+
+!!! warning "There are only two hard things in computer science"
+    Configuring caching incorrectly can result into all kinds of problems. Follow this guide carefully and thoroughly test your modifications. Don't cache endpoints that require authentication without taking the `Authorization` header into account.
+
+## Endpoints
+
+### Webfinger and hostmeta
+
+Requests to `/.well-known/webfinger` and `/.well-known/host-meta` can be safely cached. Do be careful to ensure any caching strategy takes query parameters into account when caching webfinger requests as requests to that endpoint are of the form `?resource=acct:@username@domain.tld`.
+
+### Public keys
+
+Many implementations will regularly request the public key for a user in order to validate the signature on a message they received. This will happen whenever a message gets federated amongst other things. These keys are long lived, essentially eternal, and can thus be cached with a long lifetime.
+
+## Configuration snippets
+
+### nginx
+
+For nginx, you'll need to start by configuring a cache zone. The cache zone must be created in the `http` section, not within `server` or `location`.
+
+```nginx
+http {
+    ...
+    proxy_cache_path /var/cache/nginx keys_zone=gotosocial_ap_public_responses:10m inactive=1w;
+}
+```
+
+This configures a cache of 10MB whose entries will be kept up to one week if they're not accessed.
+
+The zone is named `gotosocial_ap_public_responses` but you can name it whatever you want. 10MB is a lot of cache keys; you can probably use a smaller value on small instances.
+
+Second, we need to update our GoToSocial nginx configuration to actually use the cache for the endpoints we want to cache.
+
+```nginx
+server {
+  server_name social.example.org;
+  
+  location ~ /.well-known/(webfinger|host-meta)$ {
+    proxy_set_header Host $host;
+    proxy_set_header X-Forwarded-For $remote_addr;
+    proxy_set_header X-Forwarded-Proto $scheme;
+
+    proxy_cache gotosocial_ap_public_responses;
+    proxy_cache_background_update on;
+    proxy_cache_key $scheme://$host$uri$is_args$query_string;
+    proxy_cache_valid 200 10m;
+    proxy_cache_use_stale error timeout updating http_500 http_502 http_503 http_504 http_429;
+    proxy_cache_lock on;
+    add_header X-Cache-Status $upstream_cache_status;
+
+    proxy_pass http://localhost:8080;
+  }
+
+  location ~ ^\/users\/(?:[a-z0-9_\.]+)\/main-key$ {
+    proxy_set_header Host $host;
+    proxy_set_header X-Forwarded-For $remote_addr;
+    proxy_set_header X-Forwarded-Proto $scheme;
+
+    proxy_cache gotosocial_ap_public_responses;
+    proxy_cache_background_update on;
+    proxy_cache_key $scheme://$host$uri;
+    proxy_cache_valid 200 604800s;
+    proxy_cache_use_stale error timeout updating http_500 http_502 http_503 http_504 http_429;
+    proxy_cache_lock on;
+    add_header X-Cache-Status $upstream_cache_status;
+
+    proxy_pass http://localhost:8080;
+  }
+```
+
+The `proxy_pass` and `proxy_set_header` are mostly the same, but the `proxy_cache*` entries warrant some explanation:
+
+- `proxy_cache gotosocial_ap_public_responses` tells nginx to use the `gotosocial_ap_public_responses` cache zone we previously created. If you named it something else, you should change this value
+- `proxy_cache_background_update on` means nginx will try and refresh a cached resource that's about to expire in the background, to ensure it has a current copy on disk
+- `proxy_cache_key` is configured in such a way that it takes the query string into account for caching. So a request for `.well-known/webfinger?acct=user1@example.org` and `.well-known/webfinger?acct=user2@example.org` are not seen as the same.
+- `proxy_cache_valid 200 10m;` means we only cache 200 responses from GTS and for 10 minutes. You can add additional lines of these, like `proxy_cache_valid 404 1m;` to cache 404 responses for 1 minute
+- `proxy_cache_use_stale` tells nginx it's allowed to use a stale cache entry (so older than 10 minutes) in certain cases
+- `proxy_cache_lock on` means that if a resource is not cached and there's multiple concurrent requests for them, the queries will be queued up so that only one request goes through and the rest is then answered from cache
+- `add_header X-Cache-Status $upstream_cache_status` will add an `X-Cache-Status` header to the response so you can check if things are getting cached. You can remove this.
+
+The provided configuration will serve a stale response in case there's an error proxying to GoToSocial, if our connection to GoToSocial times out, if GoToSocial returns a `5xx` status code or if GoToSocial returns 429 (Too Many Requests). The `updating` value says that we're allowed to serve a stale entry if nginx is currently in the process of refreshing its cache. Because we configured `inactive=1w` in the `proxy_cache_path` directive, nginx may serve a response up to one week old if the conditions in `proxy_cache_use_stale` are met.

docs/advanced/caching/assets-media.md

@@ -0,0 +1,143 @@
+# Caching assets and media
+
+When you've configured your GoToSocial instance with local storage for media, you can use your [reverse proxy](../../getting_started/reverse_proxy/index.md) to serve these files directly and cache them. This avoids hitting GoToSocial for these requests and reverse proxies can typically serve assets faster than GoToSocial.
+
+You can also use your reverse proxy to cache the GoToSocial web UI assets, like the CSS and images it uses.
+
+When using a [split domain](../host-account-domain.md) deployment style, you need to ensure you configure caching of the assets and media on the host domain.
+
+!!! warning "Media pruning"
+    If you've configured media pruning, you need to ensure that when media is not found on disk the request is still sent on to GoToSocial. This will ensure the media is fetched again from the remote instance and subsequent requests for this media will then be handled by your reverse proxy again.
+
+## Endpoints
+
+There are 2 endpoints that serve assets we can serve and cache:
+
+* `/assets` which contains fonts, CSS, images etc. for the web UI
+* `/fileserver` which serves attachments for status posts when using the local storage backend
+
+The filesystem location of `/assets` is defined by the [`web-asset-base-dir`](../../configuration/web.md) configuration option. Files under `/fileserver` are retrieved from the [`storage-local-base-path`](../../configuration/storage.md).
+
+## Configuration
+
+### Apache 2.4
+
+This is intended to behave identical to the nginx section below.
+
+The `Cache-Control` header is manually set to merge the values
+from the configuration and the `expires` directive to avoid
+breakage from having two header lines. `Header set` defaults
+to ` onsuccess`, so it is also not added to error responses.
+
+Assuming your GtS installation is rooted in `/opt/GtS` with a
+`storage` subdirectory, and the webserver has been given access,
+add the following section to the vhost:
+
+```
+	<Directory /opt/GtS/web/assets>
+		Options None
+		AllowOverride None
+		Require all granted
+		ExpiresActive on
+		ExpiresDefault A300
+		Header set Cache-Control "public, max-age=300"
+	</Directory>
+	RewriteRule "^/assets/(.*)$" "/opt/GtS/web/assets/$1" [L]
+
+	<Directory /opt/GtS/storage>
+		Options None
+		AllowOverride None
+		Require all granted
+		ExpiresActive on
+		ExpiresDefault A604800
+		Header set Cache-Control "private, immutable, max-age=604800"
+	</Directory>
+	RewriteCond "/opt/GtS/storage/$1" -f
+	RewriteRule "^/fileserver/(.*)$" "/opt/GtS/storage/$1" [L]
+```
+
+The trick here is that, in an Apache 2-based reverse proxy setup…
+
+```
+	RewriteEngine On
+
+	RewriteCond %{HTTP:Upgrade} websocket [NC]
+	RewriteCond %{HTTP:Connection} upgrade [NC]
+	RewriteRule ^/?(.*) "ws://localhost:8980/$1" [P,L]
+
+	ProxyIOBufferSize 65536
+	ProxyTimeout 120
+
+	ProxyPreserveHost On
+	<Location "/">
+		ProxyPass http://127.0.0.1:8980/
+		ProxyPassReverse http://127.0.0.1:8980/
+	</Location>
+```
+
+… everything is proxied by default, the `RewriteRule` bypasses
+the proxy (by specifying a filesystem path to redirect to) for
+specific URL præficēs and the `RewriteCond` ensures to only
+disable the `/fileserver/` proxy if the file is, indeed, present.
+
+Also run the following commands (assuming a Debian-like setup)
+to enable the modules used:
+
+```
+$ sudo a2enmod expires
+$ sudo a2enmod headers
+$ sudo a2enmod rewrite
+```
+
+Then (after a configtest), restart Apache.
+
+### nginx
+
+Here's an example of the three location blocks you'll need to add to your existing configuration in nginx:
+
+```nginx
+server {
+  server_name social.example.org;
+
+  location /assets/ {
+    alias web-asset-base-dir/;
+    autoindex off;
+    expires 5m;
+    add_header Cache-Control "public";
+  }
+
+  location @fileserver {
+    proxy_pass http://localhost:8080;
+    proxy_set_header Host $host;
+    proxy_set_header Upgrade $http_upgrade;
+    proxy_set_header Connection "upgrade";
+    proxy_set_header X-Forwarded-For $remote_addr;
+    proxy_set_header X-Forwarded-Proto $scheme;
+  }
+
+  location /fileserver/ {
+    alias storage-local-base-path/;
+    autoindex off;
+    expires 1w;
+    add_header Cache-Control "private, immutable";
+    try_files $uri @fileserver;
+  }
+}
+```
+
+The `/fileserver` location is a bit special. When we fail to fetch the media from disk, we want to proxy the request on to GoToSocial so it can try and fetch it. The `try_files` directive can't take a `proxy_pass` itself so instead we created the named `@fileserver` location that we pass in last to `try_files`.
+
+!!! bug "Trailing slashes"
+    The trailing slashes in the `location` directives and the `alias` are significant, do not remove those.
+
+The `expires` directive adds the necessary headers to inform the client how long it may cache the resource:
+
+* For assets, which may change on each release, 5 minutes is used in this example
+* For attachments, which should never change once they're created, we currently use one week
+
+For other options, see the nginx documentation on the [`expires` directive](https://nginx.org/en/docs/http/ngx_http_headers_module.html#expires).
+
+Nginx does not add cache headers to 4xx or 5xx response codes so a failure to fetch an asset won't get cached by clients. The `autoindex off` directive tells nginx to not serve a directory listing. This should be the default but it doesn't hurt to be explicit. The added `add_header` lines set additional options for the `Cache-Control` header:
+
+* `public` is used to indicate that anyone may cache this resource
+* `immutable` is used to indicate this resource will never change while it is fresh (it's before the end of the expires) allowing clients to forgo conditional requests to revalidate the resource during that timespan.

docs/advanced/caching/index.md

@@ -0,0 +1,11 @@
+# Caching
+
+This section covers a number of different caching techniques that can be used to make GoToSocial more robust in the face of higher traffic and offload the GoToSocial instance from some work.
+
+!!! note
+    These guides are only relevant if you're running a [reverse proxy](../../getting_started/reverse_proxy/index.md).
+
+## Guides
+
+* [Caching API responses](api.md)
+* [Assets and media serving and caching](assets-media.md)

docs/advanced/certificates.md

@@ -0,0 +1,108 @@
+# Provisioning TLS certificates
+
+As explained in the [deployment considerations](../getting_started/index.md), federation requires the use of TLS as most instances refuse to federate over unencrypted transports.
+
+GoToSocial comes with built-in support for provisioning and renewing its own TLS certificates through Lets Encrypt. This guide looks at how you can provision your own certificates independently from GoToSocial. This can be useful if you want full control over how the certificates are provisioned, or because you're using a [reverse proxy](../getting_started/reverse_proxy/index.md) which is doing TLS termination.
+
+There are a few different ways you can get TLS certificates:
+
+* Buy them from a vendor, typically valid for 2 years
+* Get them from your cloud provider, validity depends on their product constraints
+* Get them from an [ACME](https://en.wikipedia.org/wiki/Automatic_Certificate_Management_Environment)-compatible provider like Lets Encrypt, typically valid for 3 months at a time
+
+In this guide we'll only look at option 3, an ACME-compatible vendor.
+
+## General approach
+
+The way you'll provision certificates through Lets Encrypt is:
+
+* Install an ACME client on your server
+* Configure the ACME client to provision your certificates
+* Configure a piece of software to use those certificates
+* Enable a timer/cron to regularly renew the certificates
+* Signal to the necessary applications they need to reload or restart to pick up the new certificates
+
+Certificates are provisioned [using a challenge](https://letsencrypt.org/sv/docs/challenge-types/), a way to verify that you're requesting a certificate for a domain you control. You'll typically use one of:
+
+* HTTP challenge
+* DNS challenge
+
+The HTTP challenge requires serving certain files on port 80 on the domains you're requesting a certificate for under the `/.well-known/acme/` path. This is the default challenge type.
+
+The DNS challenge happens entirely out of band but requires you to update a DNS TXT record. This approach is only feasible if your DNS registrar provides an API through which you can modify DNS records so that your ACME client can complete this challenge.
+
+## Clients
+
+The official Lets Encrypt client is [certbot](https://certbot.eff.org/) and it's usually packaged in [your (Linux) distribution](https://repology.org/project/certbot/versions) of choice. Certain reverse proxies like Caddy and Traefik have built-in support for provisioning certificates using the ACME protocol.
+
+A couple of other clients of note that you can consider using:
+
+* [acme-client](https://man.openbsd.org/acme-client.1) for OpenBSD using the privilege separation features of the platform
+* [lacme](https://git.guilhem.org/lacme/about/), which is built with process isolation and minimal privileges in mind in the same vein as acme-client but for Linux
+* [Lego](https://github.com/go-acme/lego), an ACME client and library written in Go
+* [mod_md](https://httpd.apache.org/docs/2.4/mod/mod_md.html), when using Apache 2.4.30+
+
+### DNS challenge
+
+For the DNS challenge, the API of your registrar needs to be supported by your ACME client. Though certbot has a few plugins for popular providers, you probably want to look at the [dns-multi](https://github.com/alexzorin/certbot-dns-multi) plugin instead. It leverages [Lego](https://github.com/go-acme/lego) under the hood which supports a much wider array of providers.
+
+## Configuration
+
+There are 3 configuration options that are important:
+
+* [`letsencrypt-enabled`](../configuration/tls.md) controls if GoToSocial will try to provision its own certificates
+* [`tls-certificate-chain`](../configuration/tls.md) filesystem path where GoToSocial can find the TLS certificate chain + the public key
+* [`tls-certificate-key`](../configuration/tls.md) filesystem path where GoToSocial can find the associated TLS private key
+
+### Without reverse proxy
+
+When running GoToSocial directly exposed to the internet, but you still want to use your own certificates you can set the following options:
+
+```yaml
+letsencrypt-enabled: false
+tls-certificate-chain: "/path/to/combined-certificate-chain-public.key"
+tls-certificate-key: "/path/to/private.key"
+```
+
+This disables the built-in provisioning of certificates through Lets Encrypt and tells GoToSocial where to find the certificates on disk.
+
+!!! tip
+    Restart GoToSocial after renewing your certificates. It won't pick up on certificate rotation by itself when they are provided like this.
+
+### With reverse proxy
+
+When running GoToSocial behind a [reverse proxy](../getting_started/reverse_proxy/index.md) which you also use for TLS termination, you'll need the following instead:
+
+```yaml
+letsencrypt-enabled: false
+tls-certificate-chain: ""
+tls-certificate-key: ""
+```
+
+It's important to ensure the `tls-certificate-*` options are unset or set to the empty string. Otherwise GoToSocial will attempt to handle TLS itself.
+
+!!! danger "Protocol configuration option"
+    Do **not** change the [`protocol`](../configuration/general.md) configuration option to `http`. This should only ever by set to `http` for development purposes. It needs to be set to `https` even when running behind a TLS-terminating reverse proxy.
+
+You'll also want to change the [`port`](../configuration/general.md) GoToSocial binds on, so it no longer tries to use port 443.
+
+To configure TLS in your reverse proxy, please refer to their documentation:
+
+* [nginx](https://docs.nginx.com/nginx/admin-guide/security-controls/terminating-ssl-http/)
+* [apache](https://httpd.apache.org/docs/2.4/ssl/ssl_howto.html)
+* [Traefik](https://doc.traefik.io/traefik/https/tls/)
+* [Caddy](https://caddyserver.com/docs/caddyfile/directives/tls)
+
+!!! tip
+    When configuring TLS in your reverse proxy, ensure you configure a reasonably modern set of compatible versions and ciphers. You can use the "Intermediate" configuration from the [Mozilla SSL Configuration Generator](https://ssl-config.mozilla.org/).
+
+    Check the documentation of your reverse proxy on whether you have to reload or restart it after certificates have changed. Not all reverse proxies detect this.
+
+## Guides
+
+There are a number of good resources on the internet explaining how to set all of this up.
+
+* [ArchWiki entry](https://wiki.archlinux.org/title/certbot) on certbot
+* [Gentoo wiki entry](https://wiki.gentoo.org/wiki/Let%27s_Encrypt) on Lets Encrypt
+* [Linode guide](https://www.linode.com/docs/guides/enabling-https-using-certbot-with-nginx-on-fedora/) on certbot for Fedora, RHEL/CentOS, Debian and Ubuntu
+* Digital Ocean guides on Lets Encrypt on Ubuntu 22.04 with [nginx](https://www.digitalocean.com/community/tutorials/how-to-secure-nginx-with-let-s-encrypt-on-ubuntu-22-04) or [apache](https://www.digitalocean.com/community/tutorials/how-to-secure-apache-with-let-s-encrypt-on-ubuntu-22-04)

docs/advanced/host-account-domain.md

@@ -0,0 +1,104 @@
+# Split-domain deployments
+
+This guide explains how to have usernames like `@me@example.org` but run the GoToSocial instance itself on a subdomain like `social.example.org`. Configuring this type of deployment layout **must** be done before starting GoToSocial for the first time.
+
+!!! danger
+    You cannot change your domain layout after you've federated with someone. Servers are going to get confused and you'll need to convince the admin of every instance that's federated with you before to mess with their database to resolve it. It also requires regenerating the database on your side to create a new instance account and pair of encryption keys.
+
+## Background
+
+The way ActivityPub implementations discover how to map your account domain to your host domain is through a protocol called [webfinger](https://www.rfc-editor.org/rfc/rfc7033). This mapping is typically cached by servers and hence why you can't change it after the fact.
+
+It works by doing a request to `https://<account domain>/.well-known/webfinger?resource=acct:@me@example.org`. At this point, a server can return a redirect to where the actual webfinger endpoint is, `https://<host domain>/.well-known/webfinger?resource=acct:@me@example.org` or may respond directly. The JSON document that is returned informs you what the endpoint to query is for the user:
+
+```json
+{
+  "subject": "acct:me@example.org",
+  "aliases": [
+    "https://social.example.org/users/me",
+    "https://social.example.org/@me"
+  ],
+  "links": [
+    {
+      "rel": "http://webfinger.net/rel/profile-page",
+      "type": "text/html",
+      "href": "https://social.example.org/@me"
+    },
+    {
+      "rel": "self",
+      "type": "application/activity+json",
+      "href": "https://social.example.org/users/me"
+    }
+  ]
+}
+```
+
+ActivityPub clients and servers will now use the entry from the `links` array with `rel` `self` and `type` `application/activity+json` to query for further information, like where the `inbox` is located to federated messages to.
+
+## Configuration
+
+There are 2 configuration settings you'll need to concern yourself with:
+
+* `host`, the domain the API will be served on and what clients and servers will end up using when talking to your instance
+* `account-domain`, the domain user accounts will be created on
+
+In order to achieve the setup as described in the introduction, you'll need to set these two configuration options accordingly:
+
+```yaml
+host: social.example.org
+account-domain: example.org
+```
+
+!!! info
+    The `host` must always be the DNS name that your GoToSocial instance runs on. It does not affect the IP address the GoToSocial instance binds to. That is controlled with `bind-address`.
+
+## Reverse proxy
+
+When using a [reverse proxy](../getting_started/reverse_proxy/index.md) you'll need to ensure you're set up to handle traffic on both of those domains. You'll need to redirect a few endpoints from the account domain to the host domain.
+
+Redirects are typically used so that the change of domain can be detected client side. The endpoints to redirect from the account domain to the host domain are:
+
+* `/.well-known/webfinger`
+* `/.well-known/host-meta`
+* `/.well-known/nodeinfo`
+
+!!! tip
+    Do not proxy or redirect requests to the API endpoints, `/api/...`, from the account domain to the host domain. This will confuse heuristics some clients use to detect a split-domain deployment resulting in broken login flows and other weird behaviour.
+
+### nginx
+
+In order to configure the redirect, you'll need to configure it on the account domain. Assuming the account domain is `example.org` and the host domain is `social.example.org`, the following configuration snippet showcases how to do this:
+
+```nginx
+server {
+  server_name example.org;
+
+  location /.well-known/webfinger {
+    rewrite ^.*$ https://social.example.org/.well-known/webfinger permanent;
+  }
+
+  location /.well-known/host-meta {
+      rewrite ^.*$ https://social.example.org/.well-known/host-meta permanent;
+  }
+
+  location /.well-known/nodeinfo {
+      rewrite ^.*$ https://social.example.org/.well-known/nodeinfo permanent;
+  }
+}
+```
+
+### Traefik
+
+If `example.org` is running on [Traefik](https://doc.traefik.io/traefik/), we could use labels similar to the following to setup the redirect.
+
+```yaml
+myservice:
+  image: foo
+  # Other stuff
+  labels:
+    - 'traefik.http.routers.myservice.rule=Host(`example.org`)'
+    - 'traefik.http.middlewares.myservice-gts.redirectregex.permanent=true'
+    - 'traefik.http.middlewares.myservice-gts.redirectregex.regex=^https://(.*)/.well-known/(webfinger|nodeinfo|host-meta)$$'
+    - 'traefik.http.middlewares.myservice-gts.redirectregex.replacement=https://social.$${1}/.well-known/$${2}'
+    - 'traefik.http.routers.myservice.middlewares=myservice-gts@docker'
+```

docs/advanced/index.md

@@ -0,0 +1,16 @@
+# Advanced
+
+In this section we touch on a number of more advanced topics, primarily related around deploying, operating and tuning GoToSocial.
+
+We consider these topics advanced because applying them incorrectly does have the possibility of causing client and federation issues. Applying any of these configuration changes may also make it harder for you to debug an issue with your GoToSocial instance if you don't understand the changes that you're making.
+
+## Guides
+
+* [Split-domain deployment (API vs. account domain)](host-account-domain.md)
+* [Using an HTTP proxy for client/outgoing requests](outgoing-proxy.md)
+* [Provisioning TLS certificates](certificates.md)
+* [Caching API responses](caching/api.md)
+* [Serving and caching assets and media from local storage](caching/assets-media.md)
+* [Process sandboxing](security/sandboxing.md)
+* [Firewall configuration](security/firewall.md)
+* [Tracing](tracing.md)

docs/advanced/outgoing-proxy.md

@@ -0,0 +1,21 @@
+# Outgoing HTTP proxy
+
+GoToSocial supports canonical environment variables for configuring the use of an HTTP proxy for outgoing requets:
+
+* `HTTP_PROXY`
+* `HTTPS_PROXY`
+* `NO_PROXY`
+
+The lowercase versions of these environment variables are also recognised. `HTTPS_PROXY` takes precedence over `HTTP_PROXY` for https requests.
+
+The environment values may be either a complete URL or a `host[:port]`, in which case the "http" scheme is assumed. The schemes "http", "https", and "socks5" are supported.
+
+## systemd
+
+When running with systemd, you can add the necessary environment variables using the `Environment` option in the `Service` section.
+
+How to do so is documented in the [`systemd.exec` manual](https://www.freedesktop.org/software/systemd/man/systemd.exec.html#Environment).
+
+## Container runtime
+
+Environment variables can be set in the compose file under the `environment` key. You can also pass them on the CLI to Docker or Podman's `run` command with `-e KEY=VALUE` or `--env KEY=VALUE`.

docs/advanced/security/firewall.md

@@ -0,0 +1,84 @@
+# Firewall
+
+You should deploy a firewall on your instance to close off any open ports and give you a mechanism to ban potentially misbehaving clients. Many firewall frontends will also automatically install some rules that block obvious malicious packets.
+
+It can be helpful to deploy tools that monitor your log files for certain patterns and automatically ban clients exhibiting certain behaviour. This can be use to monitor your SSH and web server access logs for things like SSH brute-force attacks.
+
+## Ports
+
+For GoToSocial, you'll want to ensure port `443` remains open. Without it, nobody will be able to reach your instance. Federation will fail and client apps won't be able to work at all.
+
+If you [provision TLS certificates](../certificates.md) using ACME or GoToSocial's built-in Lets Encrypt support, you'll also need port `80` to be open.
+
+In order to access your instance over SSH, you'll need to keep the port your SSH daemon is bound on open too. By default this is port `22`.
+
+## ICMP
+
+[Internet Control Message Protocol](https://en.wikipedia.org/wiki/Internet_Control_Message_Protocol) are exchanged between machines in order to detect certain network conditions or troubleshoot things. Many firewalls have a tendency of blocking ICMP entirely but this is undesirable. A few ICMP types should be allowed and you can use your firewall to configure rate limiting for them instead.
+
+### IPv4
+
+In order for things to work reliably, your firewall must allow:
+
+* ICMP Type 3: "Destination Unreachable" and also aids in Path-MTU Discovery
+* ICMP Type 4: "Source Quench"
+
+If you want to be able to ping things or be pinged, you should also allow:
+
+* ICMP Type 0: "Echo Reply"
+* ICMP Type 8: "Echo Request"
+
+For traceroute to work, it can be helpful to also allow:
+
+* ICMP Type 11: "Time Exceeded"
+
+### IPv6
+
+ICMP is heavily relied on by all parts of the IPv6 stack and things will break in exciting and hard to debug ways if you block it. [RFC 4890](https://www.rfc-editor.org/rfc/rfc4890) was specifically written to address this and is worthwhile to review.
+
+Roughly speaking, you must always allow:
+
+* ICMP Type 1: "Destination Unreachable"
+* ICMP Type 2: "Packet Too Big"
+* ICMP Type 3, code 0: "Time Exceeded"
+* ICMP Type 4, code 1, 2: "Parameter Problem"
+
+For ping, you should allow:
+
+* ICMP Type 128: "Echo Request"
+* ICMP Type 129: "Echo Response"
+
+## Firewall configuration
+
+On Linux, firewalling is typically done using either [iptables](https://en.wikipedia.org/wiki/Iptables) or the more modern and faster [nftables](https://en.wikipedia.org/wiki/Nftables) as the backend. Most distributions are switching to nftables and many firewall frontends can be configured to use nftables instead. You'll need to refer to your distribution's documentation on the matter, but typically there will be an `iptables` or `nftables` service your init-system can start with a predefined location to load firewall rules from.
+
+Doing this by hand using raw iptables or nftables rules offers the most control but can be challenging if you're not familiar with these systems. In order to help with that, a number of configuration frontends exist that you can use.
+
+On the Debian and Ubuntu as well as openSUSE family of distributions, UFW is commonly used. It's a simple firewall front-end and many tutorials targeting those distributions will be using it.
+
+For the Red Hat/CentOS family of distributions, firewalld is typically used. It's a much more advanced firewall configuration utility which also has a desktop GUI and [Cockpit integration](https://cockpit-project.org/).
+
+Despite distribution preferences, you can use UFW, firewalld or something else entirely with any Linux distribution.
+
+* [Ubuntu Wiki](https://wiki.ubuntu.com/UncomplicatedFirewall?action=show&redirect=UbuntuFirewall) on UFW
+* [ArchWiki](https://wiki.archlinux.org/title/Uncomplicated_Firewall) on UFW
+* Digital Ocean guide on [using UFW with Ubuntu 22.04](https://www.digitalocean.com/community/tutorials/how-to-set-up-a-firewall-with-ufw-on-ubuntu-22-04)
+* [firewalld](https://firewalld.org/) project homepage and documentation
+* [ArchWiki](https://wiki.archlinux.org/title/firewalld) on firewalld
+* [Using and configuring firewalld](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9/html/configuring_firewalls_and_packet_filters/using-and-configuring-firewalld_firewall-packet-filters) from Red Hat
+* Linode guide [on using firewalld](https://www.linode.com/docs/guides/introduction-to-firewalld-on-centos/)
+
+## Brute-force protection
+
+[fail2ban](https://www.fail2ban.org) and [SSHGuard](https://www.sshguard.net/) can be set up to monitor your log files for attempts to brute-force logins and other malicious behaviour. They can be configured to automatically insert firewall rules to block malicious IP addresses, either for a specific period of time or even indefinitely.
+
+SSHGuard was initially designed just for SSH, but nowadays supports a variety of services. Fail2ban tends to support anything you can generate consistent log lines for, whereas SSHGuard's signature approach can catch more sophisticated or stealthy attacks as it computes an attack score over time.
+
+Both SSHGuard and fail2ban ship with "backends" that can target iptables and nftables directly, or work with your frontend of choice like UFW or firewalld on Linux or pf on \*BSD. Make sure you review their documentation on how to correctly configure it.
+
+* [ArchWiki](https://wiki.archlinux.org/title/Fail2ban) on fail2ban
+* DigitalOcean guide on how to protect SSH with [fail2ban on Ubuntu](https://www.digitalocean.com/community/tutorial_collections/how-to-protect-ssh-with-fail2ban)
+* Linode guide on how to [secure your server with fail2ban](https://www.linode.com/docs/guides/using-fail2ban-to-secure-your-server-a-tutorial/)
+* [ArchWiki](https://wiki.archlinux.org/title/sshguard) on sshguard
+* [FreeBSD manual](https://man.freebsd.org/cgi/man.cgi?query=sshguard&sektion=8&manpath=FreeBSD+13.2-RELEASE+and+Ports) for sshguard
+* [SSHGuard setup](https://manpages.ubuntu.com/manpages/lunar/en/man7/sshguard-setup.7.html) manual for Ubuntu

docs/advanced/security/index.md

@@ -0,0 +1,11 @@
+# Enhanced security
+
+These guides cover improving the security posture of your GoToSocial deployment. They don't involve tweaking settings in GoToSocial, but rather additional things you can do to better lock down your instance.
+
+!!! note
+    Anything in these guides is meant to enhance the security of your GoToSocial deployment; they are not a replacement for good security practices like keeping your systems patched and up to date.
+
+## Guides
+
+* [Sandboxing the GoToSocial binary](sandboxing.md)
+* [Firewall configuration](firewall.md)

docs/advanced/security/sandboxing.md

@@ -0,0 +1,78 @@
+# Application sandboxing
+
+By sandboxing the GoToSocial binary it's possible to control which parts of the system GoToSocial can access, and limit which things it can read and write. This can be helpful to ensure that even in the face of a security issue in GoToSocial, an attacker is severely hindered in escalating their privileges and gaining a foothold on your system.
+
+!!! note
+    As GoToSocial is still early in its development, the sandboxing policies we ship may get out of date. If you happen to run into this, please raise an issue on the issue tracker or better yet submit a PR to help us fix it.
+
+Different distributions have different sandboxing mechanisms they prefer and support:
+
+* **AppArmor** for the Debian or Ubuntu family of distributions or OpenSuSE, including when running with Docker
+* **SELinux** for the Red Hat/Fedora/CentOS family of distributions or Gentoo
+
+!!! warning "Containers and sandboxing"
+    Running GoToSocial as a container does not in and of itself provide much additional security. Despite their name, "containers do not contain". Containers are a distribution mechanism, not a security sandbox. To further secure your container you can instruct the container runtime to load the AppArmor profile and look into limiting which syscalls can be used using a seccomp profile.
+
+## AppArmor
+
+We ship an example AppArmor policy for GoToSocial, which you can retrieve and install as follows:
+
+```sh
+$ curl -LO 'https://github.com/superseriousbusiness/gotosocial/raw/main/example/apparmor/gotosocial'
+$ sudo install -o root -g root gotosocial /etc/apparmor.d/gotosocial
+$ sudo apparmor_parser -Kr /etc/apparmor.d/gotosocial
+```
+
+!!! tip
+    If you're using SQLite, the AppArmor profile expects the database in `/gotosocial/db/` so you'll need to adjust your configuration paths or the policy accordingly.
+
+With the policy installed, you'll need to configure your system to use it to constrain the permissions GoToSocial has.
+
+You can disable the policy like this:
+
+```sh
+$ sudo apparmor_parser -R /etc/apparmor.d/gotosocial
+$ sudo rm -vi /etc/apparmor.d/gotosocial
+```
+Don't forget to roll back any configuration changes you made that load the AppArmor policy.
+
+### systemd
+
+Add the following to the systemd service, or create an override:
+
+```ini
+[Service]
+...
+AppArmorProfile=gotosocial
+```
+
+Reload systemd and restart GoToSocial:
+
+```
+$ systemctl daemon-reload
+$ systemctl restart gotosocial
+```
+
+### Containers
+
+!!! tip
+    You should review the [Docker](https://docs.docker.com/engine/security/apparmor/) or [Podman](https://docs.podman.io/en/latest/markdown/options/security-opt.html) documentation on AppArmor.
+
+When using our example Compose file, you can tell it to load the AppArmor policy by tweaking it like so:
+
+```yaml
+services:
+  gotosocial:
+    ...
+    security_opt:
+      - apparmor=gotosocial
+```
+
+When launching the container with `docker run` or `podman run`, you'll need the `--security-opt="apparmor=gotosocial"` command line flag.
+
+## SELinux
+
+!!! note
+    SELinux can only be used in combination with the [binary installation](../../getting_started/installation/metal.md) method. SELinux cannot be used to constrain GoToSocial when running in a container.
+
+The SELinux policy is maintained by the community in the [`lzap/gotosocial-selinux`](https://github.com/lzap/gotosocial-selinux) repository on GitHub. Make sure to read its documentation, review the policy before using it and use their issue tracker for any support requests around the SELinux policy.

docs/advanced/tracing.md

@@ -0,0 +1,44 @@
+# Tracing
+
+GoToSocial comes with [OpenTelemetry][otel] based tracing built-in. It's not wired through every function, but our HTTP handlers and database library will create spans. How to configure tracing is explained in the [Observability configuration reference][obs].
+
+In order to receive the traces, you need something to ingest them and then visualise them. There are many options available including self-hosted and commercial options.
+
+We provide an example of how to do this using [Grafana Tempo][tempo] to ingest the spans and [Grafana][grafana] to explore them. Please beware that the configuration we provide is not suitable for a production setup. It can be used safely for local development and can provide a good starting point for setting up your own tracing infrastructure.
+
+You'll need the files in [`example/tracing`][ext]. Once you have those you can run `docker-compose up -d` to get Tempo and Grafana running. With both services running, you can add the following to your GoToSocial configuration and restart your instance:
+
+```yaml
+tracing-enabled: true
+tracing-transport: "grpc"
+tracing-endpoint: "localhost:4317"
+tracing-insecure: true
+```
+
+[otel]: https://opentelemetry.io/
+[obs]: ../configuration/observability.md
+[tempo]: https://grafana.com/oss/tempo/
+[grafana]: https://grafana.com/oss/grafana/
+[ext]: https://github.com/superseriousbusiness/gotosocial/tree/main/example/tracing
+
+## Querying and visualising traces
+
+Once you execute a few queries against your instance, you'll be able to find them in Grafana. You can use the Explore tab and pick Tempo as the datasource. Because our example configuration for Grafana enables [TraceQL][traceql], the Explore tab will have the TraceQL query type selected by default. You can switch to "Search" instead and find all traces emitted by GoToSocial under the "GoToSocial" service name.
+
+Using TraceQL, a simple query to find all traces related to requests to `/api/v1/instance` would look like this:
+
+```
+{.http.route = "/api/v1/instance"}
+```
+
+If you wanted to see all GoToSocial traces, you could instead run:
+
+```
+{.service.name = "GoToSocial"}
+```
+
+Once you select a trace, a second panel will open up visualising the span. You can drill down from there, by clicking into every sub-span to see what it was doing.
+
+![Grafana showing a trace for the /api/v1/instance endpoint](../assets/tracing.png)
+
+[traceql]: https://grafana.com/docs/tempo/latest/traceql/

docs/api/ratelimiting.md

@@ -1,10 +1,16 @@
-# Rate Limit
+# Request Rate Limiting
 
-To mitigate abuse + scraping of your instance, an IP-based HTTP rate limit is in place.
+To mitigate abuse + scraping of your instance, IP-based HTTP rate limiting is in place.
 
-This rate limit applies not just to the API, but to all requests (web, federation, etc).
+There are separate rate limiters configured for different groupings of endpoints. In other words, being rate limited for one part of the API doesn't necessarily mean you will be rate limited for other parts. Each entry in the following list has a separate rate limiter:
 
-By default, a maximum of 1000 requests in a 5 minute time window are allowed.
+- `/users/*` and `/emoji/*` - ActivityPub (s2s) endpoints.
+- `/auth/*` and `/oauth/*` - Sign in + OAUTH token requests.
+- `/fileserver/*` - Media attachments, emojis, etc.
+- `/nodeinfo/*` - NodeInfo endpoint(s).
+- `/.well-known/*` - webfinger + nodeinfo requests.
+
+By default, each rate limiter allows a maximum of 300 requests in a 5 minute time window: 1 request per second per client IP address.
 
 Every response will include the current status of the rate limit with the following headers:
 

docs/api/swagger.md

@@ -6,4 +6,4 @@ The resulting API documentation is rendered below, for quick reference.
 
 If you'd like to do more with the spec, you can also view the [swagger.yaml](/api/swagger/swagger.yaml) directly, and then paste it into something like the [Swagger Editor](https://editor.swagger.io/) in order to autogenerate GoToSocial API clients in different languages, convert the doc to JSON or OpenAPI v3 specification, etc. See [here](https://swagger.io/tools/open-source/getting-started/) for more.
 
-!!swagger swagger.yaml!!
+<swagger-ui src="swagger.yaml"/>

docs/api/throttling.md

@@ -0,0 +1,35 @@
+# Request Throttling
+
+GoToSocial uses request throttling to limit the number of open connections to the API of your instance. This is designed to prevent your instance from accidentally being DDOS'd (aka [the hug of death](https://en.wikipedia.org/wiki/Slashdot_effect)) if a post gets boosted or replied to by an account with many thousands of followers.
+
+Throttling means that only a limited number of HTTP requests to the API will be handled concurrently, in order to provide a snappy response to each request and move on quickly. The rationale is that it's better to handle fewer requests quickly, than to try to handle all incoming requests at once and take multiple seconds per request.
+
+Throttling limits are applied across router groups, similar to the way that [rate limiting](./ratelimiting.md) is organized, so if one part of the API is currently being throttled, that doesn't mean they all are.
+
+Throttling limits are calculated based on the number of CPUs available to GoToSocial, and the configuration value `advanced-throttling-multiplier`. The calculation is performed as follows:
+
+- In-process queue limit = number of CPUs * CPU multiplier.
+- Backlog queue limit = in-process queue limit * CPU multiplier.
+
+This leads to the following values for the default multiplier (8):
+
+```text
+1 cpu = 08 in-process, 064 backlog
+2 cpu = 16 in-process, 128 backlog
+4 cpu = 32 in-process, 256 backlog
+8 cpu = 64 in-process, 512 backlog
+```
+
+New requests that overflow the in-process limit are held in the backlog queue, and processed as soon as a spot is freed up (ie., when a currently in-process request is finished). Requests that cannot be processed, and cannot fit in the backlog queue will be responded to with http code [503 - Service Unavailable](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/503), and the `Retry-After` header will be set to `30` (seconds), to indicate that the caller should try again later.
+
+Requests are not held in the backlog queue indefinitely: if requests in the backlog cannot be processed within 30 seconds of being received, they will also receive a code 503 and a 30s retry-after.
+
+## Throttling FAQs
+
+### Can I tune the request throttling?
+
+Yes, just change the value of `advanced-throttling-multiplier` higher (if you have very powerful CPUs) or lower (if you have relatively less powerful CPUs).
+
+### Can I disable the request throttling?
+
+Yes. To do so, just set `advanced-throttling-multiplier` to `0` or less. This will disable HTTP request throttling entirely, and instead attempt to process all incoming requests at once. This is useful in cases where you want to do request throttling using an external service or a reverse-proxy, and you don't want GoToSocial to interfere with your setup.

docs/assets/css/colours.css

@@ -0,0 +1,25 @@
+[data-md-color-scheme="slate"] {
+	--md-primary-fg-color: #fd6a00;
+	--md-default-fg-color: #fafaff;
+	--md-default-fg-color--light: var(--md-default-fg-color);
+	--md-default-fg-color--lighter: var(--md-default-fg-color);
+	--md-default-fg-color--lightest: var(--md-default-fg-color);
+	--md-default-bg-color: #2a2b2f;
+	--md-default-bg-color--light: #35363b;
+	--md-default-bg-color--lighter: #3a3b41;
+	--md-default-bg-color--lightest: #4d4e56;
+
+	--md-typeset-color: var(--md-default-fg-color);
+	--md-typeset-a-color: #66befe;
+	--md-accent-fg-color: #89caff;
+
+	--md-footer-bg-color: var(--md-default-bg-color);
+}
+
+[data-md-color-scheme="slate"] .md-nav--primary .md-nav__title {
+	background: var(--md-default-bg-color) !important;
+}
+
+.md-content {
+	background: var(--md-default-bg-color--lightest);
+}

docs/configuration/accounts.md

@@ -39,4 +39,11 @@ accounts-reason-required: true
 # Options: [true, false]
 # Default: false
 accounts-allow-custom-css: false
+
+# Int. If accounts-allow-custom-css is true, this is the permitted length in characters for
+# CSS uploaded by accounts on this instance. No effect if accounts-allow-custom-css is false.
+#
+# Examples: [500, 5000, 9999]
+# Default: 10000
+accounts-custom-css-length: 10000
 ```

docs/configuration/advanced.md

@@ -36,9 +36,8 @@ These are set to sensible defaults, so most server admins won't need to touch th
 # Default: "lax"
 advanced-cookies-samesite: "lax"
 
-# Int. Amount of requests to permit from a single IP address within a span of 5 minutes.
-# If this amount is exceeded, a 429 HTTP error code will be returned.
-# See https://docs.gotosocial.org/en/latest/api/swagger/#rate-limit.
+# Int. Amount of requests to permit per router grouping from a single IP address within
+# a span of 5 minutes. If this amount is exceeded, a 429 HTTP error code will be returned.
 #
 # If you find yourself adjusting this limit because it's regularly being exceeded,
 # you should first verify that your settings for `trusted-proxies` (above) are correct.
@@ -50,6 +49,73 @@ advanced-cookies-samesite: "lax"
 # If you set this to 0 or less, rate limiting will be disabled entirely.
 #
 # Examples: [1000, 500, 0]
-# Default: 1000
-advanced-rate-limit-requests: 1000
+# Default: 300
+advanced-rate-limit-requests: 300
+
+# Int. Amount of open requests to permit per CPU, per router grouping, before applying http
+# request throttling. Any requests beyond the calculated limit are held in a backlog queue for
+# up to 30 seconds before either being processed or timing out. Requests that don't fit in the backlog
+# queue will have status 503 returned to them, and the header 'Retry-After' will be set to 30 seconds.
+#
+# Open request limit is available CPUs * multiplier; backlog queue limit is limit * multiplier.
+#
+# Example values for multiplier 8:
+#
+# 1 cpu = 08 open, 064 backlog
+# 2 cpu = 16 open, 128 backlog
+# 4 cpu = 32 open, 256 backlog
+#
+# Example values for multiplier 4:
+#
+# 1 cpu = 04 open, 016 backlog
+# 2 cpu = 08 open, 032 backlog
+# 4 cpu = 16 open, 064 backlog
+#
+# A multiplier of 8 is a sensible default, but you may wish to increase this for instances
+# running on very performant hardware, or decrease it for instances using v. slow CPUs.
+#
+# If you set this to 0 or less, http request throttling will be disabled entirely.
+#
+# Examples: [8, 4, 9, 0]
+# Default: 8
+advanced-throttling-multiplier: 8
+
+# Duration. Time period to use as the "retry-after" header value in response to throttled requests.
+# Minimum resolution is 1 second.
+#
+# Examples: [30s, 10s, 5s, 1m]
+# Default: "30s"
+advanced-throttling-retry-after: "30s"
+
+# Int. CPU multiplier for the amount of goroutines to spawn in order to send messages via ActivityPub.
+# Messages will be batched so that at most multiplier * CPU count messages will be sent out at once.
+# This can be tuned to limit concurrent POSTing to remote inboxes, preventing your instance CPU
+# usage from skyrocketing when an account with many followers posts a new status.
+#
+# Messages are split among available senders, and each sender processes its assigned messages in serial.
+# For example, say a user with 1000 followers is on an instance with 2 CPUs. With the default multiplier
+# of 2, this means 4 senders would be in process at once on this instance. When the user creates a new post,
+# each sender would end up iterating through about 250 Create messages + delivering them to remote instances.
+#
+# If you set this to 0 or less, only 1 sender will be used regardless of CPU count. This may be
+# useful in cases where you are working with very tight network or CPU constraints.
+#
+# Example values for multiplier 2 (default):
+#
+# 1 cpu = 2 concurrent senders
+# 2 cpu = 4 concurrent senders
+# 4 cpu = 8 concurrent senders
+#
+# Example values for multiplier 4:
+#
+# 1 cpu = 4 concurrent senders
+# 2 cpu = 8 concurrent senders
+# 4 cpu = 16 concurrent senders
+#
+# Example values for multiplier <1:
+#
+# 1 cpu = 1 concurrent sender
+# 2 cpu = 1 concurrent sender
+# 4 cpu = 1 concurrent sender
+advanced-sender-multiplier: 2
 ```

docs/configuration/database.md

@@ -107,4 +107,64 @@ db-tls-mode: "disable"
 # Examples: ["/path/to/some/cert.crt"]
 # Default: ""
 db-tls-ca-cert: ""
+
+# Int. Number to multiply by CPU count to set permitted total of open database connections (in-use and idle).
+# You can use this setting to tune your database connection behavior, though most admins won't need to touch it.
+#
+# Example values for multiplier 8:
+#
+# 1 cpu = 08 open connections
+# 2 cpu = 16 open connections
+# 4 cpu = 32 open connections
+#
+# Example values for multiplier 4:
+#
+# 1 cpu = 04 open connections
+# 2 cpu = 08 open connections
+# 4 cpu = 16 open connections
+#
+# A multiplier of 8 is a sensible default, but you may wish to increase this for instances
+# running on very performant hardware, or decrease it for instances using v. slow CPUs.
+#
+# If you set the multiplier to less than 1, only one open connection will be used regardless of cpu count.
+#
+# PLEASE NOTE!!: This setting currently only applies for Postgres. SQLite will always use 1 connection regardless
+# of what is set here. This behavior will change in future when we implement better SQLITE_BUSY handling.
+# See https://github.com/superseriousbusiness/gotosocial/issues/1407 for more details.
+#
+# Examples: [16, 8, 10, 2]
+# Default: 8
+db-max-open-conns-multiplier: 8
+
+# String. SQLite journaling mode.
+# SQLite only -- unused otherwise.
+# If set to empty string, the sqlite default will be used.
+# See: https://www.sqlite.org/pragma.html#pragma_journal_mode
+# Examples: ["DELETE", "TRUNCATE", "PERSIST", "MEMORY", "WAL", "OFF"]
+# Default: "WAL"
+db-sqlite-journal-mode: "WAL"
+
+# String. SQLite synchronous mode.
+# SQLite only -- unused otherwise.
+# If set to empty string, the sqlite default will be used.
+# See: https://www.sqlite.org/pragma.html#pragma_synchronous
+# Examples: ["OFF", "NORMAL", "FULL", "EXTRA"]
+# Default: "NORMAL"
+db-sqlite-synchronous: "NORMAL"
+
+# Byte size. SQlite cache size.
+# SQLite only -- unused otherwise.
+# If set to empty string or zero, the sqlite default (2MiB) will be used.
+# See: https://www.sqlite.org/pragma.html#pragma_cache_size
+# Examples: ["0", "2MiB", "8MiB", "64MiB"]
+# Default: "8MiB"
+db-sqlite-cache-size: "8MiB"
+
+# Duration. SQlite busy timeout.
+# SQLite only -- unused otherwise.
+# If set to empty string or zero, the sqlite default will be used.
+# See: https://www.sqlite.org/pragma.html#pragma_busy_timeout
+# Examples: ["0s", "1s", "30s", "1m", "5m"]
+# Default: "30m"
+db-sqlite-busy-timeout: "30m"
 ```

docs/configuration/general.md

@@ -23,11 +23,21 @@ log-level: "info"
 # Default: false
 log-db-queries: false
 
+# Bool. Include the client IP in the emitted log lines
+# Options: [true, false]
+# Default: true
+log-client-ip: true
+
 # String. Application name to use internally.
 # Examples: ["My Application","gotosocial"]
 # Default: "gotosocial"
 application-name: "gotosocial"
 
+# String. The user that will be shown instead of the landing page. if no user is set, the landing page will be shown.
+# Examples: "admin"
+# Default: ""
+landing-page-user: ""
+
 # String. Hostname that this server will be reachable at. Defaults to localhost for local testing,
 # but you should *definitely* change this when running for real, or your server won't work at all.
 # DO NOT change this after your server has already run once, or you will break things!
@@ -43,6 +53,11 @@ host: "localhost"
 # to "gts.example.org/.well-known/webfinger" so that GtS can handle them properly.
 #
 # You should also redirect requests at "example.org/.well-known/nodeinfo" in the same way.
+#
+# You should also redirect requests at "example.org/.well-known/host-meta" in the same way. This endpoint
+# is used by a number of clients to discover the API endpoint to use when the host and account domain are
+# different.
+#
 # An empty string (ie., not set) means that the same value as 'host' will be used.
 #
 # DO NOT change this after your server has already run once, or you will break things!
@@ -64,9 +79,11 @@ protocol: "https"
 
 # String. Address to bind the GoToSocial server to.
 # This can be an IPv4 address or an IPv6 address (surrounded in square brackets), or a hostname.
-# Default value will bind to all interfaces.
-# You probably won't need to change this unless you're setting GoToSocial up in some fancy way or
-# you have specific networking requirements.
+# The default value will bind to all interfaces, which makes the server
+# accessible by other machines. For most setups there is no need to change this.
+# If you are using GoToSocial in a reverse proxy setup with the proxy running on
+# the same machine, you will want to set this to "localhost" or an equivalent,
+# so that the proxy can't be bypassed.
 # Examples: ["0.0.0.0", "172.128.0.16", "localhost", "[::]", "[2001:db8::fed1]"]
 # Default: "0.0.0.0"
 bind-address: "0.0.0.0"

docs/configuration/httpclient.md

@@ -0,0 +1,68 @@
+# HTTP Client
+
+## Settings
+
+```yaml
+################################
+##### HTTP CLIENT SETTINGS #####
+################################
+
+# Settings for OUTGOING http client connections used by GoToSocial to make
+# requests to remote resources (status GETs, media GETs, inbox POSTs, etc).
+
+http-client:
+
+  # Duration. Timeout to use for outgoing HTTP requests. If the timeout
+  # is exceeded, the connection to the remote server will be dropped.
+  # A value of 0s indicates no timeout: this is not advised!
+  # Examples: ["5s", "10s", "0s"]
+  # Default: "10s"
+  timeout: "10s"
+
+  ########################################
+  #### RESERVED IP RANGE EXCEPTIONS ######
+  ########################################
+  #
+  # Explicitly allow or block outgoing dialing within the provided IPv4/v6 CIDR ranges.
+  #
+  # By default, as a basic security precaution, GoToSocial blocks outgoing dialing within most "special-purpose"
+  # IP ranges. However, it may be desirable for admins with more exotic setups (proxies, funky NAT, etc) to
+  # explicitly override one or more of these otherwise blocked ranges.
+  #
+  # Each of the below allow/block config options accepts an array of IPv4 and/or IPv6 CIDR strings.
+  # For example, to override the hardcoded block of IPv4 and IPv6 dialing to localhost, set:
+  #
+  #   allow-ips: ["127.0.0.1/32", "::1/128"].
+  #
+  # You can also use YAML multi-line arrays to define these, but be diligent with indentation.
+  #
+  # When dialing, GoToSocial will first check if the destination falls within explicitly allowed IP ranges,
+  # then explicitly blocked IP ranges, then the default (hardcoded) blocked IP ranges, returning OK on the
+  # first allowed match, not OK on the first blocked match, or just defaulting to OK if nothing is matched.
+  #
+  # As with all security settings, it is better to start too restrictive and then ease off depending on
+  # your use case, than to start too permissive and try to close the stable door after the horse has
+  # already bolted. With this in mind:
+  # - Don't touch these settings unless you have a good reason to, and only if you know what you're doing.
+  # - When adding explicitly allowed exceptions, use the narrowest possible CIDR for your use case.
+  #
+  # For reserved / special ranges, see:
+  # - https://www.iana.org/assignments/iana-ipv4-special-registry/iana-ipv4-special-registry.xhtml
+  # - https://www.iana.org/assignments/iana-ipv6-special-registry/iana-ipv6-special-registry.xhtml
+  #
+  # Both allow-ips and block-ips default to an empty array.
+  allow-ips: []
+  block-ips: []
+
+  # Bool. Disable verification of TLS certificates of remote servers.
+  # With this set to 'true', GoToSocial will not error when a remote
+  # server presents an invalid or self-signed certificate.
+  #
+  # THIS SETTING SHOULD BE USED FOR TESTING ONLY! IF YOU TURN THIS
+  # ON WHILE RUNNING IN PRODUCTION YOU ARE LEAVING YOUR SERVER WIDE
+  # OPEN TO MAN IN THE MIDDLE ATTACKS! DO NOT CHANGE THIS SETTING 
+  # UNLESS YOU KNOW EXACTLY WHAT YOU'RE DOING AND WHY YOU'RE DOING IT.
+  #
+  # Default: false
+  tls-insecure-skip-verify: false
+```

docs/configuration/instance.md

@@ -24,6 +24,12 @@ instance-expose-peers: false
 # Default: false
 instance-expose-suspended: false
 
+# Bool. Allow unauthenticated users to view /about/suspended,
+# showing the HTML rendered list of instances that this instance blocks/suspends.
+# Options: [true, false]
+# Default: false
+instance-expose-suspended-web: false
+
 # Bool. Allow unauthenticated users to make queries to /api/v1/timelines/public in order
 # to see a list of public posts on this server. Even if set to 'false', then authenticated
 # users (members of the instance) will still be able to query the endpoint.

docs/configuration/letsencrypt.md

@@ -1,42 +0,0 @@
-# LetsEncrypt
-
-## Settings
-
-```yaml
-##############################
-##### LETSENCRYPT CONFIG #####
-##############################
-
-# Config pertaining to the automatic acquisition and use of LetsEncrypt HTTPS certificates.
-
-# Bool. Whether or not letsencrypt should be enabled for the server.
-# If false, the rest of the settings here will be ignored.
-# If you serve GoToSocial behind a reverse proxy like nginx or traefik, leave this turned off.
-# If you don't, then turn it on so that you can use https.
-# Options: [true, false]
-# Default: false
-letsencrypt-enabled: false
-
-# Int. Port to listen for letsencrypt certificate challenges on.
-# If letsencrypt is enabled, this port must be reachable or you won't be able to obtain certs.
-# If letsencrypt is disabled, this port will not be used.
-# This *must not* be the same as the webserver/API port specified above.
-# Examples: [80, 8000, 1312]
-# Default: 80
-letsencrypt-port: 80
-
-# String. Directory in which to store LetsEncrypt certificates.
-# It is a good move to make this a sub-path within your storage directory, as it makes
-# backup easier, but you might wish to move them elsewhere if they're also accessed by other services.
-# In any case, make sure GoToSocial has permissions to write to / read from this directory.
-# Examples: ["/home/gotosocial/storage/certs", "/acmecerts"]
-# Default: "/gotosocial/storage/certs"
-letsencrypt-cert-dir: "/gotosocial/storage/certs"
-
-# String. Email address to use when registering LetsEncrypt certs.
-# Most likely, this will be the email address of the instance administrator.
-# LetsEncrypt will send notifications about expiring certificates etc to this address.
-# Examples: ["admin@example.org"]
-# Default: ""
-letsencrypt-email-address: ""
-```

docs/configuration/media.md

@@ -7,7 +7,7 @@
 ##### MEDIA CONFIG #####
 ########################
 
-# Config pertaining to user media uploads (videos, image, image descriptions).
+# Config pertaining to media uploads (videos, image, image descriptions, emoji).
 
 # Int. Maximum allowed image upload size in bytes.
 # Examples: [2097152, 10485760]
@@ -37,8 +37,8 @@ media-description-max-chars: 500
 #
 # If this is set to 0, then media from remote instances will be cached indefinitely.
 # Examples: [30, 60, 7, 0]
-# Default: 30
-media-remote-cache-days: 30
+# Default: 7
+media-remote-cache-days: 7
 
 # Int. Max size in bytes of emojis uploaded to this instance via the admin API.
 # The default is the same as the Mastodon size limit for emojis (50kb), which allows
@@ -53,6 +53,6 @@ media-emoji-local-max-size: 51200
 # This strikes a good balance between decent interoperability with instances that have
 # higher emoji size limits, and not taking up too much space in storage.
 # Examples: [51200, 102400]
-# Default: 51200
+# Default: 102400
 media-emoji-remote-max-size: 102400
 ```

docs/configuration/observability.md

@@ -0,0 +1,38 @@
+# Observability
+
+These settings let you tune and configure certain observability related behaviours.
+
+## Settings
+
+```yaml
+##################################
+##### OBSERVABILITY SETTINGS #####
+##################################
+
+# String. Header name to use to extract a request or trace ID from. Typically set by a
+# loadbalancer or proxy.
+# Default: "X-Request-Id"
+request-id-header: "X-Request-Id"
+
+# Bool. Enable OpenTelemetry based tracing support.
+# Default: false
+tracing-enabled: false
+
+# String. Set the transport protocol for the tracing system. Can either be "grpc" for
+# OTLP gRPC or "jaeger" for jaeger based ingesters.
+# Options: ["grpc", "jaeger"]
+# Default: "grpc"
+tracing-transport: "grpc"
+
+# String. Endpoint of the trace ingester. When using the gRPC based transport, the
+# endpoint is usually a single address/port combination. For the jaeger transport it
+# should be a fully qualified URL.
+# OTLP gRPC or "jaeger" for jaeger based ingesters
+# Examples: ["localhost:4317", "http://localhost:14268/api/traces"]
+# Default: ""
+tracing-endpoint: ""
+
+# Bool. Disable HTTPS for the gRPC transport protocol.
+# Default: false
+tracing-insecure-transport: false
+```

docs/configuration/oidc.md

@@ -10,6 +10,9 @@ This is very convenient in the following cases:
 - You want to delegate management of users, accounts, passwords etc. to an external service to make admin easier for yourself.
 - You already have a lot of users in an external system and you don't want to have to recreate them all in GoToSocial manually.
 
+!!! tip
+    If a user doesn't exist yet, login will fail if your IdP doesn't return a non-empty `email` as part of the claims. The email needs to be unique on this instance. Though we use the `sub` claim to associate the external identity with a GtS user, when a user is created it needs an email associated with it.
+
 ## Settings
 
 GoToSocial exposes the following configuration settings for OIDC, shown below with their default values.
@@ -60,7 +63,7 @@ oidc-client-secret: ""
 # Array of string. Scopes to request from the OIDC provider. The returned values will be used to
 # populate users created in GtS as a result of the authentication flow. 'openid' and 'email' are required.
 # 'profile' is used to extract a username for the newly created user.
-# 'groups' is optional and can be used to determine if a user is an admin (if they're in the group 'admin' or 'admins').
+# 'groups' is optional and can be used to determine if a user is an admin based on oidc-admin-groups.
 # Examples: See eg., https://auth0.com/docs/scopes/openid-connect-scopes
 # Default: ["openid", "email", "profile", "groups"]
 oidc-scopes:
@@ -68,6 +71,18 @@ oidc-scopes:
   - "email"
   - "profile"
   - "groups"
+
+# Bool. Link OIDC authenticated users to existing ones based on their email address.
+# This is mostly intended for migration purposes if you were running previous versions of GTS
+# which only correlated users with their email address. Should be set to false for most usecases.
+# Options: [true, false]
+# Default: false
+oidc-link-existing: false
+
+# Array of string. If the returned ID token contains a 'groups' claim that matches one of the
+# groups in oidc-admin-groups, then this user will be granted admin rights on the GtS instance
+# Default: []
+oidc-admin-groups: []
 ```
 
 ## Behavior
@@ -76,47 +91,35 @@ When OIDC is enabled on GoToSocial, the default sign-in page redirects automatic
 
 This means that OIDC essentially *replaces* the normal GtS email/password sign-in flow.
 
-When a user logs in through OIDC, GoToSocial will request that user's preferred email address and username from the OIDC provider. It will then use the returned email address to either:
+Due to the way the ActivityPub standard works, you _cannot_ change your username
+after it has been set. This conflicts with the OIDC spec which does not
+guarantee that the `preferred_username` field is stable.
 
-*If the email address is already associated with a user/account*: sign the requester in as that user/account.
+To work with this, we ask the user to provide a username on their first login
+attempt. The field for this is pre-filled with the value of the `preferred_username` claim.
 
-Or:
+After authenticating, GtS stores the `sub` claim supplied by the OIDC provider.
+On subsequent authentication attempts, the user is looked up using this claim
+exclusively.
 
-*If the email address is not yet associated with a user/account*: create a new user and account with the returned credentials, and sign the requester in as that user/account.
-
-In other words, GoToSocial completely delegates sign-in authority to the OIDC provider, and trusts whatever credentials it returns.
-
-### Username conflicts
-
-In some cases, such as when a server has been switched to use OIDC after already using default settings for a while, there may be an overlap between usernames returned from OIDC, and usernames that already existed in the database.
-
-For example, let's say that someone with username `gordonbrownfan` and email address `gordon_is_best@example.org` has an account on a GtS instance that uses the default sign-in flow.
-
-That GtS instance then switches to using OIDC login. However, in the OIDC's storage there's also a user with username `gordonbrownfan`. If this user has the email address `gordon_is_best@example.org`, then GoToSocial will assume that the two users are the same and just log `gordonbrownfan` in as though nothing had changed. No problem!
-
-However, if the user in the OIDC storage has a different email address, GoToSocial will try to create a new user and account for this person.
-
-Since the username `gordonbrownfan` is already taken, GoToSocial will try `gordonbrownfan1`. If this is also taken, it will try `gordonbrownfan2`, and so on, until it finds a username that's not yet taken. It will then sign the requester in as that user/account, distinct from the original `gordonbrownfan`.
-
-### Malformed usernames
-
-A username returned from an OIDC provider might not always fit the pattern of what GoToSocial accepts as a valid username, ie., lower-case letters, numbers, and underscores. In this case, GoToSocial will do its best to parse the returned username into something that fits the pattern.
-
-For example, say that an OIDC provider returns the username `Marx Is Great` for a sign in, which doesn't fit the pattern because it contains upper-case letters and spaces.
-
-In this case, GtS will convert it into `marx_is_great` by applying the following rules:
-
-1. Trim any leading or trailing whitespace.
-2. Convert all letters to lowercase.
-3. Replace spaces with underscores.
-
-Unfortunately, at this point GoToSocial doesn't know how to handle returned usernames containing special characters such as `@` or `%`, so these will return an error.
+This then allows you to change the username on a provider level without losing
+access to your GtS account.
 
 ### Group membership
 
 Most OIDC providers allow for the concept of groups and group memberships in returned claims. GoToSocial can use group membership to determine whether or not a user returned from an OIDC flow should be created as an admin account or not.
 
-If the returned OIDC groups information for a user contains membership of the groups `admin` or `admins`, then that user will be created/signed in as though they are an admin.
+If the returned OIDC groups information for a user contains membership of the groups configured in `oidc-admin-groups`, then that user will be created/signed in as though they are an admin.
+
+## Migrating from old versions
+
+If you're moving from an old version of GtS which used the unstable `email`
+claim for unique user identification, you can set the `oidc-link-existing`
+configuration to `true`. If no user can be found for the ID returned by the
+provider, a lookup based on the `email` claim is performed instead. If this
+succeeds, the stable id is added to the database for the matching user.
+
+You should only use this for a limited time to avoid malicious account takeover.
 
 ## Provider Examples
 

docs/configuration/smtp.md

@@ -45,6 +45,18 @@ smtp-password: ""
 # Examples: ["mail@example.org"]
 # Default: ""
 smtp-from: ""
+
+# Bool. If true, when an email is sent that has multiple recipients, each recipient
+# will be included in the To field, so that each recipient can see who else got the
+# email, and they can 'reply all' to the other recipients if they want to.
+#
+# If false, email will be sent to Undisclosed Recipients, and each recipient will not
+# be able to see who else received the email.
+#
+# It might be useful to change this setting to 'true' if you want to be able to discuss
+# new moderation reports with other admins by 'replying-all' to the notification email.
+# Default: false
+smtp-disclose-recipients: false
 ```
 
 Note that if you don't set `Host`, then email sending via smtp will be disabled, and the other settings will be ignored. GoToSocial will still log (at trace level) emails that *would* have been sent if smtp was enabled.
@@ -59,11 +71,19 @@ The exception to this requirement is if you're running your mail server (or brid
 
 ### When are emails sent?
 
-Currently, emails are only sent to users to request email confirmation when a new account is created, or to serve password reset requests. More email functionality will probably be added later.
+Currently, emails are sent:
+
+- To the provided email address of a new user to request email confirmation when a new account is created via the API.
+- To all active instance moderators + admins when a new moderation report is received. By default, recipients are Bcc'd, but you can change this behavior with the setting `smtp-disclose-recipients`.
+- To the creator of a report (on this instance) when the report is closed by a moderator.
+
+### Can I test if my SMTP configuration is correct?
+
+Yes, you can use the API to send a test email to yourself. Check the API documentation for the `/api/v1/admin/email/test` endpoint.
 
 ### HTML versus Plaintext
 
-Emails are sent in HTML by default. At this point, there is no option to send emails in plaintext, but this is something that might be added later if there's enough demand for it.
+Emails are sent in plaintext by default. At this point, there is no option to send emails in html, but this is something that might be added later if there's enough demand for it.
 
 ## Customization
 

docs/configuration/storage.md

@@ -24,11 +24,9 @@ storage-local-base-path: "/gotosocial/storage"
 
 # String. API endpoint of the S3 compatible service.
 # Only required when running with the s3 storage backend.
-#
-# If your endpoint contains the bucket name, all files will be put into a
-# subdirectory with the name of `storage-s3-bucket`
-#
 # Examples: ["minio:9000", "s3.nl-ams.scw.cloud", "s3.us-west-002.backblazeb2.com"]
+# GoToSocial uses "DNS-style" when accessing buckets.
+# If you are using Scaleways object storage, please remove the "bucket name" from the endpoint address
 # Default: ""
 storage-s3-endpoint: ""
 
@@ -37,18 +35,27 @@ storage-s3-endpoint: ""
 # Default: false
 storage-s3-proxy: false
 
+# Bool. Use SSL for S3 connections.
+#
+# Only set this to 'false' when testing locally.
+#
+# Default: true
+storage-s3-use-ssl: true
+
 # String. Access key part of the S3 credentials.
 # Consider setting this value using environment variables to avoid leaking it via the config file
 # Only required when running with the s3 storage backend.
 # Examples: ["AKIAJSIE27KKMHXI3BJQ","miniouser"]
 # Default: ""
 storage-s3-access-key: ""
+
 # String. Secret key part of the S3 credentials.
 # Consider setting this value using environment variables to avoid leaking it via the config file
 # Only required when running with the s3 storage backend.
 # Examples: ["5bEYu26084qjSFyclM/f2pz4gviSfoOg+mFwBH39","miniopassword"]
 # Default: ""
 storage-s3-secret-key: ""
+
 # String. Name of the storage bucket.
 #
 # If you have already encoded your bucket name in the storage-s3-endpoint, this
@@ -62,6 +69,62 @@ storage-s3-secret-key: ""
 storage-s3-bucket: ""
 ```
 
+### AWS S3 Bucket Configuration
+
+#### Bucket Created
+GoToSocial by default creates signed URL's which means we dont need to change anything major on the policies of the bucket.
+Here are the steps to follow for bucket creation
+
+1. Login to AWS -> select S3 as service.
+2. click Create Bucket
+3. Provide a unique name and avoid adding "." in the name
+4. Do not change the public access settings (Let them be on "block public access" mode)
+
+#### AWS ACCESS KEY Configuration
+
+1. In AWS Console -> IAM (under Security, Identity, & Compliance)
+2. Add a user with programatic api's access
+3. We recommend setting up below listed policy, replace <bucketname> with your buckets name
+
+```json
+{
+    "Statement": [
+        {
+            "Effect": "Allow",
+            "Action": "s3:ListAllMyBuckets",
+            "Resource": "arn:aws:s3:::*"
+        },
+        {
+            "Effect": "Allow",
+            "Action": "s3:*",
+            "Resource": [
+                "arn:aws:s3:::<bucket_name>",
+                "arn:aws:s3:::<bucket_name>/*"
+            ]
+        }
+    ]
+}
+```
+
+4. Provide the values in config above
+  
+  * storage-s3-endpoint -> should be your bucket location say `s3.ap-southeast-1.amazonaws.com`
+  * storage-s3-access-key -> Access key you obtained for the user created above
+  * storage-s3-secret-key -> Secret key you obtained for the user created above
+  * storage-s3-bucket -> Keep this as the <bucketname> that you created just now.
+
+
+
+#### Migrating data from local storage to AWS s3 bucket
+
+This step is only needed if you have a running instance. Ignore this if you are setting up a fresh instance. 
+We have provided [s3cmd](https://github.com/s3tools/s3cmd) command for the copy operation.
+
+```bash
+s3cmd sync --add-header="Cache-Control:public, max-age=315576000, immutable" ./ s3://<bucket name>
+```
+
+
 ### Migrating between backends
 
 Currently, migration between backends is freely possible. To do so, you only

docs/configuration/syslog.md

@@ -32,7 +32,7 @@ syslog-enabled: false
 
 # String. Protocol to use when directing logs to syslog. Leave empty to connect to local syslog.
 # Options: ["udp", "tcp", ""]
-# Default: "tcp"
+# Default: "udp"
 syslog-protocol: "udp"
 
 # String. Address:port to send syslog logs to. Leave empty to connect to local syslog.

docs/configuration/tls.md

@@ -0,0 +1,66 @@
+# TLS
+
+It's possible to configure TLS support in one of two ways:
+* Built-in support for Lets Encrypt / ACME compatible vendors
+* Loading TLS files from disk
+
+It is not possible to have both methods enabled at the same time.
+
+Note that when using TLS files loaded from disk you are responsible for restarting the instance when the files change. They are not automatically reloaded.
+
+## Settings
+
+```yaml
+##############################
+##### LETSENCRYPT CONFIG #####
+##############################
+
+# Config pertaining to the automatic acquisition and use of LetsEncrypt HTTPS certificates.
+
+# Bool. Whether or not letsencrypt should be enabled for the server.
+# If false, the rest of the settings here will be ignored.
+# If you serve GoToSocial behind a reverse proxy like nginx or traefik, leave this turned off.
+# If you don't, then turn it on so that you can use https.
+# Options: [true, false]
+# Default: false
+letsencrypt-enabled: false
+
+# Int. Port to listen for letsencrypt certificate challenges on.
+# If letsencrypt is enabled, this port must be reachable or you won't be able to obtain certs.
+# If letsencrypt is disabled, this port will not be used.
+# This *must not* be the same as the webserver/API port specified above.
+# Examples: [80, 8000, 1312]
+# Default: 80
+letsencrypt-port: 80
+
+# String. Directory in which to store LetsEncrypt certificates.
+# It is a good move to make this a sub-path within your storage directory, as it makes
+# backup easier, but you might wish to move them elsewhere if they're also accessed by other services.
+# In any case, make sure GoToSocial has permissions to write to / read from this directory.
+# Examples: ["/home/gotosocial/storage/certs", "/acmecerts"]
+# Default: "/gotosocial/storage/certs"
+letsencrypt-cert-dir: "/gotosocial/storage/certs"
+
+# String. Email address to use when registering LetsEncrypt certs.
+# Most likely, this will be the email address of the instance administrator.
+# LetsEncrypt will send notifications about expiring certificates etc to this address.
+# Examples: ["admin@example.org"]
+# Default: ""
+letsencrypt-email-address: ""
+
+##############################
+##### MANUAL TLS CONFIG  #####
+##############################
+
+# String. Path to a PEM-encoded file on disk that includes the certificate chain
+# and the public key
+# Examples: ["/gotosocial/storage/certs/chain.pem"]
+# Default: ""
+tls-certificate-chain: ""
+
+# String. Path to a PEM-encoded file on disk containing the private key for the
+# associated tls-certificate-chain
+# Examples: ["/gotosocial/storage/certs/private.pem"]
+# Default: ""
+tls-certificate-key: ""
+```

docs/environment.yml

@@ -1,45 +1,14 @@
 name: gotosocial-docs
 channels:
-  - defaults
+  - conda-forge
+  - nodefaults
 dependencies:
-  - _libgcc_mutex=0.1=main
-  - _openmp_mutex=4.5=1_gnu
-  - bzip2=1.0.8=h7b6447c_0
-  - ca-certificates=2021.10.26=h06a4308_2
-  - certifi=2020.6.20=pyhd3eb1b0_3
-  - ld_impl_linux-64=2.35.1=h7274673_9
-  - libffi=3.3=he6710b0_2
-  - libgcc-ng=9.3.0=h5101ec6_17
-  - libgomp=9.3.0=h5101ec6_17
-  - libstdcxx-ng=9.3.0=hd4cf53a_17
-  - libuuid=1.0.3=h7f8727e_2
-  - ncurses=6.3=h7f8727e_2
-  - openssl=1.1.1l=h7f8727e_0
-  - python=3.10.0=h151d27f_3
-  - readline=8.1=h27cfd23_0
-  - sqlite=3.36.0=hc218d9a_0
-  - tk=8.6.11=h1ccaba5_0
-  - tzdata=2021e=hda174b7_0
-  - wheel=0.37.0=pyhd3eb1b0_1
-  - xz=5.2.5=h7b6447c_0
-  - zlib=1.2.11=h7b6447c_3
+  - python=3.11.3=h2755cc3_0_cpython
+  - pip=23.1.2=pyhd8ed1ab_0
+  - mkdocs=1.4.3=pyhd8ed1ab_0
+  - mkdocs-material=9.1.9=pyhd8ed1ab_0
+  - mkdocs-material-extensions=1.1.1=pyhd8ed1ab_0
+  - pillow=9.5.0=py311h573f0d3_0
+  - cairosvg=2.6.0=pyhd8ed1ab_0
   - pip:
-    - click==8.0.3
-    - ghp-import==2.0.2
-    - importlib-metadata==4.8.2
-    - jinja2==3.0.3
-    - markdown==3.3.4
-    - markupsafe==2.0.1
-    - mergedeep==1.3.4
-    - mkdocs==1.2.3
-    - mkdocs-render-swagger-plugin==0.0.3
-    - packaging==21.2
-    - pip==21.2.4
-    - pyparsing==2.4.7
-    - python-dateutil==2.8.2
-    - pyyaml==6.0
-    - pyyaml-env-tag==0.1
-    - setuptools==58.0.4
-    - six==1.16.0
-    - watchdog==2.1.6
-    - zipp==3.6.0
+      - mkdocs-swagger-ui-tag==0.6.1

docs/faq.md

@@ -1,30 +1,50 @@
 # Frequently Asked Questions
 
-- **Where's the user interface?** GoToSocial is just a bare server for the most part and is designed to be used thru external applications. [Pinafore](https://pinafore.social) and [Tusky](https://tusky.app/) are the best-supported, but anything that supports the Mastodon API should work, other than the features GoToSocial doesn't yet have. Permalinks and profile pages are served directly thru GoToSocial as well as the admin panel, but most interaction goes thru the apps.
+## Where's the user interface?
 
-- **What happened to the gifs?** While GoToSocial supports gifs, it doesn't support videos. This wouldn't be a big problem, except that Mastodon doesn't support gifs; it converts them into videos when they get uploaded. So if someone posts a gif from a Mastodon server, it won't be visible. At the time of this writing, the video will be dropped altogether, but [in the future there should be at least a placeholder link](https://github.com/superseriousbusiness/gotosocial/issues/765).
+GoToSocial is just a bare server for the most part and is designed to be used thru external applications. [Semaphore](https://semaphore.social/) and [Tusky](https://tusky.app/) are the best-supported, but anything that supports the Mastodon API should work, other than the features GoToSocial doesn't yet have. Permalinks and profile pages are served directly through GoToSocial as well as the settings panel, but most interaction goes through the apps.
 
-- **Why aren't my posts showing up on my profile page?** Unlike Mastodon, the default post visibility is Unlisted. If you want something to be visible on your profile page, the post must have Public visibility.
+## Why aren't my posts showing up on my profile page?
 
-- **Why aren't my posts showing up on other servers?** First check the visibility as noted above. TODO: explain how to debug common federation issues
+Unlike Mastodon, the default post visibility is Unlisted. If you want something to be visible on your profile page, the post must have Public visibility.
 
-- **Why am I getting frequent error responses?** GoToSocial is configured to use per-IP [rate limiting](https://docs.gotosocial.org/en/latest/api/ratelimiting/) by default, but in certain situations it can't accurately identify the remote IP and will treat all connections as coming from the same place. In those cases, the rate limiting needs to be disabled or reconfigured.
+## Why aren't my posts showing up on other servers?
 
-- **My instance is deployed and I'm logged in to a client but my timelines are empty, what's up there?** To see posts, you have to start following people! Once you've followed a few people and they've posted or boosted things, you'll start seeing them in your timelines. Right now GoToSocial doesn't have a way of 'backfilling' posts -- that is, fetching previous posts from other instances -- so you'll only see new posts of people you follow. If you want to interact with an older post of theirs, you can copy the link to the post from their web profile, and paste it in to your client's search bar.
+First check the visibility as noted above. TODO: explain how to debug common federation issues
 
-- **How can I sign up for a server?** Right now the only way to create an account is by the server's admin to run a command directly on the server. A web-based signup flow is in the roadmap but not implemented yet.
+## Why am I getting frequent http 429 error responses?
 
-- **Why's it still in alpha?** Take a look at the [list of open bugs](https://github.com/superseriousbusiness/gotosocial/issues?q=is%3Aissue+is%3Aopen+label%3Abug) and the [roadmap](https://github.com/superseriousbusiness/gotosocial/blob/main/ROADMAP.md) for a more detailed rundown, but the main missing features at the time of this writing are:
-    * videos
-    * reporting posts to admins
-    * muting conversations
-    * backfill of posts
-    * web-based signup
-    * profile metadata fields
-    * lists of users
-    * pinning posts to your profile
-    * polls
-    * scheduling posts
-    * account migration
-    * federated hashtag search
-    * shared block lists across servers
+GoToSocial is configured to use per-IP [rate limiting](./api/ratelimiting.md) by default, but in certain situations it can't accurately identify the remote IP and will treat all connections as coming from the same place. In those cases, the rate limiting needs to be disabled or reconfigured.
+
+## Why am I getting frequent HTTP 503 error responses?
+
+Code 503 is returned to callers when your instance is under heavy load and requests are being throttled. This behavior can be tuned as desired, or turned off entirely, see [here](./api/throttling.md).
+
+## I keep getting a 400 Bad Request error, and I have done everything suggested by the error message. What should I do?
+
+Verify that the `host` configuration matches the domain that GoToSocial is served from (the domain that users use to acces the server).
+
+## I keep seeing 'dial within blocked / reserved IP range' in my server logs, and I can't connect to some instances from my instance, what do I do?
+
+The IP address of the remote instance may be in one of the blocked "special use" IP ranges hardcoded into GoToSocial for security reasons. If you need to, you can override this in your configuration file. Have a look at the [http client docs](./configuration/httpclient.md) for this, and please read the warnings there carefully! If you add an explicit allow, you will have to restart your GoToSocial instance to make the config take effect.
+
+## My instance is deployed and I'm logged in to a client but my timelines are empty, what's up there?
+
+To see posts, you have to start following people! Once you've followed a few people and they've posted or boosted things, you'll start seeing them in your timelines. Right now GoToSocial doesn't have a way of 'backfilling' posts -- that is, fetching previous posts from other instances -- so you'll only see new posts of people you follow. If you want to interact with an older post of theirs, you can copy the link to the post from their web profile, and paste it in to your client's search bar.
+
+## How can I sign up for a server?
+
+Right now the only way to create an account is by the server's admin to run a command directly on the server. A web-based signup flow is in the roadmap but not implemented yet.
+
+## Why's it still in alpha?
+
+Take a look at the [list of open bugs](https://github.com/superseriousbusiness/gotosocial/issues?q=is%3Aissue+is%3Aopen+label%3Abug) and the [roadmap](https://github.com/superseriousbusiness/gotosocial/blob/main/ROADMAP.md) for a more detailed rundown, but the main missing features at the time of this writing are:
+
+- muting conversations
+- backfill of posts
+- web-based signup
+- polls
+- scheduling posts
+- account migration
+- federated hashtag search
+- shared block lists across servers

docs/federation/behaviors/conversation_threads.md

@@ -1,45 +0,0 @@
-# Conversation Threads
-
-Due to the nature of decentralization and federation, it is practically impossible for any one server on the fediverse to be aware of every post in a given conversation thread.
-
-With that said, it is possible to do 'best effort' dereferencing of threads, whereby remote replies are fetched from one server onto another, to try to more fully flesh out a conversation.
-
-GoToSocial does this by iterating up and down the thread of a conversation, pulling in remote statuses where possible.
-
-## Example
-
-Let's say we have two accounts: `local_account` on `our.server`, and `remote_1` on `remote.1`.
-
-In this scenario, `local_account` follows `remote_1`, so posts from `remote_1` show up in the home timeline of `local_account`.
-
-Now, `remote_1` boosts/reblogs a post from a third account, `remote_2`, residing on server `remote.2`.
-
-`local_account` does not follow `remote_2`, and neither does anybody else on `our.server`, which means that `our.server` has not seen this post by `remote_2` before.
-
-![A diagram of the conversation thread, showing the post from remote_2, and possible ancestor and descendant posts](../../assets/diagrams/conversation_thread.png)
-
-What GoToSocial will do now, is 'dereference' the post by `remote_2` to check if it is part of a thread and, if so, whether any other parts of the thread can be obtained.
-
-GtS begins by checking the `inReplyTo` property of the post, which is set when a post is a reply to another post. [See here](https://www.w3.org/TR/activitystreams-vocabulary/#dfn-inreplyto). If `inReplyTo` is set, GoToSocial derefences the replied-to post. If *this* post also has an `inReplyTo` set, then GoToSocial dereferences that too, and so on.
-
-Once all of these **ancestors** of a status have been retrieved, GtS will begin working down through the **descendants** of posts.
-
-It does this by checking the `replies` property of a derefenced post, and working through replies, and replies of replies. [See here](https://www.w3.org/TR/activitystreams-vocabulary/#dfn-replies).
-
-This process of thread dereferencing will likely involve making multiple HTTP calls to different servers, especially if the thread is long and complicated.
-
-The end result of this dereferencing is that, assuming the reblogged post by `remote_2` was part of a thread, then `local_account` should now be able to see posts in the thread when they open the status on their home timeline. In other words, they will see replies from accounts on other servers (who they may not have come across yet), in addition to any previous and next posts in the thread as posted by `remote_2`.
-
-This gives `local_account` a more complete view on the conversation, as opposed to just seeing the reblogged post in isolation and out of context. It also gives `local_account` the opportunity to discover new accounts to follow, based on replies to `remote_2`.
-
-## Privacy and Security
-
-During the dereferencing process, GoToSocial signs outgoing requests using the key of the actor who received the activity that necessitated dereferencing. To use the above example, this means that all dereferencing requests would be signed by `local_account`. This gives remote servers the ability to refuse these dereferencing requests, assuming that `local_account` is blocked by one or more participants in the conversation.
-
-From GoToSocial's side, domain blocks will be respected during the dereferencing process, to avoid making calls to servers that `our.server` has blocked.
-
-Individual account blocks will also be respected, meaning that `our.server` won't try to dereference posts from accounts blocked by `local_account`.
-
-Finally, GoToSocial expects that remote servers will only list replies that are marked as public (either `to` or `cc`). GtS may *try* to dereference followers-only posts, but it will assume that remote servers will check whether or not `local_account` is allowed to view them, and refuse accordingly.
-
-Of course, when `local_account` opens up the conversation thread in whatever application they are using, GoToSocial will apply the usual post visibility filtering to ensure that they do not see any posts that they shouldn't have access to.

docs/federation/behaviors/outbox.md

@@ -1,49 +0,0 @@
-# ActivityPub Outbox
-
-GoToSocial implements Outboxes for Actors (ie., instance accounts) following the ActivityPub specification [here](https://www.w3.org/TR/activitypub/#outbox).
-
-To get an [OrderedCollection](https://www.w3.org/TR/activitystreams-vocabulary/#dfn-orderedcollection) of Activities that an Actor has published recently, remote servers can do a `GET` request to a user's outbox. The address of this will be something like `https://example.org/users/whatever/outbox`.
-
-The server will return an OrderedCollection of the following structure:
-
-```json
-{
-    "@context": "https://www.w3.org/ns/activitystreams",
-    "id": "https://example.org/users/whatever/outbox",
-    "type": "OrderedCollection",
-    "first": "https://example.org/users/whatever/outbox?page=true"
-}
-```
-
-Note that the `OrderedCollection` itself contains no items. Callers must dereference the `first` page to start getting items. For example, a `GET` to `https://example.org/users/whatever/outbox?page=true` will produce something like the following:
-
-```json
-{
-    "id": "https://example.org/users/whatever/outbox?page=true",
-    "type": "OrderedCollectionPage",
-    "next": "https://example.org/users/whatever/outbox?max_id=01FJC1Q0E3SSQR59TD2M1KP4V8&page=true",
-    "prev": "https://example.org/users/whatever/outbox?min_id=01FJC1Q0E3SSQR59TD2M1KP4V8&page=true",
-    "partOf": "https://example.org/users/whatever/outbox",
-    "orderedItems": [
-        {
-            "id": "https://example.org/users/whatever/statuses/01FJC1MKPVX2VMWP2ST93Q90K7/activity",
-            "type": "Create",
-            "actor": "https://example.org/users/whatever",
-            "published": "2021-10-18T20:06:18Z",
-            "to": [
-                "https://www.w3.org/ns/activitystreams#Public"
-            ],
-            "cc": [
-                "https://example.org/users/whatever/followers"
-            ],
-            "object": "https://example.org/users/whatever/statuses/01FJC1MKPVX2VMWP2ST93Q90K7"
-        }
-    ]
-}
-```
-
-The `orderedItems` array will contain up to 30 entries. To get more entries beyond that, the caller can use the `next` link provided in the response.
-
-Note that in the returned `orderedItems`, all activity types will be `Create`. On each activity, the `object` field will be the AP URI of an original public status created by the Actor who owns the Outbox (ie., a `Note` with `https://www.w3.org/ns/activitystreams#Public` in the `to` field, which is not a reply to another status). Callers can use the returned AP URIs to dereference the content of the notes.
-
-Contrary to the ActivityPub spec, GoToSocial will deny requests that are not HTTP signed--that is, unauthenticated requests. This is consistent with GoToSocial's authentication policies for other federation API endpoints. This is to ensure that GoToSocial can deny requests from domains or users that have been blocked either by the GoToSocial instance itself (domain block), or by the individual owner of the Outbox.

docs/federation/federating_with_gotosocial.md

@@ -0,0 +1,429 @@
+# Federating with GoToSocial
+
+Information on the various (ActivityPub) elements needed to federate with GoToSocial.
+
+## HTTP Signatures
+
+GoToSocial requires all `GET` and `POST` requests to ActivityPub s2s endpoints to be accompanied by a valid http signature.
+
+GoToSocial will also sign all outgoing `GET` and `POST` requests that it makes to other servers.
+
+This behavior is the equivalent of Mastodon's [AUTHORIZED_FETCH / "secure mode"](https://docs.joinmastodon.org/admin/config/#authorized_fetch).
+
+GoToSocial uses the [go-fed/httpsig](https://github.com/go-fed/httpsig) library for signing outgoing requests, and for parsing and validating the signatures of incoming requests. This library strictly follows the [Cavage http signature RFC](https://datatracker.ietf.org/doc/html/draft-cavage-http-signatures), which is the same RFC used by other implementations like Mastodon, Pixelfed, Akkoma/Pleroma, etc. (This RFC has since been superceded by the [httpbis http signature RFC](https://datatracker.ietf.org/doc/html/draft-ietf-httpbis-message-signatures), but this is not yet widely implemented.)
+
+### Incoming Requests
+
+GoToSocial request signature validation is implemented in [internal/federation](https://github.com/superseriousbusiness/gotosocial/blob/main/internal/federation/authenticate.go).
+
+GoToSocial will attempt to parse the signature using the following algorithms (in order), stopping at the first success:
+
+```text
+RSA_SHA256
+RSA_SHA512
+ED25519
+```
+
+### Outgoing Requests
+
+GoToSocial request signing is implemented in [internal/transport](https://github.com/superseriousbusiness/gotosocial/blob/main/internal/transport/signing.go).
+
+When assembling signatures:
+
+- outgoing `GET` requests use `(request-target) host date`
+- outgoing `POST` requests use `(request-target) host date digest` 
+
+GoToSocial uses the `RSA_SHA256` algorithm for signing requests, which is in line with other ActivityPub implementations.
+
+### Quirks
+
+The `keyId` used by GoToSocial in the `Signature` header will look something like the following:
+
+```text
+https://example.org/users/example_user/main-key
+```
+
+This is different from most other implementations, which usually use a fragment (`#`) in the `keyId` uri. For example, on Mastodon the user's key would instead be found at:
+
+```text
+https://example.org/users/example_user#main-key
+```
+
+For Mastodon, the public key of a user is served as part of that user's Actor representation. GoToSocial mimics this behavior when serving the public key of a user, but instead of returning the entire Actor at the `main-key` endpoint (which may contain sensitive fields), will return only a partial stub of the actor. This looks like the following:
+
+```json
+{
+  "@context": [
+    "https://w3id.org/security/v1",
+    "https://www.w3.org/ns/activitystreams"
+  ],
+  "id": "https://example.org/users/example_user",
+  "preferredUsername": "example_user",
+  "publicKey": {
+    "id": "https://example.org/users/example_user/main-key",
+    "owner": "https://example.org/users/example_user",
+    "publicKeyPem": "-----BEGIN PUBLIC KEY-----\nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAzGB3yDvMl+8p+ViutVRG\nVDl9FO7ZURYXnwB3TedSfG13jyskoiMDNvsbLoUQM9ajZPB0zxJPZUlB/W3BWHRC\nNFQglE5DkB30GjTClNZoOrx64vLRT5wAEwIOjklKVNk9GJi1hFFxrgj931WtxyML\nBvo+TdEblBcoru6MKAov8IU4JjQj5KUmjnW12Rox8dj/rfGtdaH8uJ14vLgvlrAb\neQbN5Ghaxh9DGTo1337O9a9qOsir8YQqazl8ahzS2gvYleV+ou09RDhS75q9hdF2\nLI+1IvFEQ2ZO2tLk3umUP1ioa+5CWKsWD0GAXbQu9uunAV0VoExP4+/9WYOuP0ei\nKwIDAQAB\n-----END PUBLIC KEY-----\n"
+  },
+  "type": "Person"
+}
+```
+
+Remote servers federating with GoToSocial should extract the public key from the `publicKey` field. Then, they should use the `owner` field of the public key to further dereference the full version of the Actor, using a signed `GET` request.
+
+This behavior was introduced as a way of avoiding having remote servers make unsigned `GET` requests to the full Actor endpoint. However, this may change in future as it is not compliant and causes issues. Tracked in [this issue](https://github.com/superseriousbusiness/gotosocial/issues/1186).
+
+## Access Control
+
+GoToSocial uses access control restrictions to protect users and resources from unwanted interactions with remote accounts and instances.
+
+As shown in the [HTTP Signatures](#http-signatures) section, GoToSocial requires all incoming `GET` and `POST` requests from remote servers to be signed. Unsigned requests will be denied with http code `401 Unauthorized`.
+
+Access control restrictions are implemented by checking the `keyId` of the signature (who owns the public/private key pair making the request).
+
+First, the host value of the `keyId` uri is checked against the GoToSocial instance's list of blocked (defederated) domains. If the host is recognized as a blocked domain, then the http request will immediately be aborted with http code `403 Forbidden`.
+
+Next, GoToSocial will check for the existence of a block (in either direction) between the owner of the public key making the http request, and the owner of the resource that the request is targeting. If the GoToSocial user blocks the remote account making the request, then the request will be aborted with http code `403 Forbidden`.
+
+## Request Throttling & Rate Limiting
+
+GoToSocial applies http request throttling and rate limiting to the ActivityPub API endpoints (inboxes, user endpoints, emojis, etc).
+
+This ensures that remote servers cannot flood a GoToSocial instance with spurious requests. Instead, remote servers making GET or POST requests to the ActivityPub API endpoints should respect 429 and 503 http codes, and take account of the `retry-after` http response header.
+
+For more details on request throttling and rate limiting behavior, please see the [throttling](../api/throttling.md) and [rate limiting](../api/ratelimiting.md) documents.
+
+## Outbox
+
+GoToSocial implements Outboxes for Actors (ie., instance accounts) following the ActivityPub specification [here](https://www.w3.org/TR/activitypub/#outbox).
+
+To get an [OrderedCollection](https://www.w3.org/TR/activitystreams-vocabulary/#dfn-orderedcollection) of Activities that an Actor has published recently, remote servers can do a `GET` request to a user's outbox. The address of this will be something like `https://example.org/users/whatever/outbox`.
+
+The server will return an OrderedCollection of the following structure:
+
+```json
+{
+    "@context": "https://www.w3.org/ns/activitystreams",
+    "id": "https://example.org/users/whatever/outbox",
+    "type": "OrderedCollection",
+    "first": "https://example.org/users/whatever/outbox?page=true"
+}
+```
+
+Note that the `OrderedCollection` itself contains no items. Callers must dereference the `first` page to start getting items. For example, a `GET` to `https://example.org/users/whatever/outbox?page=true` will produce something like the following:
+
+```json
+{
+    "id": "https://example.org/users/whatever/outbox?page=true",
+    "type": "OrderedCollectionPage",
+    "next": "https://example.org/users/whatever/outbox?max_id=01FJC1Q0E3SSQR59TD2M1KP4V8&page=true",
+    "prev": "https://example.org/users/whatever/outbox?min_id=01FJC1Q0E3SSQR59TD2M1KP4V8&page=true",
+    "partOf": "https://example.org/users/whatever/outbox",
+    "orderedItems": [
+        {
+            "id": "https://example.org/users/whatever/statuses/01FJC1MKPVX2VMWP2ST93Q90K7/activity",
+            "type": "Create",
+            "actor": "https://example.org/users/whatever",
+            "published": "2021-10-18T20:06:18Z",
+            "to": [
+                "https://www.w3.org/ns/activitystreams#Public"
+            ],
+            "cc": [
+                "https://example.org/users/whatever/followers"
+            ],
+            "object": "https://example.org/users/whatever/statuses/01FJC1MKPVX2VMWP2ST93Q90K7"
+        }
+    ]
+}
+```
+
+The `orderedItems` array will contain up to 30 entries. To get more entries beyond that, the caller can use the `next` link provided in the response.
+
+Note that in the returned `orderedItems`, all activity types will be `Create`. On each activity, the `object` field will be the AP URI of an original public status created by the Actor who owns the Outbox (ie., a `Note` with `https://www.w3.org/ns/activitystreams#Public` in the `to` field, which is not a reply to another status). Callers can use the returned AP URIs to dereference the content of the notes.
+
+## Conversation Threads
+
+Due to the nature of decentralization and federation, it is practically impossible for any one server on the fediverse to be aware of every post in a given conversation thread.
+
+With that said, it is possible to do 'best effort' dereferencing of threads, whereby remote replies are fetched from one server onto another, to try to more fully flesh out a conversation.
+
+GoToSocial does this by iterating up and down the thread of a conversation, pulling in remote statuses where possible.
+
+Let's say we have two accounts: `local_account` on `our.server`, and `remote_1` on `remote.1`.
+
+In this scenario, `local_account` follows `remote_1`, so posts from `remote_1` show up in the home timeline of `local_account`.
+
+Now, `remote_1` boosts/reblogs a post from a third account, `remote_2`, residing on server `remote.2`.
+
+`local_account` does not follow `remote_2`, and neither does anybody else on `our.server`, which means that `our.server` has not seen this post by `remote_2` before.
+
+![A diagram of the conversation thread, showing the post from remote_2, and possible ancestor and descendant posts](../assets/diagrams/conversation_thread.png)
+
+What GoToSocial will do now, is 'dereference' the post by `remote_2` to check if it is part of a thread and, if so, whether any other parts of the thread can be obtained.
+
+GtS begins by checking the `inReplyTo` property of the post, which is set when a post is a reply to another post. [See here](https://www.w3.org/TR/activitystreams-vocabulary/#dfn-inreplyto). If `inReplyTo` is set, GoToSocial derefences the replied-to post. If *this* post also has an `inReplyTo` set, then GoToSocial dereferences that too, and so on.
+
+Once all of these **ancestors** of a status have been retrieved, GtS will begin working down through the **descendants** of posts.
+
+It does this by checking the `replies` property of a derefenced post, and working through replies, and replies of replies. [See here](https://www.w3.org/TR/activitystreams-vocabulary/#dfn-replies).
+
+This process of thread dereferencing will likely involve making multiple HTTP calls to different servers, especially if the thread is long and complicated.
+
+The end result of this dereferencing is that, assuming the reblogged post by `remote_2` was part of a thread, then `local_account` should now be able to see posts in the thread when they open the status on their home timeline. In other words, they will see replies from accounts on other servers (who they may not have come across yet), in addition to any previous and next posts in the thread as posted by `remote_2`.
+
+This gives `local_account` a more complete view on the conversation, as opposed to just seeing the reblogged post in isolation and out of context. It also gives `local_account` the opportunity to discover new accounts to follow, based on replies to `remote_2`.
+
+## Reports / Flags
+
+Like other microblogging ActivityPub implementations, GoToSocial uses the [Flag](https://www.w3.org/TR/activitystreams-vocabulary/#dfn-flag) Activity type to communicate user moderation reports to other servers.
+
+### Outgoing
+
+The json of an outgoing GoToSocial `Flag` looks like the following:
+
+```json
+{
+  "@context": "https://www.w3.org/ns/activitystreams",
+  "actor": "http://example.org/users/example.org",
+  "content": "dark souls sucks, please yeet this nerd",
+  "id": "http://example.org/reports/01GP3AWY4CRDVRNZKW0TEAMB5R",
+  "object": [
+    "http://fossbros-anonymous.io/users/foss_satan",
+    "http://fossbros-anonymous.io/users/foss_satan/statuses/01FVW7JHQFSFK166WWKR8CBA6M"
+  ],
+  "type": "Flag"
+}
+```
+
+The `actor` of the `Flag` will always be the instance actor of the GoToSocial instance on which the `Flag` was created. This is done to preserve partial anonymity of the user who created the report, in order to prevent them becoming a target for harassment.
+
+The `content` of the `Flag` is a piece of text submitted by the user who created the `Flag`, which should give remote instance admins a reason why the report was created. This may be an empty string, or may not be present on the json, if no reason was submitted by the user.
+
+The value of the `object` field of the `Flag` will either be a string (the ActivityPub `id` of the user being reported), or it will be an array of strings, where the first entry in the array is the `id` of the reported user, and subsequent entries are the `id`s of one or more reported `Note`s / statuses.
+
+The `Flag` activity is delivered as-is to the `inbox` (or shared inbox) of the reported user. It is not wrapped in a `Create` activity.
+
+### Incoming
+
+GoToSocial assumes incoming reports will be delivered as a `Flag` Activity to the `inbox` of the account being reported.  It will parse the incoming `Flag` following the same formula that it uses for creating outgoing `Flag`s, with one difference: it will attempt to parse status URLs from both the `object` field, and from a Misskey/Calckey-formatted `content` value, which includes in-line status URLs.
+
+GoToSocial will not assume that the `to` field will be set on an incoming `Flag` activity. Instead, it assumes that remote instances use `bto` to direct the `Flag` to its recipient.
+
+A valid incoming `Flag` Activity will be made available as a report to the admin(s) of the GoToSocial instance that received the report, so that they can take any necessary moderation action against the reported user.
+
+The reported user themself will not see the report, or be notified that they have been reported, unless the GtS admin chooses to share this information with them via some other channel.
+
+## Featured (aka pinned) Posts
+
+GoToSocial allows users to feature (or 'pin') posts on their profile.
+
+In ActivityPub terms, GoToSocial serves these pinned posts as an [OrderedCollection](https://www.w3.org/TR/activitystreams-vocabulary/#dfn-orderedcollection) at the endpoint indicated in an Actor's [featured](https://docs.joinmastodon.org/spec/activitypub/#featured) field. The value of this field will be set to something like `https://example.org/users/some_user/collections/featured`.
+
+By making a signed GET request to this endpoint, remote instances can dereference the featured posts collection, which will return an `OrderedCollection` with a list of post URIs in the `orderedItems` field.
+
+Example of a featured collection of a user who has pinned multiple `Note`s:
+
+```json
+{
+  "@context": "https://www.w3.org/ns/activitystreams",
+  "id": "https://example.org/users/some_user/collections/featured",
+  "orderedItems": [
+    "https://example.org/users/some_user/statuses/01GS7VTYH0S77NNXTP6W4G9EAG",
+    "https://example.org/users/some_user/statuses/01GSFY2SZK9TPCJFQ1WCCPGDRT",
+    "https://example.org/users/some_user/statuses/01GSCXY70MZCBFMH5EKJW9ENC8"
+  ],
+  "totalItems": 3,
+  "type": "OrderedCollection"
+}
+```
+
+Example of a user who has pinned one `Note`:
+
+```json
+{
+  "@context": "https://www.w3.org/ns/activitystreams",
+  "id": "https://example.org/users/some_user/collections/featured",
+  "orderedItems": [
+    "https://example.org/users/some_user/statuses/01GS7VTYH0S77NNXTP6W4G9EAG"
+  ],
+  "totalItems": 1,
+  "type": "OrderedCollection"
+}
+```
+
+Example with no pinned `Note`s:
+
+```json
+{
+  "@context": "https://www.w3.org/ns/activitystreams",
+  "id": "https://example.org/users/some_user/collections/featured",
+  "orderedItems": [],
+  "totalItems": 0,
+  "type": "OrderedCollection"
+}
+```
+
+Unlike Mastodon and some other implementations, GoToSocial does *not* serve full `Note` representations as `orderedItems` values. Instead, it provides just the URI of each `Note`, which the remote server can then dereference (or not, if they already have the `Note` cached locally).
+
+Some of the URIs served as part of the collection may point to followers-only posts which the requesting `Actor` won't necessarily have permission to view. Remote servers should make sure to do their own filtering (as with any other post type) to ensure that these posts are only shown to users who are permitted to view them.
+
+Another difference between GoToSocial and other server implementations is that GoToSocial does not send updates to remote servers when a post is pinned or unpinned by a user. Mastodon does this by sending [Add](https://www.w3.org/TR/activitypub/#add-activity-inbox) and [Remove](https://www.w3.org/TR/activitypub/#remove-activity-inbox) Activity types where the `object` is the post being pinned or unpinned, and the `target` is the sending `Actor`'s `featured` collection. While this conceptually makes sense, it is not in line with what the ActivityPub protocol recommends, since the `target` of the Activity "is not owned by the receiving server, and thus they can't update it".
+
+Instead, to build a view of a GoToSocial user's pinned posts, it is recommended that remote instances simply poll a GoToSocial Actor's `featured` collection every so often, and add/remove posts in their cached representation as appropriate.
+
+## Post Deletes
+
+GoToSocial allows users to delete posts that they have created. These deletes will be federated out to other instances, which are expected to also delete their local cache of the post.
+
+### Outgoing
+
+When a post is deleted by a GoToSocial user, the server will send a `Delete` activity out to other instances.
+
+The `Delete` will have the ActivityPub URI of the post set as the value of the `Object` entry.
+
+`to` and `cc` will be set according to the visibility of the original post, and any users mentioned/replied to by the original post.
+
+If the original post was not a direct message, the ActivityPub `Public` URI will be addressed in `to`. Otherwise, only mentioned and replied to users will be addressed.
+
+In the following example, the 'admin' user deletes a public post of theirs in which the 'foss_satan' user was mentioned:
+
+```json
+{
+  "@context": "https://www.w3.org/ns/activitystreams",
+  "actor": "http://example.org/users/admin",
+  "cc": [
+    "http://example.org/users/admin/followers",
+    "http://fossbros-anonymous.io/users/foss_satan"
+  ],
+  "object": "http://example.org/users/admin/statuses/01FF25D5Q0DH7CHD57CTRS6WK0",
+  "to": "https://www.w3.org/ns/activitystreams#Public",
+  "type": "Delete"
+}
+```
+
+In the next example, the '1happyturtle' user deletes a direct message which was originally addressed to the 'the_mighty_zork' user.
+
+```json
+{
+  "@context": "https://www.w3.org/ns/activitystreams",
+  "actor": "http://example.org/users/1happyturtle",
+  "cc": [],
+  "object": "http://example.org/users/1happyturtle/statuses/01FN3VJGFH10KR7S2PB0GFJZYG",
+  "to": "http://somewhere.com/users/the_mighty_zork",
+  "type": "Delete"
+}
+```
+
+To process a `Delete` activity coming from a GoToSocial instance, remote instances should check if they have the `Object` stored according to the provided URI. If they do, they should remove it from their local cache. If not, then no action is required, since they never had the now-deleted post stored in the first place.
+
+### Incoming
+
+GoToSocial processes `Delete` activities coming in from remote instances as thoroughly as possible in order to respect the privacy of other users.
+
+When a GoToSocial instance receives a `Delete`, it will attempt to derive the deleted post URI from the `Object` field. If the `Object` is just a URI, then this URI will be taken. If the `Object` is a `Note` or another type commonly used to represent a post, then the URI will be extracted from it.
+
+Then, GoToSocial will check if it has a post stored with the given URI. If it does, it will be completely deleted from the database and all user timelines.
+
+GoToSocial will only delete a post if it can be sure that the original post was owned by the `actor` that the `Delete` is attributed to.
+
+## Profile Fields
+
+Like Mastodon and other fediverse softwares, GoToSocial lets users set key/value pairs on their profile; useful for conveying short pieces of information like links, pronouns, age, etc.
+
+For the sake of compatibility with other implementations, GoToSocial uses the same schema.org PropertyValue extension that Mastodon uses, present as an `attachment` array value on `actor`s that have fields set. For example, the below JSON shows an account with two PropertyValue fields:
+
+```json
+{
+  "@context": [
+    "http://joinmastodon.org/ns",
+    "https://w3id.org/security/v1",
+    "https://www.w3.org/ns/activitystreams",
+    "http://schema.org"
+  ],
+  "attachment": [
+    {
+      "name": "should you follow me?",
+      "type": "PropertyValue",
+      "value": "maybe!"
+    },
+    {
+      "name": "age",
+      "type": "PropertyValue",
+      "value": "120"
+    }
+  ],
+  "discoverable": false,
+  "featured": "http://example.org/users/1happyturtle/collections/featured",
+  "followers": "http://example.org/users/1happyturtle/followers",
+  "following": "http://example.org/users/1happyturtle/following",
+  "id": "http://example.org/users/1happyturtle",
+  "inbox": "http://example.org/users/1happyturtle/inbox",
+  "manuallyApprovesFollowers": true,
+  "name": "happy little turtle :3",
+  "outbox": "http://example.org/users/1happyturtle/outbox",
+  "preferredUsername": "1happyturtle",
+  "publicKey": {
+    "id": "http://example.org/users/1happyturtle#main-key",
+    "owner": "http://example.org/users/1happyturtle",
+    "publicKeyPem": "-----BEGIN PUBLIC KEY-----\nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAtTc6Jpg6LrRPhVQG4KLz\n2+YqEUUtZPd4YR+TKXuCnwEG9ZNGhgP046xa9h3EWzrZXaOhXvkUQgJuRqPrAcfN\nvc8jBHV2xrUeD8pu/MWKEabAsA/tgCv3nUC47HQ3/c12aHfYoPz3ufWsGGnrkhci\nv8PaveJ3LohO5vjCn1yZ00v6osMJMViEZvZQaazyE9A8FwraIexXabDpoy7tkHRg\nA1fvSkg4FeSG1XMcIz2NN7xyUuFACD+XkuOk7UqzRd4cjPUPLxiDwIsTlcgGOd3E\nUFMWVlPxSGjY2hIKa3lEHytaYK9IMYdSuyCsJshd3/yYC9LqxZY2KdlKJ80VOVyh\nyQIDAQAB\n-----END PUBLIC KEY-----\n"
+  },
+  "summary": "\u003cp\u003ei post about things that concern me\u003c/p\u003e",
+  "tag": [],
+  "type": "Person",
+  "url": "http://example.org/@1happyturtle"
+}
+```
+
+For `actor`s that have no `PropertyValue` fields set, the `attachment` property will not be set at all. That is, the `attachment` key value will not be present on the `actor` (not even as an empty array or null value).
+
+While `attachment` is not technically an ordered collection, GoToSocial--again, in line with what other implementations do--does present `attachment` `PropertyValue` fields in the order in which they should to be displayed.
+
+GoToSocial will also parse PropertyValue fields from remote `actor`s discovered by the GoToSocial instance, to allow them to be displayed to users on the GoToSocial instance.
+
+GoToSocial allows up to 6 `PropertyValue` fields by default, as opposed to Mastodon's default 4.
+
+## Hashtags
+
+GoToSocial users can include hashtags in their posts, which indicate to other instances that that user wishes their post to be grouped together with other posts using the same hashtag, for discovery purposes.
+
+In line with other ActivityPub server implementations, GoToSocial implicitly expects that only public-addressed posts will be grouped by hashtag.
+
+To federate hashtags in and out, GoToSocial uses the widely-adopted [ActivityStreams `Hashtag` type extension](https://www.w3.org/wiki/Activity_Streams_extensions#as:Hashtag_type) in the `tag` property of objects.
+
+Here's what the `tag` property might look like on an outgoing message that uses one custom emoji, and one tag:
+
+```json
+"tag": [
+  {
+    "icon": {
+      "mediaType": "image/png",
+      "type": "Image",
+      "url": "https://example.org/fileserver/01AY6P665V14JJR0AFVRT7311Y/emoji/original/01F8MH9H8E4VG3KDYJR9EGPXCQ.png"
+    },
+    "id": "https://example.org/emoji/01F8MH9H8E4VG3KDYJR9EGPXCQ",
+    "name": ":rainbow:",
+    "type": "Emoji",
+    "updated": "2021-09-20T10:40:37Z"
+  },
+  {
+    "href": "https://example.org/tags/welcome",
+    "name": "#welcome",
+    "type": "Hashtag"
+  }
+]
+```
+
+With just one tag, the `tag` property will be an object rather than an array, which will look like this:
+
+```json
+"tag": {
+  "href": "https://example.org/tags/welcome",
+  "name": "#welcome",
+  "type": "Hashtag"
+}
+```
+
+### Hashtag `href` property
+
+The `href` URL provided by GoToSocial in outgoing tags points to a web URL that serves `text/html`.
+
+GoToSocial makes no guarantees whatsoever about what the content of the given `text/html` will be, and remote servers should not interpret the URL as a canonical ActivityPub ID/URI property. The `href` URL is provided merely as an endpoint which *might* contain more information about the given hashtag.

docs/federation/security.md

@@ -1,7 +0,0 @@
-# Security
-
-TODO: describe the security model we use for federation.
-
-* http signatures
-* behavior for refusing requests
-* how data is protected

docs/getting_started/index.md

@@ -0,0 +1,75 @@
+# Deployment considerations
+
+Before deploying GoToSocial, it's important to think through a few things as some choices will have long-term consequences for how you run and manage GoToSocial.
+
+!!! danger
+
+    It's not supported across the Fediverse to switch between implementations on the same domain. This means that if you run GoToSocial on example.org, you'll run into federation issues if you try to switch to a different implementation like Pleroma/Akkoma, Misskey/Calckey etc.
+    
+    In that same vein, if you already have another ActivityPub implementation running on example.org you should not attempt to switch to GoToSocial on that domain.
+
+## Database
+
+GoToSocial supports both SQLite and Postgres and you can start using either. We do not currently have tooling to support migrating from SQLite to Postgres or vice-versa, but it is possible in theory.
+
+SQLite is great for a single-user instance. If you're planning on hosting multiple people it's advisable to use Postgres instead. You can always use Postgres regardless of the instance size.
+
+!!! tip
+    Please [backup  your database](../admin/backup_and_restore.md). The database contains encryption keys for the instance and any user accounts. You won't be able to federate again from the same domain if you lose these keys.
+
+## Domain name
+
+In order to federate with others, you'll need a domain like `example.org`. You can register your domain name through any domain registrar, like [Namecheap](https://www.namecheap.com/). Make sure you pick a registrar that also lets you manage DNS entries, so you can point your domain to the IP of the server that's running your GoToSocial instance.
+
+You'll commonly see usernames existing at the apex of the domain, for example `@me@example.org` but this is not required. It's perfectly fine to have users exist on `@me@social.example.org` instead. Many people prefer to have usernames on the apex as its shorter to type, but you can use any (subdomain) you control.
+
+It is possible to have usernames like `@me@example.org` but have GoToSocial running on `social.example.org` instead. This is done by distinguishing between the API domain, called the "host", and the domain used for usernames, called the "account domain".
+
+!!! danger
+    It's not possible to safely change whether the host and account domain are different after the fact. It requires regenerating the database and will cause confusion for any server you have already federated with.
+
+When using a single domain, you only need to configure the "host" in the GoToSocial configuration:
+
+```yaml
+host: "example.org"
+```
+
+When using a split domain approach, you need to configure both the "host" and the "account-domain":
+
+```yaml
+host: "social.example.org"
+account-domain: "example.org"
+```
+
+## TLS
+
+For federation to work, you have to use TLS. Most implementations, including GoToSocial, will generally refuse to federate over unencrypted transports.
+
+GoToSocial comes with built-in support for provisioning certificates through Lets Encrypt. It can also load certificates from disk. If you have a reverse-proxy in front of GoToSocial you can handle TLS at that level instead.
+
+!!! tip
+    Make sure you configure the use of modern versions of TLS, TLSv1.2 and higher, in order to keep communications between servers and clients safe. When GoToSocial handles TLS termination this is done automatically for you. If you have a reverse-proxy in use, use the [Mozilla SSL Configuration Generator](https://ssl-config.mozilla.org/).
+
+## Server / VPS
+
+GoToSocial aims to fit in small spaces so we try and ensure that the system requirements are fairly minimal: for a single-user instance with about 100 followers/followees, it uses somewhere between 50 to 100MB of RAM. CPU usage is only intensive when handling media (encoding blurhashes, mostly) and/or doing a lot of federation requests at the same time.
+
+These light requirements mean GtS runs pretty well on something like a Raspberry Pi (a €40 single-board computer). It's been tested on a Raspberry Pi Zero W as well (a €9 computer smaller than a credit card), but it's not quite able to run on that. It should run on a Raspberry Pi Zero W 2 (which costs €14!), but we haven't tested that yet. You can also repurpose an old laptop or desktop to run GoToSocial for you.
+
+If you decide to use a VPS instead, you can spin yourself up something cheap with Linux running on it. Most of the VPS offerings in the €2-€5 range will perform admirably for a personal GoToSocial instance.
+
+[Hostwinds](https://www.hostwinds.com/) is a good option here: it's cheap and they throw in a static IP address for free.
+
+[Greenhost](https://greenhost.net) is also great: it has zero CO2 emissions, but is a bit more costly.
+
+## Ports
+
+GoToSocial needs ports `80` and `443` open.
+
+* `80` is used for Lets Encrypt. As such, you don't need it if you don't use the built-in Lets Encrypt provisioning.
+* `443` is used to serve the API on with TLS and is what any instance you're federating with will try to connect to.
+
+If you can't leave `443` and `80` open on the machine, don't worry! You can configure these ports in GoToSocial, but you'll have to also configure port forwarding to properly forward traffic on `443` and `80` to whatever ports you choose.
+
+!!! tip
+    You should configure a firewall on your machine, as well as some protection against brute-force SSH login attempts and the like. Take a look at our [firewall documentation](../advanced/security/firewall.md) for pointers on what to configure and tools that can help you out.

docs/getting_started/installation/container.md

@@ -0,0 +1,123 @@
+# Container
+
+This guide walks you through getting GoToSocial up and running using the official container images we publish. In this case we'll be using the Docker runtime directly through [Docker Compose](https://docs.docker.com/compose) together with SQLite as the database.
+
+You can also run GoToSocial using a container orchestration system such as [Kubernetes](https://kubernetes.io/) or [Nomad](https://www.nomadproject.io/), but that is beyond the scope of this guide.
+
+## Create a Working Directory
+
+You need a working directory in which your docker-compose file will be located, and a directory for GoToSocial to store data in, so create these directories with the following command:
+
+```bash
+mkdir -p ~/gotosocial/data
+```
+
+Now change to the working directory you created:
+
+```bash
+cd ~/gotosocial
+```
+
+## Get the latest docker-compose.yaml
+
+Use `wget` to download the latest [docker-compose.yaml](https://raw.githubusercontent.com/superseriousbusiness/gotosocial/main/example/docker-compose/docker-compose.yaml) example, which we'll customize for our needs:
+
+```bash
+wget https://raw.githubusercontent.com/superseriousbusiness/gotosocial/main/example/docker-compose/docker-compose.yaml
+```
+
+## Edit the docker-compose.yaml
+
+Because GoToSocial can be configured using [Environment Variables](../../configuration/index.md#environment-variables), we can skip mounting a config.yaml file into the container, to make our configuration simpler. We just need to edit the docker-compose.yaml file to change a few things.
+
+First open the docker-compose.yaml file in your editor of choice. For example:
+
+```bash
+nano docker-compose.yaml
+```
+
+### Version
+
+If desired, update the GoToSocial Docker image tag to the version of GtS you want to use.
+
+`latest`   - the default. This points to the latest stable release of GoToSocial.
+
+`snapshot` - points to whatever code is currently on the main branch. Not guaranteed to be stable, and may often be broken. Use with caution.
+
+!!! tip
+    You can also replace `latest` with a specific GoToSocial version number. This is recommended when you want to make sure that you don't update your GoToSocial version by accident, which can cause problems. The list of releases can be found [right here](https://github.com/superseriousbusiness/gotosocial/releases), with the newest release at the top. Replace `latest` in the docker-compose.yaml with the number of the desired release (without the leading `v` or trailing version name).
+
+### Host
+
+Change the `GTS_HOST` environment variable to the domain you are running GoToSocial on.
+
+### Server Timezone (optional but recommended)
+
+To ensure that your GoToSocial server displays the correct time on posts and in logs, you can set the timezone of the server by editing the `TZ` environment variable.
+
+1. Remove the `#` before `TZ: UTC` in the environment section.
+2. Change the `UTC` part to your timezone identifier. For a list of these identifiers, see https://en.wikipedia.org/wiki/List_of_tz_database_time_zones.
+
+For example, if you are running your server in Minsk, you would set `TZ: Europe/Minsk`, Japan would be `TZ: Japan`, Dubai would be `TZ: Asia/Dubai`, etc.
+
+If you don't set this, the default `UTC` will be used.
+
+### User (optional / probably not necessary)
+
+By default, Dockerized GoToSocial runs with Linux user/group `1000:1000`, which is fine in most cases. If you want to run as a different user/group, you should change the `user` field in the docker-compose.yaml accordingly.
+
+For example, let's say you created the `~/gotosocial/data` directory for a user with id `1001`, and group id `1001`. If you now try to run GoToSocial without changing the `user` field, it will get a permissions error trying to open its database file in the directory. In this case, you would have to change the `user` field of the docker compose file to `1001:1001`.
+
+### LetsEncrypt (optional)
+
+If you want to use [LetsEncrypt](../../configuration/tls.md) for TLS certificates (https), you should also:
+
+1. Change the value of `GTS_LETSENCRYPT_ENABLED` to `"true"`.
+2. Remove the `#` before `- "80:80"` in the `ports` section.
+3. (Optional) Set `GTS_LETSENCRYPT_EMAIL_ADDRESS` to a valid email address to receive certificate expiry warnings etc.
+
+## Start GoToSocial
+
+With those small changes out of the way, you can now start GoToSocial with the following command:
+
+```shell
+docker-compose up -d
+```
+
+After running this command, you should get an output like:
+
+```text
+Creating network "gotosocial_gotosocial" with the default driver
+Creating gotosocial ... done
+```
+
+If you want to follow the logs of GoToSocial, you can use:
+
+```bash
+docker logs -f gotosocial
+```
+
+If everything is OK, you should see something similar to the following:
+
+```text
+time=2022-04-19T09:48:35Z level=info msg=connected to SQLITE database
+time=2022-04-19T09:48:35Z level=info msg=MIGRATED DATABASE TO group #1 (20211113114307, 20220214175650, 20220305130328, 20220315160814) func=doMigration
+time=2022-04-19T09:48:36Z level=info msg=instance account example.org CREATED with id 01EXX0TJ9PPPXF2C4N2MMMVK50
+time=2022-04-19T09:48:36Z level=info msg=created instance instance example.org with id 01PQT31C7BZJ1Q2Z4BMEV90ZCV
+time=2022-04-19T09:48:36Z level=info msg=media manager cron logger: start[]
+time=2022-04-19T09:48:36Z level=info msg=media manager cron logger: schedule[now 2022-04-19 09:48:36.096127852 +0000 UTC entry 1 next 2022-04-20 00:00:00 +0000 UTC]
+time=2022-04-19T09:48:36Z level=info msg=started media manager remote cache cleanup job: will run next at 2022-04-20 00:00:00 +0000 UTC
+time=2022-04-19T09:48:36Z level=info msg=listening on 0.0.0.0:8080
+```
+
+## Create your first User
+
+Now that GoToSocial is running, you should create at least a user for yourself. How to do so is documented in our [Creating users](../user_creation.md) guide.
+
+### Done
+
+GoToSocial should now be running on your machine! To verify this, open your browser navigate to whatever you set as your `GTS_HOST` value. You should see the GoToSocial landing page. Well done!
+
+## (Optional) Reverse Proxy
+
+If you want to run other webservers on port 443 or want to add an additional layer of security you might want to use a [reverse proxy](../reverse_proxy/index.md). We have guides available for a couple of popular open source options and will gladly take pull requests to add more.

docs/getting_started/installation/index.md

@@ -0,0 +1,14 @@
+# Installation
+
+As we noted in [Releases](../releases.md), we publish official binary release as well as containers. We have a number of guides available on how to deploy your own GoToSocial instance this way.
+
+Before proceeding with your installation, please ensure you've read through the [Deployment considerations](../index.md) first and have a domain and server ready to go.
+
+Also take a minute to familiarise yourself with [how to configure](../../configuration/index.md) GoToSocial.
+
+## Guides
+
+For third-party releases we don't provide guides on how to use them. You need to refer to their own documentation instead. Our guides might still be useful to review in order to familiarise yourself with which configuration options you likely want to set and tweak.
+
+* [Bare metal](metal.md)
+* [Container](container.md)

docs/getting_started/installation/metal.md

@@ -0,0 +1,123 @@
+# Bare metal
+
+This guide walks you through getting GoToSocial up and running on bare metal using the official binary releases.
+
+## Prepare VPS
+
+In a terminal on the VPS or your homeserver, make the directory that GoToSocial will run from, the directory it will use as storage, and the directory it will store LetsEncrypt certificates in:
+
+```bash
+mkdir /gotosocial && mkdir /gotosocial/storage && mkdir /gotosocial/storage/certs
+```
+
+If you don't have root permissions on the machine, use something like `~/gotosocial` instead.
+
+## Download Release
+
+In a terminal on the VPS or your homeserver, cd into the base directory for GoToSocial you just created above:
+
+```bash
+cd /gotosocial
+```
+
+Now, download the latest GoToSocial release archive corresponding to the operating system and architecture you're running on.
+
+(You can find the list of releases [right here](https://github.com/superseriousbusiness/gotosocial/releases), arranged with the newest release at the top.)
+
+For example, to download version 0.5.2 for running on 64-bit Linux:
+
+```bash
+wget https://github.com/superseriousbusiness/gotosocial/releases/download/v0.5.2/gotosocial_0.5.2_linux_amd64.tar.gz
+```
+
+Then extract it:
+
+```bash
+tar -xzf gotosocial_0.5.2_linux_amd64.tar.gz
+```
+
+This will put the `gotosocial` binary in your current directory, in addition to the `web` folder, which contains assets for the web frontend, and an `example` folder, which contains a sample configuration file.
+
+## Edit Configuration File
+
+Copy the configuration file from the example folder into your current directory:
+
+```bash
+cp ./example/config.yaml .
+```
+
+Now open the file in your text editor of choice so that you can set some important configuration values. Change the following settings:
+
+- Set `host` to whatever hostname you're going to be running the server on (eg., `example.org`).
+- Set `port` to `443`.
+- Set `db-type` to `sqlite`.
+- Set `db-address` to `sqlite.db`.
+- Set `storage-local-base-path` to the storage directory you created above (eg., `/gotosocial/storage`).
+- Set `letsencrypt-enabled` to `true`.
+- Set `letsencrypt-cert-dir` to the certificate storage directory you created above (eg., `/gotosocial/storage/certs`).
+
+The above options assume you're using SQLite as your database. If you want to use Postgres instead, see [here](../../configuration/database.md) for the config options.
+
+## Run the Binary
+
+You can now run the binary.
+
+Start the GoToSocial server with the following command:
+
+```bash
+./gotosocial --config-path ./config.yaml server start
+```
+
+The server should now start up and you should be able to access the splash page by navigating to your domain in the browser. Note that it might take up to a minute or so for your LetsEncrypt certificates to be created for the first time, so refresh a few times if necessary.
+
+Note that for this example we're assuming that we're allowed to run on port 443 (standard https port), and that nothing else is running on this port.
+
+## Create your user
+
+You can use the GoToSocial binary to also create and promote your user account. This is all documented in our [Creating users](../user_creation.md) guide.
+
+## Login
+
+You should now be able to log in to your instance using the email address and password of the account you just created. We recommend using [Semaphore](https://semaphore.social) or [Tusky](https://tusky.app) for this.
+
+## (Optional) Enable the systemd service
+
+If you don't like manually starting GoToSocial on every boot you might want to create a systemd service that does that for you.
+
+First stop your GoToSocial instance.
+
+Then create a new user and group for your GoToSocial installation:
+
+```bash
+sudo useradd -r gotosocial
+sudo groupadd gotosocial
+sudo usermod -a -G gotosocial gotosocial
+```
+
+Then make them the owner of your GoToSocial installation since they will need to read and write in it:
+
+```bash
+sudo chown -R gotosocial:gotosocial /gotosocial
+```
+
+You can find a `gotosocial.service` file in the `example` folder on [github](https://raw.githubusercontent.com/superseriousbusiness/gotosocial/main/example/gotosocial.service) or your installation.
+
+Copy it to `/etc/systemd/system/gotosocial.service`:
+
+```bash
+sudo cp /gotosocial/example/gotosocial.service /etc/systemd/system/
+```
+
+Then use `sudoedit /etc/systemd/system/gotosocial.service` to change the `ExecStart=` and `WorkingDirectory=` lines according to your installation.
+
+If you have been following this guide word for word the defaults should be fine.
+
+After you're done enable the service:
+
+```bash
+sudo systemctl enable --now gotosocial.service
+```
+
+## (Optional) Reverse proxy
+
+If you want to run other webservers on port 443 or want to add an additional layer of security you might want to use a [reverse proxy](../reverse_proxy/index.md). We have guides available for a couple of popular open source options and will gladly take pull requests to add more.

docs/getting_started/releases.md

@@ -0,0 +1,40 @@
+# Releases
+
+GoToSocial can be installed in a number of different ways. We publish official binary releases as well as container images. A number of third-party packages are maintained by different distributions and some people have created additional deployment tooling to make it easy to deploy GoToSocial yourself.
+
+## Binary releases
+We publish binary builds for Linux to [our GitHub project](https://github.com/superseriousbusiness/gotosocial/releases):
+
+* 32-bit Intel/AMD (i386/x86)
+* 64-bit Intel/AMD (amd64/x86_64)
+* 32-bit ARM (v6 and v7)
+* 64-bit ARM64
+
+For FreeBSD we publish:
+
+* 64-bit Intel/AMD (amd64/x86_64)
+
+## Containers
+
+We also publish container images [on the Docker Hub](https://hub.docker.com/r/superseriousbusiness/gotosocial).
+
+Containers are released for the same Linux platforms as our binary releases, with the exception of 32-bit Intel/AMD.
+
+## Third-party
+
+Some folks have created distribution packages for GoToSocial or additional tooling to aid in installing GoToSocial.
+
+### Distribution packages
+
+These packages are not maintained by GoToSocial, so please direct questions and issues to the repository maintainers (and donate to them!).
+
+[![Packaging status](https://repology.org/badge/vertical-allrepos/gotosocial.svg)](https://repology.org/project/gotosocial/versions)
+
+### Deployment tools
+
+You can deploy your own instance of GoToSocial with the help of:
+
+- [YunoHost GoToSocial Packaging](https://github.com/YunoHost-Apps/gotosocial_ynh) by [OniriCorpe](https://github.com/OniriCorpe).
+- [Ansible Playbook (MASH)](https://github.com/mother-of-all-self-hosting/mash-playbook): The playbook supports a many services, including GoToSocial. [Documentation](https://github.com/mother-of-all-self-hosting/mash-playbook/blob/main/docs/services/gotosocial.md)
+- GoToSocial Helm Charts:
+  - [GoToSocial Helm Chart](https://github.com/fSocietySocial/charts/tree/main/charts/gotosocial) by [0hlov3](https://github.com/0hlov3).

docs/getting_started/reverse_proxy/apache-httpd.md

@@ -0,0 +1,248 @@
+# Apache HTTP Server
+
+In order to use Apache as a reverse proxy for GoToSocial you'll need to have it installed on your server. If you intend for the Apache instance to also handle TLS, you'll need to [provision TLS certificates](../../advanced/certificates.md) too.
+
+Apache is [packaged for many distributions](https://repology.org/project/apache/versions). It's very likely you can install it with your distribution's package manager. You can also run Apache using a container runtime with the [official Apache image](https://hub.docker.com/_/httpd) that's published to the Docker Hub.
+
+In this guide we'll also show how to use certbot to provision the TLS certificates. It too is [packaged in many distributions](https://repology.org/project/certbot/versions) but many distributions tend to ship fairly old versions of certbot. If you run into trouble it may be worth considering using the [container image](https://hub.docker.com/r/certbot/certbot) instead.
+
+## Configure GoToSocial
+
+We're going to have Apache handle LetsEncrypt certificates, so you need to turn off built-in LetsEncrypt support in your GoToSocial config.
+
+First open the file in your text editor:
+
+```bash
+sudoedit /gotosocial/config.yaml
+```
+
+Then set `letsencrypt-enabled: false`.
+
+If the reverse proxy will be running on the same machine, set the `bind-address` to `"localhost"` so that the GoToSocial server is only accessible via loopback. Otherwise it may be possible to bypass your proxy by connecting to GoToSocial directly, which might be undesirable.
+
+If GoToSocial is already running, restart it.
+
+```bash
+sudo systemctl restart gotosocial.service
+```
+
+Or if you don't have a systemd service just restart it manually.
+
+## Set up Apache
+
+### Required Apache modules
+
+You need to ensure you have a number of Apache modules installed and enabled. All these modules *should* ship with your distribution's Apache package, but they may have been split out into separate packages.
+
+You can check which modules you have installed with `apachectl -M`.
+
+You'll need to have the following modules loaded:
+
+* `proxy_http` to proxy HTTP requests to GoToSocial
+* `ssl` to handle SSL/TLS
+* `headers` to manipulate HTTP request and response headers
+* `rewrite` to rewrite HTTP requests
+* `md` for Lets Encrypt, available since 2.4.30
+
+On Debian, Ubuntu and openSUSE, you can use the [`a2enmod`](https://manpages.debian.org/bookworm/apache2/a2enmod.8.en.html) utility to load any additional modules. For the Red Hat/CentOS family of distributions, you'll need to add a [`LoadModule` directive](https://httpd.apache.org/docs/2.4/mod/mod_so.html#loadmodule) to your Apache configuration instead.
+
+### TLS with mod_md
+
+!!! note
+    `mod_md` is only available since Apache 2.4.30 and still considered experimental. It works well enough in practice and is the most convenient method.
+
+Now we'll configure Apache HTTP Server to serve GoToSocial requests.
+
+First we'll write a configuration for Apache HTTP Server and put it in `/etc/apache2/sites-available`:
+
+```bash
+sudo mkdir -p /etc/apache2/sites-available/
+sudoedit /etc/apache2/sites-available/example.com.conf
+```
+
+In the above `sudoedit` command, replace `example.com` with the hostname of your GoToSocial server.
+
+The file you're about to create should look a bit like this:
+
+```apache
+MDomain example.com auto
+MDCertificateAgreement accepted
+
+<VirtualHost *:80 >
+  ServerName example.com
+</VirtualHost>
+
+<VirtualHost *:443>
+  ServerName example.com
+
+  RewriteEngine On
+  RewriteCond %{HTTP:Upgrade} websocket [NC]
+  RewriteCond %{HTTP:Connection} upgrade [NC]
+  # set to 127.0.0.1 instead of localhost to work around https://stackoverflow.com/a/52550758
+  RewriteRule ^/?(.*) "ws://127.0.0.1:8080/$1" [P,L]
+
+  SSLEngine On
+  ProxyPreserveHost On
+  # set to 127.0.0.1 instead of localhost to work around https://stackoverflow.com/a/52550758
+  ProxyPass / http://127.0.0.1:8080/
+  ProxyPassReverse / http://127.0.0.1:8080/
+
+  RequestHeader set "X-Forwarded-Proto" expr=https
+</VirtualHost>
+```
+
+or, if you have [Apache httpd 2.4.47+](https://httpd.apache.org/docs/2.4/mod/mod_proxy.html#protoupgrade), you can get rid of both `mod_rewrite` and `mod_proxy_wstunnel` and simplify the whole config to:
+
+```apache
+MDomain example.com auto
+MDCertificateAgreement accepted
+
+<VirtualHost *:80 >
+  ServerName example.com
+</VirtualHost>
+
+<VirtualHost *:443>
+  ServerName example.com
+
+  SSLEngine On
+  ProxyPreserveHost On
+  # set to 127.0.0.1 instead of localhost to work around https://stackoverflow.com/a/52550758
+  ProxyPass / http://127.0.0.1:8080/ upgrade=websocket
+  ProxyPassReverse / http://127.0.0.1:8080/
+
+  RequestHeader set "X-Forwarded-Proto" expr=https
+</VirtualHost>
+```
+
+Again, replace occurrences of `example.com` in the above config file with the hostname of your GtS server. If your domain name is `gotosocial.example.com`, then `gotosocial.example.com` would be the correct value.
+
+You should also change `http://127.0.0.1:8080` to the correct address and port (if it's not on `127.0.0.1:8080`) of your GtS server. For example, if you're running GoToSocial on another machine with the local ip of `192.168.178.69` and on port `8080` then `http://192.168.178.69:8080/` would be the correct value.
+
+`Rewrite*` directives are needed to ensure that Websocket streaming connections also work. See the [websocket](./websocket.md) document for more information on this.
+
+`ProxyPreserveHost On` is essential: It guarantees that the proxy and the GoToSocial speak of the same Server name. If not, GoToSocial will build the wrong authentication headers, and all attempts at federation will be rejected with 401 Unauthorized.
+
+By default, apache sets `X-Forwarded-For` in forwarded requests. To make this and rate limiting work, set the `trusted-proxies` configuration variable. See the [rate limiting](../../api/ratelimiting.md) and [general configuration](../../configuration/general.md) docs
+
+Save and close the config file.
+
+Now we'll need to link the file we just created to the folder that Apache HTTP Server reads configurations for active sites from.
+
+```bash
+sudo mkdir /etc/apache2/sites-enabled
+sudo ln -s /etc/apache2/sites-available/example.com.conf /etc/apache2/sites-enabled/
+```
+
+In the above `ln` command, replace `example.com` with the hostname of your GoToSocial server.
+
+Now check for configuration errors.
+
+```bash
+sudo apachectl -t
+```
+
+If everything is fine you should get this as output:
+
+```text
+Syntax OK
+```
+
+Everything working? Great! Then restart Apache HTTP Server to load your new config file.
+
+```bash
+sudo systemctl restart apache2
+```
+
+Now, monitor the logs to see when the new LetsEncrypt certificate arrives (`tail -F /var/log/apache2/error.log`), and then reload Apache one last time with the above `systemctl restart` command. After that you should be good to go!
+
+Apache HTTP Server needs to be restart (or reloaded), every time `mod_md` gets a new certificate; see the module's docs for [more information](https://github.com/icing/mod_md#how-to-manage-server-reloads).
+
+Depending on your version of Apache HTTP Server, you may see the following error: `error (specific information not available): acme problem urn:ietf:params:acme:error:invalidEmail: Error creating new account :: contact email "webmaster@localhost" has invalid domain : Domain name needs at least one dot`
+
+If this happens, you'll need to do one (or all) of the below:
+
+1. Update `/etc/apache2/sites-enabled/000-default.conf` and change the `ServerAdmin` value to a valid email address (then reload Apache HTTP Server).
+2. Add the line `MDContactEmail your.email.address@whatever.com` below the `MDomain` line in `/etc/apache2/sites-available/example.com.conf`, replacing `your.email.address@whatever.com` with a valid email address, and `example.com` with your GtS host name.
+
+### TLS with externally managed certificates
+
+!!! note
+    We have additional documentation on how to [provision TLS certificates](../../advanced/certificates.md) that also provides links to additional content and tutorials for different distributions that may be good to review.
+
+If you prefer to have a manual setup or setting SSL using a different service to manage it (Certbot, etc), then you can use a simpler setup for your Apache HTTP Server.
+
+First we'll write a configuration for Apache HTTP Server and put it in `/etc/apache2/sites-available`:
+
+```bash
+sudo mkdir -p /etc/apache2/sites-available/
+sudoedit /etc/apache2/sites-available/example.com.conf
+```
+
+In the above `sudoedit` command, replace `example.com` with the hostname of your GoToSocial server.
+
+The file you're about to create should look initially for both 80 (required) and 443 ports (optional) a bit like this:
+
+```apache
+<VirtualHost *:80>
+  ServerName example.com
+
+  RewriteEngine On
+  RewriteCond %{HTTP:Upgrade} websocket [NC]
+  RewriteCond %{HTTP:Connection} upgrade [NC]
+  # set to 127.0.0.1 instead of localhost to work around https://stackoverflow.com/a/52550758
+  RewriteRule ^/?(.*) "ws://127.0.0.1:8080/$1" [P,L]
+
+  ProxyPreserveHost On
+  # set to 127.0.0.1 instead of localhost to work around https://stackoverflow.com/a/52550758
+  ProxyPass / http://127.0.0.1:8080/
+  ProxyPassReverse / http://127.0.0.1:8080/
+
+</VirtualHost>
+```
+
+Again, replace occurrences of `example.com` in the above config file with the hostname of your GtS server. If your domain name is `gotosocial.example.com`, then `gotosocial.example.com` would be the correct value.
+
+You should also change `http://127.0.0.1:8080` to the correct address and port (if it's not on `127.0.0.1:8080`) of your GtS server. For example, if you're running GoToSocial on another machine with the local ip of `192.168.178.69` and on port `8080` then `http://192.168.178.69:8080/` would be the correct value.
+
+`Rewrite*` directives are needed to ensure that Websocket streaming connections also work. See the [websocket](websocket.md) document for more information on this.
+
+`ProxyPreserveHost On` is essential: It guarantees that the proxy and the GoToSocial speak of the same Server name. If not, GoToSocial will build the wrong authentication headers, and all attempts at federation will be rejected with 401 Unauthorized.
+
+In the case of providing an initial setup for the 443 port looking for additional managing by an external tool, you could use default certificates provided by the server which you can find referenced in the `default-ssl.conf` file at `/etc/apache2/sites-available/`.
+
+Save and close the config file.
+
+Now we'll need to link the file we just created to the folder that Apache HTTP Server reads configurations for active sites from.
+
+```bash
+sudo mkdir /etc/apache2/sites-enabled
+sudo ln -s /etc/apache2/sites-available/example.com.conf /etc/apache2/sites-enabled/
+```
+
+In the above `ln` command, replace `example.com` with the hostname of your GoToSocial server.
+
+Now check for configuration errors.
+
+```bash
+sudo apachectl -t
+```
+
+If everything is fine you should get this as output:
+
+```text
+Syntax OK
+```
+
+Everything working? Great! Then restart Apache HTTP Server to load your new config file.
+
+```bash
+sudo systemctl restart apache2
+```
+
+## Troubleshooting
+
+If you cannot connect to the site in your browser, the reverse proxy setup doesn't work. Compare the Apache log file (`tail -F /var/log/apache2/access.log`) with the GoToSocial log file. Requests made must show up in both places. Double check the `ProxyPass` setting.
+
+If you can connect but your posts don't federate and your account cannot be found from elsewhere, check your logs. Federation is broken if you see messages attempting to read your profile (something like `level=INFO … method=GET statusCode=401 path=/users/your_username msg="Unauthorized: …"`) or post to your inbox (something like `level=INFO … method=POST statusCode=404 path=/your_username/inbox msg="Not Found: …"`). Double check the `ProxyPreserveHost` setting.
+
+If you can connect but you cannot authorize your account in a Mastodon client app, check your headers. Use `curl -I https://example.com` and look for the `Content-Security-Policy` header. If your webserver sets it, you might have to unset it. One way to do that is to use `Header unset Content-Security-Policy` in the Apache site config file (something like `example.com.conf`).

docs/getting_started/reverse_proxy/caddy.md

@@ -1,4 +1,4 @@
-# Reverse proxy with Caddy 2
+# Caddy 2
 
 ## Requirements
 
@@ -49,6 +49,8 @@ In your GoToSocial config turn off Lets Encrypt by setting `letsencrypt-enabled`
 
 If you we running GoToSocial on port 443, change the `port` value back to the default `8080`.
 
+If the reverse proxy will be running on the same machine, set the `bind-address` to `"localhost"` so that the GoToSocial server is only accessible via loopback. Otherwise it may be possible to bypass your proxy by connecting to GoToSocial directly, which might be undesirable.
+
 ## Set up Caddy
 
 We will configure Caddy 2 to use GoToSocial on our main domain example.org. Since Caddy takes care of obtaining the Lets Encrypt certificate, we only need to configure it properly once.
@@ -77,7 +79,7 @@ example.org {
 }
 ```
 
-By default, caddy sets `X-Forwarded-For` in forwarded requests. To make this and rate limiting work, set the `trusted-proxies` configuration variable. See the [rate limiting](../api/ratelimiting.md) and [general configuration](../configuration/general.md) docs
+By default, caddy sets `X-Forwarded-For` in forwarded requests. To make this and rate limiting work, set the `trusted-proxies` configuration variable. See the [rate limiting](../../api/ratelimiting.md) and [general configuration](../../configuration/general.md) docs
 
 For advanced configuration check the [reverse_proxy directive](https://caddyserver.com/docs/caddyfile/directives/reverse_proxy) at the Caddy documentation.