This adds a section to the docs instructing how to ensure apps will be
able to successfully login to an instance when host- and account-domain
differ.
Resolves#1609
* [feature] Allow admins to send test emails
* implement unwrap on new error type
* add + use gtserror types
* GoToSocial Email Test -> GoToSocial Test Email
* add + use getInstance db call
* removed unused "unknown" error type
* [chore] Remove years from all license headers
Years or year ranges aren't required in license headers. Many projects
have removed them in recent years and it avoids a bit of yearly toil.
In many cases our copyright claim was also a bit dodgy since we added
the 2021-2023 header to files created after 2021 but you can't claim
copyright into the past that way.
* [chore] Add license header check
This ensures a license header is always added to any new file. This
avoids maintainers/reviewers needing to remember to check for and ask
for it in case a contribution doesn't include it.
* [chore] Add missing license headers
* [chore] Further updates to license header
* Use the more common // indentend comment format
* Remove the hack we had for the linter now that we use the // format
* Add SPDX license identifier
* [feature] Provide .well-known/host-meta endpoint
This adds the host-meta endpoint as Mastodon clients use this to
discover the API domain to use when the host and account domains aren't
the same.
* Address review comments
Currently, GtS only supports using the built-in LE client directly for
TLS. However, admins may still want to use GtS directly (so without a
reverse proxy) but with certificates provided through some other
mechanism. They may have some centralised way of provisioning these
things themselves, or simply prefer to use LE but with a different
challenge like DNS-01 which is not supported by autocert.
This adds support for loading a public/private keypair from disk instead
of using LE and reconfigures the server to use a TLS listener if we
succeed in doing so.
Additionally, being able to load TLS keypair from disk opens up the path
to using a custom CA for testing purposes avoinding the need for a
constellation of containers and something like Pebble or Step CA to
provide LE APIs.
* Advertise rich text formats, support content_type field
* Update JSON in instance patch tests
* Replace format with content_type everywhere
* update migration to work with both pg and sqlite
* regenerate swagger docs
* update instance serialization + tests
* fix up
* learn to code tobi please, i'm begging you
---------
Co-authored-by: tsmethurst <tobi.smethurst@protonmail.com>
* start fiddling
* the ol' fiddle + update
* start working on fetching statuses
* poopy doopy doo where r u uwu
* further adventures in featuring statuses
* finishing up
* fmt
* simply status unpin loop
* move empty featured check back to caller function
* remove unnecessary log.WithContext calls
* remove unnecessary IsIRI() checks
* add explanatory comment about status URIs
* change log level to error
* better test names
This removes the current default of checking for membership of the admin
or admins group and makes it required to explicitly configure which
groups should grant admin access, if any.
Relying on the implicit default of admin or admins is potentially
dangerous as that group may contain a different subset of people that we
may wish to grant admin access to GtS. This is probably not an issue for
a single-person instance, but for a community instance different admin
groups may exist in an OIDC provider for different applications.
I'm explicitly opting for not defaulting the value of oidc-admin-groups
to admin,admins because I think it's better for those things to be
explicitly configured.
* implement status pin client api + web handler
* make test names + comments more descriptive
* don't use separate table for status pins
* remove unused add + remove checking
* tidy up + add some more tests
* [feature] Use Material theme for docs
This changes the theme to use Material with the slate (dark) theme and
the accent colour set to orange. It also replaces the swagger plugin
with one that works correctly with the Material theming.
* Add theme by @f0x52
Co-authored-by: f0x52 <f0x@cthu.lu>
* Enable social cards
The dependencies for cairosvg and pillow can be satisfied on RTD, so
this adds them to our requirements.txt and enables the social plugin.
This generates opengraph and twitter card tags as well as fancy preview
images in GTS colour style for platforms that show that as part of link
previews.
* Update Conda environment.yml
* update theme for swagger, basic light theme option
* dark mode mobile header bg color
---------
Co-authored-by: f0x52 <f0x@cthu.lu>
This adds a lightweight form of tracing to GTS. Each incoming request is
assigned a Request ID which we then pass on and log in all our log
lines. Any function that gets called downstream from an HTTP handler
should now emit a requestID=value pair whenever it logs something.
Co-authored-by: kim <grufwub@gmail.com>
* [bug] Fix nginx fileserver caching example
This updates the example to ensure the nginx proxies the request on to
GTS if the file is not found on disk. This can happen due to media
pruning.
* [chore] Set cache-control in nginx to private
This makes the header match with the backend. For things from the
fileserver it may not be appropriate for anything other than a private
cache (i.e the client) to cache things.
* serve publickey separately from AP, don't throttle it
* update nginx cache documentation, cache main-key too
* throttle public key, but separately from other endpoints
since 2.4.47 (released April 22nd 2021), Apache httpd can ProxyPass to
websockets on the same URL, without mod_rewrite (and, without
mod_proxy_wstunnel).
* [chore] set max open / idle conns + conn max lifetime for both postgres and sqlite
* reduce cache size default to 8MiB, reduce connections to 2 * cpu
* introduce max open conns multiplier, tune sqlite and pg separately
* go fmt
* start adding report client api
* route + test reports get
* start report create endpoint
* you can create reports now babyy
* stub account report processor
* add single reportGet endpoint
* fix test
* add more filtering params to /api/v1/reports GET
* update swagger
* use marshalIndent in tests
* add + test missing Link info
* Add local user and post count to nodeinfo responses
This fixes#1307 (at least partially). The nodeinfo endpoint should now
return the total users on an instance, along with their post count.
* Update NodeInfoUsers docstring and swagger yaml file
* AWS S3 config details added
It was interesting to note that since presigned urls are used buckets dont need to be exposed publically. this was an interesting change compared to other mastodon specific s3 bucket guides hence documented here for correct directions.
* Update storage.md
1. Added AWS identified to make it clear its aws specific.
2. Adjusted text around data migration
* updation as requested
Refining the doc as per request.
* [feature] Add throttling middleware to AP endpoints
* refactor a lil bit
* use config setting, start updating docs
* doc updates
* use relative links in faq doc
* small docs fixes
* return code 503 instead of 429 when throttled
* throttle other endpoints too
* simplify token channel prefills
* interim commit: start refactoring middlewares into package under router
* another interim commit, this is becoming a big job
* another fucking massive interim commit
* refactor bookmarks to new style
* ambassador, wiz zeze commits you are spoiling uz
* she compiles, we're getting there
* we're just normal men; we're just innocent men
* apiutil
* whoopsie
* i'm glad noone reads commit msgs haha :blob_sweat:
* use that weirdo go-bytesize library for maxMultipartMemory
* fix media module paths
Lots of these were appearing:
```
*459 connect() failed (111: Connection refused) while connecting to upstream
```
This change resolves it, see https://stackoverflow.com/a/52550758
* [docs] Serve static assets with nginx
This explains how to use nginx to serve static assets and offload GTS
from that responsibility. It also shows how to have nginx add caching
headers to indicate to clients how long they may cache an asset.
* [docs] Move additional nginx config to advanced
This moves a bunch of additional nginx configuration into the Advanced
page instead. It declutters the nginx configuration page.
This explains how nginx can be used to cache webfinger responses and
potentially serve stale responses in case GTS is down. This can be
useful to do in order to ensure webfinger keeps working even if you're
doing some maintenance.
* [chore] Move ShortcodeDomain to its own little util func
* [feature] Add RefetchEmojis function to media manager
* [feature] Expose admin media refresh via admin API
* update following review feedback
- change/fix log levels
- make sure not to try to refetch local emojis
- small style refactoring + comments
* log on emoji refetch start
Signed-off-by: kim <grufwub@gmail.com>
Co-authored-by: kim <grufwub@gmail.com>
* [docs] Update contributing.md
- Add Pull Request process and guidelines.
- Add feature/bug issue process.
- Rearrange some sections for clarity.
- Add overview of package structure.
* [docs] Add build from source links
* [chore] add pull request templates
These link to the new CONTRIBUTING.md document, and include a checklist to validate that contributors have read the guidelines.
* [docs] Put existing stub CoC in separate doc
* update web related stuff in CONTRIBUTING.md
Co-authored-by: f0x <f0x@cthu.lu>
* Implement Bookmarks
* Update based on review comments
* Update swagger doc
* Fix argument passing to status.Bookmark
* Update changed test
* Updates based on latest PR review
* [feature] overhaul the oidc system
this allows for more flexible username handling and prevents account
takeover using old email addresses
* [feature] add migration path for old OIDC users
* [feature] nicer error reporting for users
* [docs] document the new OIDC flow
* [fix] return early on oidc error
* [docs]: add comments on the finalization logic
* remove filesystem logging directives from example systemd unit config
* [docs] Update docs to reflect new systemd config
Co-authored-by: tsmethurst <tobi.smethurst@protonmail.com>
* Enable the 'admonitions' Markdown extension for Mkdocs.
The admonitions extension to Python-Markdown allows you to include
rST-style "admonitions" to Markdown documents, for instance,
!!! note
Here's an important note to keep in mind!
In general, the current documentation uses bold text to try to achieve
the same effect, which is a bit harder to notice and makes it difficult
to differentiate between "here's something useful to know" versus "here
there be dragons".
* Add AppArmor profile and documentation for LSM-related sandboxing
This commit adds an AppArmor profile for gotosocial in
examples/apparmor/gotosocial. This will (hopefully) serve as a helpful
security mitigation for people are planning on deploying GTS on a
Debian-family Linux distribution.
I've also updates the documentation to include some information about
deploying GTS with either AppArmor or SELinux (moving the documentation
for the former out of the "binary installation guide" docs).
Since the documentation site only shows the latest version of the
docs, we need the docs to explain how to use the latest stable
release, not just the latest git version.
* add FilePath regex
* add `admin media prune orphaned` command
* add prune orphaned function to media manager
* don't mark flag as required
* document admin media prune orphaned cmd
* oh envparsing.sh you coy minx
The NewSignup method was already being called with
requireApproval=false, but it had emailVerified=false as well, which
meant that it was required to use the `admin account confirm` command
to verify the email before the newly-created user could log in.
I think that was probably an oversight; effectively it did require
approval anyway. Changing emailVerified to true allows you to just
create the account and log in immediately, reducing the opportunity
for manual error to sneak in.
Also updated the docs to remove the mention of needing to confirm new
accounts. However, I've left the confirmation command alone because I
think once we have web signups, it will be needed in that context.
* start adding admin emoji PATCH stuff
* updating works OK, now how about copying
* allow emojis to be copied
* update swagger docs
* update admin processer to use non-interface storage driver
* remove shortcode updating for local emojis
* go fmt
Co-authored-by: f0x52 <f0x@cthu.lu>
* Clarify that gotosocial doesn't use recommendation algorithms
It uses lots of algorithms.
~/src/gotosocial $ rgrep algorithm . | wc
856 8971 143395
* Add first draft of the FAQ.
* Suggested FAQ changes.
Mention backfill of posts as not yet being implemented; link to FAQ
from readme/index.
Mention that GtS does serve up some UI; just not most interaction.
Also explain specifically that setting the rate limit to zero turns
off rate limiting rather than actually setting the limit to zero,
since this is not intuitive.
* [feature] Add 'role' field to api serialization of local accounts
* [chore] Add a bit of license text while I'm here
* [frogend] render account role on same line as username in web view of profile
* style tweaking on role badges, general profile header layout
* profile stats wrapping
* don't render standard 'user' role on web view
Co-authored-by: f0x <f0x@cthu.lu>
* [feature] Add emoji categories GET
Serialize emojis in appropriate categories; make it possible to get categories via the admin API
* [feature] Create (or use existing) category for new emoji uploads
* fix lint issue
* update misleading line in swagger docs
* Add instance-expose-public-timeline flag
Adds a config flag that allows unauthenticated access to /api/v1/timelines/public. Defaults to false to replicate existing behaviour.
* Update structure following review
* Add comment
* Fix linting
* S3: add config value "proxy" for not redirecting
Signed-off-by: Mara Sophie Grosch <littlefox@lf-net.org>
* S3: document new config value "proxy"
* S3: add new config value "proxy" to test scripts
Signed-off-by: Mara Sophie Grosch <littlefox@lf-net.org>
* [feature] Make instance thumbnail configurable via admin panel
* log db errors in InstanceToAPIInstance
* only update instance in db if necessary
* start adding tests
* finish test
* add admin emojis get path + model + docs
* stub admin emojis get processor function
* add id + disabled fields to admin emoji
* add emoji -> api admin emoji converter
* tidy up a bit
* add GetEmojis function
* finish up get emojis function
* order by shortcodedomain
* ASC
* tidy up + explain
* update to allow paging
* make admin emojis pageable
* fix mixed case paging
* normalize emoji queries a bit better
* test emoji get paging
* make limit optional
* fix incorrect path in media cleanup tests
* i have bad coder syndrome
* don't trimspace
* rename -> GetUseableEmojis
* wrap emoji query in subquery
avoid selecting more than we need
* fix a bit of sillyness teehee
* fix subquery postgres woes
* start adding rss functionality
* add gorilla/feeds dependency
* first bash at building rss feed
still needs work, this is an interim commit
* tidy up a bit
* add publicOnly option to GetAccountLastPosted
* implement rss endpoint
* fix test
* add initial user docs for rss
* update rss logo
* docs update
* add rssFeed to frontend
* feed -> feed.rss
* enableRSS
* increase rss logo size a lil bit
* add rss toggle
* move emojify to text package
* fiddle with rss feed formatting
* add Text field to test statuses
* move status to rss item to typeconverter
* update bun schema for enablerss
* simplify 304 checking
* assume account not rss
* update tests
* update swagger docs
* allow more characters in title, trim nicer
* update last posted to be more consistent
* update dependencies, bump Go version to 1.19
* bump test image Go version
* update golangci-lint
* update gotosocial-drone-build
* sign
* linting, go fmt
* update swagger docs
* update swagger docs
* whitespace
* update contributing.md
* fuckin whoopsie doopsie
* linterino, linteroni
* fix followrequest test not starting processor
* fix other api/client tests not starting processor
* fix remaining tests where processor not started
* bump go-runners version
* don't check last-webfingered-at, processor may have updated this
* update swagger command
* update bun to latest version
* fix embed to work the same as before with new bun
Signed-off-by: kim <grufwub@gmail.com>
Co-authored-by: tsmethurst <tobi.smethurst@protonmail.com>
* update Activity
* add instance-deliver-to-shared-inboxes setting
* update activity version again
* add SharedInboxURI field to accounts
* serdes for endpoints/sharedInbox
* deliver to sharedInbox if one is available
* update tests
* only assign shared inbox if shared domain
* look for shared inbox if currently nil
* go fmt
* finger to get params.RemoteAccountID if necessary
* make comments clearer
* compare dns more consistently
Previously we had a few examples referring to --config-file (which is
not accepted) but most were missing it altogether. Put this argument
last in all the examples.
Also replaced "./example.json" with just "example.json" in the
import/export examples because the "./" was unnecessary.
* add custom css account property + db func to fetch
* allow account to get/set custom css
* serve custom css for an account
* go fmt
* use monospace for customcss, add link
* add custom css to account cache
* fix broken field
* add custom css docs to user guide
* add `accounts-allow-custom-css` config flag
* add allow custom css to /api/v1/instance response
* only show/set custom css if allowed to do so
* only set/serve custom account css if enabled
* update swagger docs
* chain promise
* make bool a bit clearer
* use cache for GetAccountCustomCSSByUsername
* start implementing remote emoji fetcher
* update status where pk
* aaa
* tidy up a little
* check size limits for emojis
* thank you linter, i love you <3
* update swagger docs
* add emoji dereference test
* make emoji max sizes configurable
* normalize db.ErrAlreadyExists
* add post_format to acct & use it when making post
* update swagger docs
* add status_format updating to frontend
* fix up tests
* post_format => status_format
* add status_format to account validation
* [bugfix] Correctly style inputs and buttons
<input>, <textarea>, and <button> were incorrectly using the system-ui font previously; this commit fixes that. text-align: center; was added to <button> due to an inconsistency with .button where text-align would be off.
* [chore] Update binary installation instructions
This commit updates the example release mentioned in the docs, and mentions Caddy in the reverse proxy options.
* [bugfix] Remove redundant Caddyfile
Caddy automatically upgrades HTTP to HTTPS (see https://caddyserver.com/docs/automatic-https) so the upgrading part of the Caddyfile is redundant.
* feat: vendor minio client
* feat: introduce storage package with s3 support
* feat: serve s3 files directly
this saves a lot of bandwith as the files are fetched from the object
store directly
* fix: use explicit local storage in tests
* feat: integrate s3 storage with the main server
* fix: add s3 config to cli tests
* docs: explicitly set values in example config
also adds license header to the storage package
* fix: use better http status code on s3 redirect
HTTP 302 Found is the best fit, as it signifies that the resource
requested was found but not under its presumed URL
307/TemporaryRedirect would mean that this resource is usually located
here, not in this case
303/SeeOther indicates that the redirection does not link to the
requested resource but to another page
* refactor: use context in storage driver interface
* add configuration object to api instance model
* regenerate swagger docs
* add func to return all supported mimes for media
* add instance configuration to api serialization
* fix json tags
* update instance endpoint tests
* fix typeutils tests
* final regen of swagger docs
* omitempty instance configuration
* add missing license headers
* start adding instance peers get
* rename domainblock.go
* embed domain in domainblock so it can be reused
* update swagger docs
* add test instances to db
* update tests
* add/update instancepeersget
* update domain model
* add getinstancepeers to db
* instance-expose-peers, instance-expose-suspended
* add auth checks for both current filters
* attach endpoint to router
* include public comment
* obfuscate domain if required
* go mod tidy
* update swagger docs
* remove unnecessary comment
* return 'flat' peerlist if no query params provided
* 🐸restructure frontend stuff, include admin and future user panel in main repo, properly deduplicate bundles for css+js across uses
* rename bundled to dist, caught by gitignore
* re-include status.css for profile template
* default to localhost
* serve frontend panels
* add todo message for abstraction
* refactor oauth registration flow
* oauth restructure
* update footer template
* change panel routes
* remove superfluous css imports
* write bundle to disk from test server, use forked budo-express
* wrap all page content in container
for robustness with addons etc injection other elements in body
* update documentation, goreleaser, Dockerfile
* update template meta tags
* add AGPL-3.0+ license header everywhere
* only attach update listener on EventEmitter
* cleaner config for various frontend bundles
* fix bundler script paths
* Merge commit 'd191931932b9293ce1be44ed08a1e69b9fcc1e25'
* fix up dockerfile, goreleaser
* go mod tidy
* add uglifyify
* move status hide/show js to frontend bundle
* fix stylesheet color( func regressions
* update contributing docs for new build path
* update goreleaser + docker building
* resolve dependency paths properly
* update package name
* use api errorhandler
Co-authored-by: tsmethurst <tobi.smethurst@protonmail.com>
* update templates
* start reworking api error handling
* update template
* return AP status at web endpoint if negotiated
* start making api error handling much more consistent
* update account endpoints to new error handling
* use new api error handling in admin endpoints
* go fmt ./...
* use api error logic in app
* use generic error handling in auth
* don't export generic error handler
* don't defer clearing session
* user nicer error handling on oidc callback handler
* tidy up the sign in handler
* tidy up the token handler
* use nicer error handling in blocksget
* auth emojis endpoint
* fix up remaining api endpoints
* fix whoopsie during login flow
* regenerate swagger docs
* change http error logging to debug
* Add instructions for working with Caddy 2
Add instructions for working with Caddy 2. Some texts are duplicated from the NGINX part (mainly the configuration part, about systemctl).
* Add new Caddy docs to mkdocs.yml
Adds the new Caddy document to the document tree
* Remove up and downstream headers from configuration
Removed the header statements for the proxy, as proxying those are transparant. Kept the flush_interval directive, disabling the response buffer completely so we can write without delays.
* Update caddy.md
Corrects the opening link
* Apply comments mentioned in PR
@igalic mentioned a few comments to improve these docs. I've processed those in this PR.
* rename + tidy up remote pruning
* fix media attachment account join
see https://bun.uptrace.dev/guide/golang-orm.html#table-relationships
* update logging to new function name
* add get avatars and headers to bun
* add pruneallmeta function
* don't set uncached since we're deleting anyway
* fix totalPruned being in wrong place
* test pruning meta
* go fmt ./...
* rename mediaprune
* add meta pruning to routine mediaprune
* tidy up cleanup job scheduling
* rename adminmediaremoteprune
* update mediacleanup to use renamed prune func
* update swagger docs a little bit
* reuse cancel + context
* admin panel documentation
* wrap link so it's clickable
* format img so it works with mkdocs serve
Co-authored-by: tsmethurst <tobi.smethurst@protonmail.com>
* docs: unify nginx explainers and add apache httpd
there are two places where nginx + certbot is explained, unify that into
one place.
Add apache httpd, following the same steps, but using mod_md for
LetsEncrypt
add a note about #453 in both guides.
Link to both, and call the section reverse proxy, instead of NGINX
* restore full nginx.conf from docker.md
* add installation_guide/apache-httpd.md to mkdocs
* add accountAction to the admin API
* model admin account action
* add admin account action to the processor
* add migration for new AdminAccountActions table
* fix accounts admin path
* Update swagger docs
* Add whereNotEmptyAndNotNull
* Add GetRemoteOlderThanDays
* Add GetRemoteOlderThanDays
* Add PruneRemote to Manager interface
* Start implementing PruneRemote
* add new attachment + status to tests
* fix up and test GetRemoteOlderThan
* fix bad import
* PruneRemote: return number pruned
* add Cached column to mediaattachment
* update + test pruneRemote
* update mediaTest
* use Cached column
* upstep bun to latest version
* embed structs in mediaAttachment
* migrate mediaAttachment to new format
* don't default cached to true
* select only remote media
* update db dependencies
* step bun back to last working version
* update pruneRemote to use Cached field
* fix storage path of test attachments
* add recache logic to manager
* fix trimmed aspect ratio
* test prune and recache
* return errwithcode
* tidy up different paths for emoji vs attachment
* fix incorrect thumbnail type being stored
* expose TransportController to media processor
* implement tee-ing recached content
* add thoughts of dog to test fedi attachments
* test get remote files
* add comment on PruneRemote
* add postData cleanup to recache
* test thumbnail fetching
* add incredible diagram
* go mod tidy
* buffer pipes for recache streaming
* test for client stops reading after 1kb
* add media-remote-cache-days to config
* add cron package
* wrap logrus so it's available to cron
* start and stop cron jobs gracefully
* exludes docker-volume from exemple/docker-compose
* Adds an docker-compose.yaml example and adds some readme to the
docker.md
* Changes Docker-Compose Example
* Configures docker-compose.yaml and docker.md
* Adds some cosmetics to the Documentation
* Adds UserID:GroupID
* Describes how to create a new user
When I tried to start the server using the command `./gotosocial server start --config-file ./config.yaml`, I got the following error:
```
root@gotosocial-experiment:/gotosocial# ./gotosocial server start --config-file ./config.yaml
Incorrect Usage: flag provided but not defined: -config-file
NAME:
gotosocial server start - start the gotosocial server
USAGE:
gotosocial server start [command options] [arguments...]
OPTIONS:
--help, -h show help (default: false)
FATA[0000] flag provided but not defined: -config-file
```
Putting the flag after the `./gotosocial` but before the `server start` seems to work properly
