Commit graph

48 commits

Author SHA1 Message Date
tobi 782169da76
[chore] set max open / idle conns + conn max lifetime for both postgres and sqlite (#1369)
* [chore] set max open / idle conns + conn max lifetime for both postgres and sqlite

* reduce cache size default to 8MiB, reduce connections to 2 * cpu

* introduce max open conns multiplier, tune sqlite and pg separately

* go fmt
2023-01-26 15:12:48 +01:00
f0x52 17eecfb6d9
[feature] Public list of suspended domains (#1362)
* basic rendered domain blocklist (unauthenticated!)

* style basic domain block list

* better formatting for domain blocklist

* add opt-in config option for showing suspended domains

* format/linter

* re-use InstancePeersGet for web-accessible domain blocklist

* reword explanation, border styling

* always attach blocklist handler, update error message

* domain blocklist error message grammar
2023-01-25 18:06:41 +01:00
tobi 627b8eeae6
[feature] Tune sqlite pragmas (#1349)
* sqlite pragma tuning

* use formatuint

* add sqlite busy timeout

* fix incorrect cache size format

* update envparsing test

* add sqlite tuning flags to cli

* set sqlite timeout to 30s default
2023-01-17 12:29:44 +00:00
tobi d6487933c7
[feature] Implement Report database model and utility functions (#1310)
* implement report database model

* implement report cache + config changes

* implement report database functions

* report uri / regex functions

* update envparsing test

* remove unnecessary uri index

* remove unused function + cache lookup

* process error when storing report
2023-01-10 14:19:05 +00:00
tobi 0dbe6c514f
[chore] Update/add license headers for 2023 (#1304) 2023-01-05 12:43:00 +01:00
tobi 90a14abb0c
[feature] HTTP request throttling middleware (#1297)
* [feature] Add throttling middleware to AP endpoints

* refactor a lil bit

* use config setting, start updating docs

* doc updates

* use relative links in faq doc

* small docs fixes

* return code 503 instead of 429 when throttled

* throttle other endpoints too

* simplify token channel prefills
2023-01-04 11:57:59 +01:00
kim cb2b2fd805
[feature] support configuring database caches (#1246)
* update config generator to support nested structs, add cache configuration options

* update envparsing test

* add cache configuration to config parse tests

* set cache configuration in testrig

* move caches to sub-cache "gts" namespace, update envparsing, add cache config docs to example config

Signed-off-by: kim <grufwub@gmail.com>
2022-12-11 13:03:15 +00:00
Dominik Süß 199b685f43
[feature] overhaul the oidc system (#961)
* [feature] overhaul the oidc system

this allows for more flexible username handling and prevents account
takeover using old email addresses

* [feature] add migration path for old OIDC users

* [feature] nicer error reporting for users

* [docs] document the new OIDC flow

* [fix] return early on oidc error

* [docs]: add comments on the finalization logic
2022-12-06 14:15:56 +01:00
tobi 13e9abd02a
[feature] Add admin media prune orphaned CLI command (#1146)
* add FilePath regex

* add `admin media prune orphaned` command

* add prune orphaned function to media manager

* don't mark flag as required

* document admin media prune orphaned cmd

* oh envparsing.sh you coy minx
2022-11-25 17:23:42 +00:00
Rob Sargant d120743e8b
[feature] add instance-expose-public-timeline flag (#1039)
* Add instance-expose-public-timeline flag

Adds a config flag that allows unauthenticated access to /api/v1/timelines/public. Defaults to false to replicate existing behaviour.

* Update structure following review

* Add comment

* Fix linting
2022-11-14 10:30:01 +01:00
Mara Sophie Grosch 948e90b95a
[feature] S3: add config flag to proxy S3 media (#1014)
* S3: add config value "proxy" for not redirecting

Signed-off-by: Mara Sophie Grosch <littlefox@lf-net.org>

* S3: document new config value "proxy"

* S3: add new config value "proxy" to test scripts

Signed-off-by: Mara Sophie Grosch <littlefox@lf-net.org>
2022-11-11 12:03:18 +01:00
tobi 4d66fb9603
[feature] Make rate limit requests amount configurable (#966)
* update rate limit documentation

* regenerate landingpage config helpers

* make rate limit rate configurable
2022-11-06 09:47:48 +00:00
Mia Heidenstedt f3fc040c2e
[feature] Allow user to show instead of landing page on / (#922)
* configurable user instead of landing page

* rename DefaultUser to LandingPageUser

* code review changes and fix tests

* try to fix pipeline error

* code review changes

* code review changes

* code review changes

* remove unnecessary line
2022-11-03 13:15:31 +01:00
kim 1d999712e6
[feature] update config types to use bytesize.Size (#828)
* update config size types to use bytesize.Size

* submit unchecked-out file ... 🤦

* fix bytesize config var decoding

* bump bytesize version

* update kim's libraries in readme

* update envparse.sh to output more useful errors

* improve envparse.sh

* remove reliance on jq

* instead, use uint64 for bytesize flag types

* remove redundant type

* fix viper unmarshaling

* Update envparsing.sh

* fix envparsing test

Signed-off-by: kim <grufwub@gmail.com>
Co-authored-by: tobi <31960611+tsmethurst@users.noreply.github.com>
2022-09-29 21:50:43 +01:00
tobi 69a193dae5
[feature] Allow delivery to sharedInboxes where possible (#847)
* update Activity

* add instance-deliver-to-shared-inboxes setting

* update activity version again

* add SharedInboxURI field to accounts

* serdes for endpoints/sharedInbox

* deliver to sharedInbox if one is available

* update tests

* only assign shared inbox if shared domain

* look for shared inbox if currently nil

* go fmt

* finger to get params.RemoteAccountID if necessary

* make comments clearer

* compare dns more consistently
2022-09-23 20:27:35 +01:00
tobi b42469e4e0
[feature] Allow users to set custom css for their profiles + threads (#808)
* add custom css account property + db func to fetch

* allow account to get/set custom css

* serve custom css for an account

* go fmt

* use monospace for customcss, add link

* add custom css to account cache

* fix broken field

* add custom css docs to user guide

* add `accounts-allow-custom-css` config flag

* add allow custom css to /api/v1/instance response

* only show/set custom css if allowed to do so

* only set/serve custom account css if enabled

* update swagger docs

* chain promise

* make bool a bit clearer

* use cache for GetAccountCustomCSSByUsername
2022-09-12 13:14:29 +02:00
tobi 268f252e0d
[feature] Fetch + display custom emoji in statuses from remote instances (#807)
* start implementing remote emoji fetcher

* update status where pk

* aaa

* tidy up a little

* check size limits for emojis

* thank you linter, i love you <3

* update swagger docs

* add emoji dereference test

* make emoji max sizes configurable

* normalize db.ErrAlreadyExists
2022-09-12 13:03:23 +02:00
Dominik Süß 9d0df426da
[feature] S3 support (#674)
* feat: vendor minio client

* feat: introduce storage package with s3 support

* feat: serve s3 files directly

this saves a lot of bandwith as the files are fetched from the object
store directly

* fix: use explicit local storage in tests

* feat: integrate s3 storage with the main server

* fix: add s3 config to cli tests

* docs: explicitly set values in example config

also adds license header to the storage package

* fix: use better http status code on s3 redirect

HTTP 302 Found is the best fit, as it signifies that the resource
requested was found but not under its presumed URL

307/TemporaryRedirect would mean that this resource is usually located
here, not in this case

303/SeeOther indicates that the redirection does not link to the
requested resource but to another page

* refactor: use context in storage driver interface
2022-07-03 12:08:30 +02:00
tobi 5f00d4980b
[feature] Implement /api/v1/instance/peers endpoint (#660)
* add missing license headers

* start adding instance peers get

* rename domainblock.go

* embed domain in domainblock so it can be reused

* update swagger docs

* add test instances to db

* update tests

* add/update instancepeersget

* update domain model

* add getinstancepeers to db

* instance-expose-peers, instance-expose-suspended

* add auth checks for both current filters

* attach endpoint to router

* include public comment

* obfuscate domain if required

* go mod tidy

* update swagger docs

* remove unnecessary comment

* return 'flat' peerlist if no query params provided
2022-06-23 16:54:54 +02:00
kim 91c0ed863a
[bugfix] #621: add weak type handing to mapstructure decode (#625)
* Drone sig (#623)

* accept weakly typed input on mapstructure decode i.e. .UnmarshalMap()

Signed-off-by: kim <grufwub@gmail.com>

* add envparsing script to test for panics during environment variable parsing

Signed-off-by: kim <grufwub@gmail.com>

* add envparsing.sh script to drone commands

Signed-off-by: kim <grufwub@gmail.com>

* update drone signature

Co-authored-by: kim <grufwub@gmail.com>

* compare expected with output

* update expected output of envparsing

* update expected output to correct value

* use viper's unmarshal function instead
There were problems with marshalling
string slices from viper into the st.config
struct with the other function. Now, we
can use viper's unmarshal function and pass
in the custom decoder config that we need
as a hook. This ensures that we marshal
string slices from viper into our config
struct correctly.

Co-authored-by: tobi <31960611+tsmethurst@users.noreply.github.com>
Co-authored-by: tsmethurst <tobi.smethurst@protonmail.com>
2022-06-08 20:28:28 +02:00
tobi 327d3f001f
[feature] Start adding advanced configuration options, starting with samesite (#628)
* fix incorrect port being used for db

* start adding advanced config flags

* use samesite lax by default
2022-06-03 15:40:38 +02:00
kim 43ac0cdb9c
[chore] Global server configuration overhaul (#575)
* move config flag names and usage to config package, rewrite config package to use global Configuration{} struct

Signed-off-by: kim <grufwub@gmail.com>

* improved code comment

Signed-off-by: kim <grufwub@gmail.com>

* linter

Signed-off-by: kim <grufwub@gmail.com>

* fix unmarshaling

Signed-off-by: kim <grufwub@gmail.com>

* remove kim's custom go compiler changes

Signed-off-by: kim <grufwub@gmail.com>

* generate setter and flag-name functions, implement these in codebase

Signed-off-by: kim <grufwub@gmail.com>

* update deps

Signed-off-by: kim <grufwub@gmail.com>

* small change

Signed-off-by: kim <grufwub@gmail.com>

* appease the linter...

Signed-off-by: kim <grufwub@gmail.com>

* move configuration into ConfigState structure, ensure reloading to/from viper settings to keep in sync

Signed-off-by: kim <grufwub@gmail.com>

* lint

Signed-off-by: kim <grufwub@gmail.com>

* update code comments

Signed-off-by: kim <grufwub@gmail.com>

* fix merge issue

Signed-off-by: kim <grufwub@gmail.com>

* fix merge issue

Signed-off-by: kim <grufwub@gmail.com>

* improved version string (removes time + go version)

Signed-off-by: kim <grufwub@gmail.com>

* fix version string build to pass test script + consolidate logic in func

Signed-off-by: kim <grufwub@gmail.com>

* add license text, update config.Defaults comment

Signed-off-by: kim <grufwub@gmail.com>

* add license text to generated config helpers file

Signed-off-by: kim <grufwub@gmail.com>

* defer unlock on config.Set___(), to ensure unlocked on panic

Signed-off-by: kim <grufwub@gmail.com>

* make it more obvious which cmd flags are being attached

Signed-off-by: kim <grufwub@gmail.com>
2022-05-30 14:41:24 +02:00
tobi 0884f89431
Implement Cobra CLI tooling, Viper config tooling (#336)
* start pulling out + replacing urfave and config

* replace many many instances of config

* move more stuff => viper

* properly remove urfave

* move some flags to root command

* add testrig commands to root

* alias config file keys

* start adding cli parsing tests

* reorder viper init

* remove config path alias

* fmt

* change config file keys to non-nested

* we're more or less in business now

* tidy up the common func

* go fmt

* get tests passing again

* add note about the cliparsing tests

* reorganize

* update docs with changes

* structure cmd dir better

* rename + move some files around

* fix dangling comma
2021-12-07 13:31:39 +01:00
Forest Johnson ab316d2250
Fix broken defaults and broken flags (#314)
* start with a default config, not an empty config.

* some data structures were present on Empty config but not Default config

* the monkey patched CLIContext is working

* remove print debugging log

* make the behaviour of the flags consistent across all data types

Conflicts:
	internal/config/config.go

* try to fix accidentally broken test
2021-11-24 17:01:38 +01:00
tobi 1ded58b34b
add bindAddress configuration option (#320)
* add bindAddress configuration option

* clarify that bindAddress can be a hostname
2021-11-22 10:55:52 +01:00
tobi 2aaec82732
smtp + email confirmation (#285)
* add smtp configuration

* add email confirm + reset templates

* add email sender to testrig

* flesh out the email sender interface

* go fmt

* golint

* update from field with more clarity

* tidy up the email formatting

* fix tests

* add email sender to processor

* tidy client api processing a bit

* further tidying in fromClientAPI

* pin new account to user

* send msg to processor on new account creation

* generate confirm email uri

* remove emailer from account processor again

* add processCreateAccountFromClientAPI

* move emailer accountprocessor => userprocessor

* add email sender to user processor

* SendConfirmEmail function

* add noop email sender

* use noop email sender in tests

* only assemble message if callback is not nil

* use noop email sender if no smtp host is defined

* minify email html before sending

* fix wrong email address

* email confirm test

* fmt

* serve web hndler

* add email confirm handler

* init test log properly on testrig

* log emails that *would* have been sent

* go fmt ./...

* unexport confirm email handler

* updatedAt

* test confirm email function

* don't allow tokens older than 7 days

* change error message a bit

* add basic smtp docs

* add a few more snippets

* typo

* add email sender to outbox tests

* don't use dutch wikipedia link

* don't minify email html
2021-10-31 15:46:23 +01:00
tobi 555ea8edfb
Import export (#194)
* start with export/import code

* messing about with decoding/encoding

* some more fiddling

* stuff is WORKING

* working pretty alright!

* go fmt

* fix up tests, add docs

* start backup/restore doc

* tweaks

* credits

* update advancedVisibility settings

* update bun library -> v1.0.4

Signed-off-by: kim (grufwub) <grufwub@gmail.com>

* update oauth library -> v4.3.1-SSB

Signed-off-by: kim (grufwub) <grufwub@gmail.com>

* handle oauth token scope, fix user.SigninCount + token.UserID

Signed-off-by: kim (grufwub) <grufwub@gmail.com>

* update oauth library --> v4.3.2-SSB

Signed-off-by: kim (grufwub) <grufwub@gmail.com>

* update sqlite library -> v1.13.0

Signed-off-by: kim (grufwub) <grufwub@gmail.com>

* review changes

* start with export/import code

* messing about with decoding/encoding

* some more fiddling

* stuff is WORKING

* working pretty alright!

* go fmt

* fix up tests, add docs

* start backup/restore doc

* tweaks

* credits

* update advancedVisibility settings

* review changes

Co-authored-by: kim (grufwub) <grufwub@gmail.com>
Co-authored-by: kim <89579420+NyaaaWhatsUpDoc@users.noreply.github.com>
2021-09-09 16:15:25 +02:00
Tobi Smethurst e2757ae676
add trusted proxy for parsing client IPs (#115) 2021-07-26 16:15:36 +02:00
tsmethurst 318e377359 allow custom ports for webserver and le 2021-07-24 18:55:24 +02:00
Tobi Smethurst 05e9af089c
Oidc (#109)
* add oidc config

* inching forward with oidc idp

* lil webfingy fix

* bit more progress

* further oidc

* oidc now working

* document dex config

* replace broken images

* add additional credits

* tiny doc update

* update

* add oidc config

* inching forward with oidc idp

* bit more progress

* further oidc

* oidc now working

* document dex config

* replace broken images

* add additional credits

* tiny doc update

* update

* document

* docs + comments
2021-07-23 10:36:28 +02:00
Tobi Smethurst 29bdc41baa
Config bugfix (#104)
* fix some config bugs

* go fmt
2021-07-19 19:31:47 +02:00
Tobi Smethurst b1a4f38e38
allow different host + accountDomain (#103)
* allow different host + accountDomain

* use accountDomain in tags
2021-07-19 18:42:08 +02:00
Tobi Smethurst 677490bc4e
Db tls (#102)
* go mod tidy

* complete example config

* add tls support for db connection

* add certpool to tlsConfig

* add some lil docker scripts
2021-07-19 18:03:07 +02:00
Tobi Smethurst 4f3b3f5c0b
put version in binary properly (#73)
Addresses #71 :

    Set version on the CLI framework.
    Add a build.sh script that injects variables into the build tooling using git and a version file.
    Set version in config.
2021-06-28 12:17:20 +02:00
Tobi Smethurst c1e107266f
nodeinfo compliance (#61) 2021-06-24 14:26:08 +02:00
f0x52 f9bc305aca
new styling for frontpage, update login and authorize templates (#46)
* new styling for frontpage, update login and authorize templates

* run go fmt

* add AssetBaseDir to command flag parsing

* untested: move landing page to it's own router

* go fmt, fix typo

* fix package, adapt to proper Route structure
2021-06-21 19:46:10 +02:00
Tobi Smethurst 3d77f81c7f
Move a lot of stuff + tidy stuff (#37)
Lots of renaming and moving stuff, some bug fixes, more lenient parsing of notifications and home timeline.
2021-05-30 13:12:00 +02:00
Tobi Smethurst 43c3a47773
Admin cli (#29)
Now you can use the CLI tool to:

* Create a new account with the given username, email address and password (which will be hashed of course).
* Confirm the account's so that it can log in and post.
* Promote the account to admin.
* Demote the account from admin.
* Disable the account.
* Suspend the account.
2021-05-22 14:26:45 +02:00
Tobi Smethurst 0cbab627c7
Letsencrypt (#17) 2021-05-09 11:25:13 +02:00
tsmethurst dafc3b5b92 linting + organizing 2021-04-20 18:14:23 +02:00
Tobi Smethurst 32c5fd987a
Api/v1/statuses (#11)
This PR adds:
Statuses

    New status creation.
    View existing status
    Delete a status
    Fave a status
    Unfave a status
    See who's faved a status

Media

    Upload media attachment and store/retrieve it
    Upload custom emoji and store/retrieve it

Fileserver

    Serve files from storage

Testing

    Test models, testrig -- run a GTS test instance and play around with it.
2021-04-19 19:42:19 +02:00
Tobi Smethurst 71a49e2b43
Api/v1/accounts (#8)
* start work on accounts module

* plodding away on the accounts endpoint

* groundwork for other account routes

* add password validator

* validation utils

* require account approval flags

* comments

* comments

* go fmt

* comments

* add distributor stub

* rename api to federator

* tidy a bit

* validate new account requests

* rename r router

* comments

* add domain blocks

* add some more shortcuts

* add some more shortcuts

* check email + username availability

* email block checking for signups

* chunking away at it

* tick off a few more things

* some fiddling with tests

* add mock package

* relocate repo

* move mocks around

* set app id on new signups

* initialize oauth server properly

* rename oauth server

* proper mocking tests

* go fmt ./...

* add required fields

* change name of func

* move validation to account.go

* more tests!

* add some file utility tools

* add mediaconfig

* new shortcut

* add some more fields

* add followrequest model

* add notify

* update mastotypes

* mock out storage interface

* start building media interface

* start on update credentials

* mess about with media a bit more

* test image manipulation

* media more or less working

* account update nearly working

* rearranging my package ;) ;) ;)

* phew big stuff!!!!

* fix type checking

* *fiddles*

* Add CreateTables func

* account registration flow working

* tidy

* script to step through auth flow

* add a lil helper for generating user uris

* fiddling with federation a bit

* update progress

* Tidying and linting
2021-04-01 20:46:45 +02:00
Tobi Smethurst aa9ce272dc
Oauth/token (#7)
* add host and protocol options

* some fiddling

* tidying up and comments

* tick off /oauth/token

* tidying a bit

* tidying

* go mod tidy

* allow attaching middleware to server

* add middleware

* more user friendly

* add comments

* comments

* store account + app

* tidying

* lots of restructuring

* lint + tidy
2021-03-22 22:26:54 +01:00
tsmethurst 81760963b0 formatting,comments 2021-03-20 19:04:27 +01:00
tsmethurst 1b11884121 auth flow working for code 2021-03-18 23:27:43 +01:00
tsmethurst 052783db66 tidying up here and there 2021-03-04 14:38:18 +01:00
tsmethurst 18d0685ef1 further fun 2021-03-03 21:15:20 +01:00
tsmethurst 54c4b8de20 further yak shaving 2021-03-03 18:12:02 +01:00