Commit graph

26 commits

Author SHA1 Message Date
kim c9c0773f2c
[performance] update remaining worker pools to use queues (#2865)
* start replacing client + federator + media workers with new worker + queue types

* refactor federatingDB.Delete(), drop queued messages when deleting account / status

* move all queue purging to the processor workers

* undo toolchain updates

* code comments, ensure dereferencer worker pool gets started

* update gruf libraries in readme

* start the job scheduler separately to the worker pools

* reshuffle ordering or server.go + remove duplicate worker start / stop

* update go-list version

* fix vendoring

* move queue invalidation to before wipeing / deletion, to ensure queued work not dropped

* add logging to worker processing functions in testrig, don't start workers in unexpected places

* update go-structr to add (+then rely on) QueueCtx{} type

* ensure more worker pools get started properly in tests

* fix remaining broken tests relying on worker queue logic

* fix account test suite queue popping logic, ensure noop workers do not pull from queue

* move back accidentally shuffled account deletion order

* ensure error (non nil!!) gets passed in refactored federatingDB{}.Delete()

* silently drop deletes from accounts not permitted to

* don't warn log on forwarded deletes

* make if else clauses easier to parse

* use getFederatorMsg()

* improved code comment

* improved code comment re: requesting account delete checks

* remove boolean result from worker start / stop since false = already running or already stopped

* remove optional passed-in http.client

* remove worker starting from the admin CLI commands (we don't need to handle side-effects)

* update prune cli to start scheduler but not all of the workers

* fix rebase issues

* remove redundant return statements

* i'm sorry sir linter
2024-04-26 13:50:46 +01:00
tobi 9fb8a78f91
[feature] New user sign-up via web page (#2796)
* [feature] User sign-up form and admin notifs

* add chosen + filtered languages to migration

* remove stray comment

* chosen languages schmosen schmanguages

* proper error on local account missing
2024-04-11 11:45:53 +02:00
tobi dc96562b40
[bugfix] Use custom bluemonday policy to disallow inline img tags (#2100) 2023-08-11 14:40:11 +02:00
Vyr Cossont 0f812746b7
[feature] Allow full BCP 47 in language inputs (#2067)
* Allow full BCP 47 in language inputs

Fixes #2066

* Fuse validation and normalization for languages

* Remove outdated comment line

* Move post language canonicalization test
2023-08-07 10:25:54 +02:00
tobi 5a29a031ad
[chore] Admin CLI + new account creation refactoring (#2008)
* set maxPasswordLength to 72 bytes, rename validate function

* refactor NewSignup

* refactor admin account CLI commands

* refactor oidc create user

* refactor processor create

* tweak password change, check old != new password
2023-07-23 12:33:17 +02:00
Daenney 5e2bf0bdca
[chore] Improve copyright header handling (#1608)
* [chore] Remove years from all license headers

Years or year ranges aren't required in license headers. Many projects
have removed them in recent years and it avoids a bit of yearly toil.

In many cases our copyright claim was also a bit dodgy since we added
the 2021-2023 header to files created after 2021 but you can't claim
copyright into the past that way.

* [chore] Add license header check

This ensures a license header is always added to any new file. This
avoids maintainers/reviewers needing to remember to check for and ask
for it in case a contribution doesn't include it.

* [chore] Add missing license headers

* [chore] Further updates to license header

* Use the more common // indentend comment format
* Remove the hack we had for the linter now that we use the // format
* Add SPDX license identifier
2023-03-12 16:00:57 +01:00
kim baf933cb9f
[chore] move client/federator workerpools to Workers{} (#1575)
* replace concurrency worker pools with base models in State.Workers, update code and tests accordingly

* improve code comment

* change back testrig default log level

* un-comment-out TestAnnounceTwice() and fix

---------

Signed-off-by: kim <grufwub@gmail.com>
Reviewed-by: tobi
2023-03-01 18:26:53 +00:00
tobi b6fbdc66c1
[chore] Deinterface processor and subprocessors (#1501)
* [chore] Deinterface processor and subprocessors

* expose subprocessors via function calls

* missing license header
2023-02-22 16:05:26 +01:00
Daenney 68e6d08c76
[feature] Add a request ID and include it in logs (#1476)
This adds a lightweight form of tracing to GTS. Each incoming request is
assigned a Request ID which we then pass on and log in all our log
lines. Any function that gets called downstream from an HTTP handler
should now emit a requestID=value pair whenever it logs something.

Co-authored-by: kim <grufwub@gmail.com>
2023-02-17 12:02:29 +01:00
tobi 0dbe6c514f
[chore] Update/add license headers for 2023 (#1304) 2023-01-05 12:43:00 +01:00
Dominik Süß 199b685f43
[feature] overhaul the oidc system (#961)
* [feature] overhaul the oidc system

this allows for more flexible username handling and prevents account
takeover using old email addresses

* [feature] add migration path for old OIDC users

* [feature] nicer error reporting for users

* [docs] document the new OIDC flow

* [fix] return early on oidc error

* [docs]: add comments on the finalization logic
2022-12-06 14:15:56 +01:00
kim 098dbe6ff4
[chore] use our own logging implementation (#716)
* first commit

Signed-off-by: kim <grufwub@gmail.com>

* replace logging with our own log library

Signed-off-by: kim <grufwub@gmail.com>

* fix imports

Signed-off-by: kim <grufwub@gmail.com>

* fix log imports

Signed-off-by: kim <grufwub@gmail.com>

* add license text

Signed-off-by: kim <grufwub@gmail.com>

* fix package import cycle between config and log package

Signed-off-by: kim <grufwub@gmail.com>

* fix empty kv.Fields{} being passed to WithFields()

Signed-off-by: kim <grufwub@gmail.com>

* fix uses of log.WithFields() with whitespace issues and empty slices

Signed-off-by: kim <grufwub@gmail.com>

* *linter related grumbling*

Signed-off-by: kim <grufwub@gmail.com>

* gofmt the codebase! also fix more log.WithFields() formatting issues

Signed-off-by: kim <grufwub@gmail.com>

* update testrig code to match new changes

Signed-off-by: kim <grufwub@gmail.com>

* fix error wrapping in non fmt.Errorf function

Signed-off-by: kim <grufwub@gmail.com>

* add benchmarking of log.Caller() vs non-cached

Signed-off-by: kim <grufwub@gmail.com>

* fix syslog tests, add standard build tags to test runner to ensure consistency

Signed-off-by: kim <grufwub@gmail.com>

* make syslog tests more robust

Signed-off-by: kim <grufwub@gmail.com>

* fix caller depth arithmatic (is that how you spell it?)

Signed-off-by: kim <grufwub@gmail.com>

* update to use unkeyed fields in kv.Field{} instances

Signed-off-by: kim <grufwub@gmail.com>

* update go-kv library

Signed-off-by: kim <grufwub@gmail.com>

* update libraries list

Signed-off-by: kim <grufwub@gmail.com>

* fuck you linter get nerfed

Signed-off-by: kim <grufwub@gmail.com>

Co-authored-by: tobi <31960611+tsmethurst@users.noreply.github.com>
2022-07-19 10:47:55 +02:00
tobi 1ede54ddf6
[feature] More consistent API error handling (#637)
* update templates

* start reworking api error handling

* update template

* return AP status at web endpoint if negotiated

* start making api error handling much more consistent

* update account endpoints to new error handling

* use new api error handling in admin endpoints

* go fmt ./...

* use api error logic in app

* use generic error handling in auth

* don't export generic error handler

* don't defer clearing session

* user nicer error handling on oidc callback handler

* tidy up the sign in handler

* tidy up the token handler

* use nicer error handling in blocksget

* auth emojis endpoint

* fix up remaining api endpoints

* fix whoopsie during login flow

* regenerate swagger docs

* change http error logging to debug
2022-06-08 20:38:03 +02:00
kim 43ac0cdb9c
[chore] Global server configuration overhaul (#575)
* move config flag names and usage to config package, rewrite config package to use global Configuration{} struct

Signed-off-by: kim <grufwub@gmail.com>

* improved code comment

Signed-off-by: kim <grufwub@gmail.com>

* linter

Signed-off-by: kim <grufwub@gmail.com>

* fix unmarshaling

Signed-off-by: kim <grufwub@gmail.com>

* remove kim's custom go compiler changes

Signed-off-by: kim <grufwub@gmail.com>

* generate setter and flag-name functions, implement these in codebase

Signed-off-by: kim <grufwub@gmail.com>

* update deps

Signed-off-by: kim <grufwub@gmail.com>

* small change

Signed-off-by: kim <grufwub@gmail.com>

* appease the linter...

Signed-off-by: kim <grufwub@gmail.com>

* move configuration into ConfigState structure, ensure reloading to/from viper settings to keep in sync

Signed-off-by: kim <grufwub@gmail.com>

* lint

Signed-off-by: kim <grufwub@gmail.com>

* update code comments

Signed-off-by: kim <grufwub@gmail.com>

* fix merge issue

Signed-off-by: kim <grufwub@gmail.com>

* fix merge issue

Signed-off-by: kim <grufwub@gmail.com>

* improved version string (removes time + go version)

Signed-off-by: kim <grufwub@gmail.com>

* fix version string build to pass test script + consolidate logic in func

Signed-off-by: kim <grufwub@gmail.com>

* add license text, update config.Defaults comment

Signed-off-by: kim <grufwub@gmail.com>

* add license text to generated config helpers file

Signed-off-by: kim <grufwub@gmail.com>

* defer unlock on config.Set___(), to ensure unlocked on panic

Signed-off-by: kim <grufwub@gmail.com>

* make it more obvious which cmd flags are being attached

Signed-off-by: kim <grufwub@gmail.com>
2022-05-30 14:41:24 +02:00
tobi 5668ce1ec7
[bugfix] Fix HTML escaping in instance title (#607)
* move caption sanitization -> sanitize.go

* use sanitizeplaintext rather than removehtml

* rename sanitizecaption to sanitizeplaintext

* avoid removing html twice from statuses

* unexport remoteHTML
it's no longer used outside the text package so this
makes it less confusing

* test instance PATCH
2022-05-26 11:37:13 +02:00
kim 420e2fb22b
replace async client API / federator msg processing with worker pools (#497)
* replace async client API / federator msg processing with worker pools
* appease our lord-and-saviour, the linter
2022-04-28 13:23:11 +01:00
tobi ef5a9256a8
Extend license notices to 2022 (#354) 2021-12-20 18:42:19 +01:00
tobi 0884f89431
Implement Cobra CLI tooling, Viper config tooling (#336)
* start pulling out + replacing urfave and config

* replace many many instances of config

* move more stuff => viper

* properly remove urfave

* move some flags to root command

* add testrig commands to root

* alias config file keys

* start adding cli parsing tests

* reorder viper init

* remove config path alias

* fmt

* change config file keys to non-nested

* we're more or less in business now

* tidy up the common func

* go fmt

* get tests passing again

* add note about the cliparsing tests

* reorganize

* update docs with changes

* structure cmd dir better

* rename + move some files around

* fix dangling comma
2021-12-07 13:31:39 +01:00
tobi 2aaec82732
smtp + email confirmation (#285)
* add smtp configuration

* add email confirm + reset templates

* add email sender to testrig

* flesh out the email sender interface

* go fmt

* golint

* update from field with more clarity

* tidy up the email formatting

* fix tests

* add email sender to processor

* tidy client api processing a bit

* further tidying in fromClientAPI

* pin new account to user

* send msg to processor on new account creation

* generate confirm email uri

* remove emailer from account processor again

* add processCreateAccountFromClientAPI

* move emailer accountprocessor => userprocessor

* add email sender to user processor

* SendConfirmEmail function

* add noop email sender

* use noop email sender in tests

* only assemble message if callback is not nil

* use noop email sender if no smtp host is defined

* minify email html before sending

* fix wrong email address

* email confirm test

* fmt

* serve web hndler

* add email confirm handler

* init test log properly on testrig

* log emails that *would* have been sent

* go fmt ./...

* unexport confirm email handler

* updatedAt

* test confirm email function

* don't allow tokens older than 7 days

* change error message a bit

* add basic smtp docs

* add a few more snippets

* typo

* add email sender to outbox tests

* don't use dutch wikipedia link

* don't minify email html
2021-10-31 15:46:23 +01:00
R. Aidan Campbell 083099a957
reference global logrus (#274)
* reference logrus' global logger instead of passing and storing a logger reference everywhere

* always directly use global logrus logger instead of referencing an instance

* test suites should also directly use the global logrus logger

* rename gin logging function to clarify that it's middleware

* correct comments which erroneously referenced removed logger parameter

* setting log level for tests now uses logrus' exported type instead of the string value, to guarantee error isn't possible
2021-10-11 14:37:33 +02:00
tobi e04b187702
Refactor/tidy (#261)
* tidy up streaming

* cut down code duplication

* test get followers/following

* test streaming processor

* fix some test models

* add TimeMustParse

* fix uri / url typo

* make trace logging less verbose

* make logging more consistent

* disable quote on logging

* remove context.Background

* remove many extraneous mastodon references

* regenerate swagger

* don't log query on no rows result

* log latency first for easier reading
2021-10-04 15:24:19 +02:00
tobi 2dc9fc1626
Pg to bun (#148)
* start moving to bun

* changing more stuff

* more

* and yet more

* tests passing

* seems stable now

* more big changes

* small fix

* little fixes
2021-08-25 15:34:33 +02:00
Tobi Smethurst ad0e26dc04
Markdown Statuses (#116)
* parse markdown statuses if desired

* add some preliminary docs for writing posts
2021-07-26 20:25:54 +02:00
Tobi Smethurst 05e9af089c
Oidc (#109)
* add oidc config

* inching forward with oidc idp

* lil webfingy fix

* bit more progress

* further oidc

* oidc now working

* document dex config

* replace broken images

* add additional credits

* tiny doc update

* update

* add oidc config

* inching forward with oidc idp

* bit more progress

* further oidc

* oidc now working

* document dex config

* replace broken images

* add additional credits

* tiny doc update

* update

* document

* docs + comments
2021-07-23 10:36:28 +02:00
Tobi Smethurst bdba3ff9a9
sanitize html for statuses + instance (#97)
* sanitize html for statuses + instance

* sanitization
2021-07-13 16:03:51 +02:00
Tobi Smethurst d389e7b150
Domain block (#76)
* start work on admin domain blocking

* move stuff around + further work on domain blocks

* move + restructure processor

* prep work for deleting account

* tidy

* go fmt

* formatting

* domain blocking more work

* check domain blocks way earlier on

* progress on delete account

* delete more stuff when an account is gone

* and more...

* domain blocky block block

* get individual domain block, delete a block
2021-07-05 13:23:03 +02:00