Commit graph

61 commits

Author SHA1 Message Date
kim
43ac0cdb9c
[chore] Global server configuration overhaul (#575)
* move config flag names and usage to config package, rewrite config package to use global Configuration{} struct

Signed-off-by: kim <grufwub@gmail.com>

* improved code comment

Signed-off-by: kim <grufwub@gmail.com>

* linter

Signed-off-by: kim <grufwub@gmail.com>

* fix unmarshaling

Signed-off-by: kim <grufwub@gmail.com>

* remove kim's custom go compiler changes

Signed-off-by: kim <grufwub@gmail.com>

* generate setter and flag-name functions, implement these in codebase

Signed-off-by: kim <grufwub@gmail.com>

* update deps

Signed-off-by: kim <grufwub@gmail.com>

* small change

Signed-off-by: kim <grufwub@gmail.com>

* appease the linter...

Signed-off-by: kim <grufwub@gmail.com>

* move configuration into ConfigState structure, ensure reloading to/from viper settings to keep in sync

Signed-off-by: kim <grufwub@gmail.com>

* lint

Signed-off-by: kim <grufwub@gmail.com>

* update code comments

Signed-off-by: kim <grufwub@gmail.com>

* fix merge issue

Signed-off-by: kim <grufwub@gmail.com>

* fix merge issue

Signed-off-by: kim <grufwub@gmail.com>

* improved version string (removes time + go version)

Signed-off-by: kim <grufwub@gmail.com>

* fix version string build to pass test script + consolidate logic in func

Signed-off-by: kim <grufwub@gmail.com>

* add license text, update config.Defaults comment

Signed-off-by: kim <grufwub@gmail.com>

* add license text to generated config helpers file

Signed-off-by: kim <grufwub@gmail.com>

* defer unlock on config.Set___(), to ensure unlocked on panic

Signed-off-by: kim <grufwub@gmail.com>

* make it more obvious which cmd flags are being attached

Signed-off-by: kim <grufwub@gmail.com>
2022-05-30 14:41:24 +02:00
tobi
b915a41811
[feature] Basic config validation (#562)
* add optional config validation

* clarify that host and protocol are required

* add validation for host and protocol

* pass prerunArgs as a struct (validate by default)
2022-05-16 14:13:19 +02:00
tobi
b143877995
[feature] Unused avatar and header cleanup (#574)
* rename + tidy up remote pruning

* fix media attachment account join
see https://bun.uptrace.dev/guide/golang-orm.html#table-relationships

* update logging to new function name

* add get avatars and headers to bun

* add pruneallmeta function

* don't set uncached since we're deleting anyway

* fix totalPruned being in wrong place

* test pruning meta

* go fmt ./...

* rename mediaprune

* add meta pruning to routine mediaprune

* tidy up cleanup job scheduling

* rename adminmediaremoteprune

* update mediacleanup to use renamed prune func

* update swagger docs a little bit

* reuse cancel + context
2022-05-15 15:45:04 +01:00
kim
223025fc27
[security] transport.Controller{} and transport.Transport{} security and performance improvements (#564)
* cache transports in controller by privkey-generated pubkey, add retry logic to transport requests

Signed-off-by: kim <grufwub@gmail.com>

* update code comments, defer mutex unlocks

Signed-off-by: kim <grufwub@gmail.com>

* add count to 'performing request' log message

Signed-off-by: kim <grufwub@gmail.com>

* reduce repeated conversions of same url.URL object

Signed-off-by: kim <grufwub@gmail.com>

* move worker.Worker to concurrency subpackage, add WorkQueue type, limit transport http client use by WorkQueue

Signed-off-by: kim <grufwub@gmail.com>

* fix security advisories regarding max outgoing conns, max rsp body size

- implemented by a new httpclient.Client{} that wraps an underlying
  client with a queue to limit connections, and limit reader wrapping
  a response body with a configured maximum size
- update pub.HttpClient args passed around to be this new httpclient.Client{}

Signed-off-by: kim <grufwub@gmail.com>

* add httpclient tests, move ip validation to separate package + change mechanism

Signed-off-by: kim <grufwub@gmail.com>

* fix merge conflicts

Signed-off-by: kim <grufwub@gmail.com>

* use singular mutex in transport rather than separate signer mus

Signed-off-by: kim <grufwub@gmail.com>

* improved useragent string

Signed-off-by: kim <grufwub@gmail.com>

* add note regarding missing test

Signed-off-by: kim <grufwub@gmail.com>

* remove useragent field from transport (instead store in controller)

Signed-off-by: kim <grufwub@gmail.com>

* shutup linter

Signed-off-by: kim <grufwub@gmail.com>

* reset other signing headers on each loop iteration

Signed-off-by: kim <grufwub@gmail.com>

* respect request ctx during retry-backoff sleep period

Signed-off-by: kim <grufwub@gmail.com>

* use external pkg with docs explaining performance "hack"

Signed-off-by: kim <grufwub@gmail.com>

* use http package constants instead of string method literals

Signed-off-by: kim <grufwub@gmail.com>

* add license file headers

Signed-off-by: kim <grufwub@gmail.com>

* update code comment to match new func names

Signed-off-by: kim <grufwub@gmail.com>

* updates to user-agent string

Signed-off-by: kim <grufwub@gmail.com>

* update signed testrig models to fit with new transport logic (instead uses separate signer now)

Signed-off-by: kim <grufwub@gmail.com>

* fuck you linter

Signed-off-by: kim <grufwub@gmail.com>
2022-05-15 11:16:43 +02:00
tobi
d7b46a4b63
[chore] Move initial MediaRemotePrune to cmd.server function (#536)
* move initial prune up to the server start function

* don't do initial prune when starting media manager

* add admin/mediaremoteprune in processor
2022-05-03 10:56:18 +01:00
kim
420e2fb22b
replace async client API / federator msg processing with worker pools (#497)
* replace async client API / federator msg processing with worker pools
* appease our lord-and-saviour, the linter
2022-04-28 13:23:11 +01:00
tobi
094f032f74
[feature] Add log-db-queries config option (#465) 2022-04-18 16:47:11 +02:00
tobi
26683b3d49
[feature] Web profile pages for accounts (#449)
* add default avatars

* allow webModule to error

* return errWithCode from account get

* add AccountGetLocalByUsername

* check nil requesting account

* add timestampShort function for just month/year

* move loading logic to New + add default avatars

* add profile page view

* update swagger docs

* add excludeReblogs to GetAccountStatuses

* ignore casing when selecting local account by username

* appropriate redirects

* css fiddling

* add 'about' heading

* adjust thread page to work with routing

* return AP representation if requested + authorized

* simplify auth check

* go fmt

* golangci-lint ignore math/rand
2022-04-15 14:33:01 +02:00
tobi
03d7c75ebf
[chore] Update Go version to 1.18 (#444)
* linting with new golangci-lint version

* update go to 1.18

* bump versions in drone.yml

* use new runtime/debug package for version info

* remove Commit build flag from goreleaser

* remove mock commit + version from build script

* go fmt

* add dummy version env flag to test container

* install git in golang container for testing

* only set versionString if Version is defined
2022-04-02 15:40:09 +02:00
tobi
e63b653199
[performance] Add dereference shortcuts to avoid making http calls to self (#430)
* update transport (controller) to allow shortcuts

* go fmt

* expose underlying sig transport to allow test sigs
2022-03-15 15:01:19 +01:00
tobi
07727753b9
[feature] Clean up/uncache remote media (#407)
* Add whereNotEmptyAndNotNull

* Add GetRemoteOlderThanDays

* Add GetRemoteOlderThanDays

* Add PruneRemote to Manager interface

* Start implementing PruneRemote

* add new attachment + status to tests

* fix up and test GetRemoteOlderThan

* fix bad import

* PruneRemote: return number pruned

* add Cached column to mediaattachment

* update + test pruneRemote

* update mediaTest

* use Cached column

* upstep bun to latest version

* embed structs in mediaAttachment

* migrate mediaAttachment to new format

* don't default cached to true

* select only remote media

* update db dependencies

* step bun back to last working version

* update pruneRemote to use Cached field

* fix storage path of test attachments

* add recache logic to manager

* fix trimmed aspect ratio

* test prune and recache

* return errwithcode

* tidy up different paths for emoji vs attachment

* fix incorrect thumbnail type being stored

* expose TransportController to media processor

* implement tee-ing recached content

* add thoughts of dog to test fedi attachments

* test get remote files

* add comment on PruneRemote

* add postData cleanup to recache

* test thumbnail fetching

* add incredible diagram

* go mod tidy

* buffer pipes for recache streaming

* test for client stops reading after 1kb

* add media-remote-cache-days to config

* add cron package

* wrap logrus so it's available to cron

* start and stop cron jobs gracefully
2022-03-07 11:08:26 +01:00
tsmethurst
b42b0a667e go fmt 2022-02-08 12:19:54 +01:00
tsmethurst
3301148bb7 merge fixup 2022-02-08 12:17:53 +01:00
tsmethurst
4c294a596a Merge branch 'main' into media_refactor 2022-02-08 12:17:18 +01:00
tsmethurst
9db58dfd57 add comment explaining store.lock 2022-02-08 10:06:43 +01:00
tobi
1b36e85840
[feature] Rework timeline code to make it useful for more than just statuses (#373)
* add preparable and timelineable interfaces

* initialize timeline manager within the processor

* generic renaming

* move status-specific timeline logic into the processor

* refactor timeline to make it useful for more than statuses
2022-02-05 12:47:38 +01:00
tsmethurst
d9a778a795 put store lock file in base of storage 2022-01-29 12:16:12 +01:00
tsmethurst
e0f9323b9a test the media manager a bit, add shutdown logic 2022-01-10 18:36:09 +01:00
tsmethurst
dccf21dd87 tests are passing, but there's still much to be done 2022-01-09 18:41:22 +01:00
tsmethurst
c4d63d125b more refactoring, media handler => manager 2021-12-28 16:36:00 +01:00
tobi
ef5a9256a8
Extend license notices to 2022 (#354) 2021-12-20 18:42:19 +01:00
tobi
cb8688f429
Remove unnecessary storage config variables (#344)
* rewire config to not use extraneous serve vars

* rename 'file' to 'local' for consistency

* use Type and Size again
2021-12-20 15:19:53 +01:00
tobi
c111b239f7
Add optional syslog logrus hook (#343)
* add optional syslog logrus hook

* document syslog
2021-12-12 18:00:20 +01:00
tobi
0884f89431
Implement Cobra CLI tooling, Viper config tooling (#336)
* start pulling out + replacing urfave and config

* replace many many instances of config

* move more stuff => viper

* properly remove urfave

* move some flags to root command

* add testrig commands to root

* alias config file keys

* start adding cli parsing tests

* reorder viper init

* remove config path alias

* fmt

* change config file keys to non-nested

* we're more or less in business now

* tidy up the common func

* go fmt

* get tests passing again

* add note about the cliparsing tests

* reorganize

* update docs with changes

* structure cmd dir better

* rename + move some files around

* fix dangling comma
2021-12-07 13:31:39 +01:00
Forest Johnson
ab316d2250
Fix broken defaults and broken flags (#314)
* start with a default config, not an empty config.

* some data structures were present on Empty config but not Default config

* the monkey patched CLIContext is working

* remove print debugging log

* make the behaviour of the flags consistent across all data types

Conflicts:
	internal/config/config.go

* try to fix accidentally broken test
2021-11-24 17:01:38 +01:00
Mara Sophie Grosch
a7882fabc7
Use IPv6 doc prefix for docs (#324)
IPv6 has a dedicated prefix for documentation and examples, this commit makes use of it for BindAddress config flag docs and examples.
2021-11-22 17:58:19 +01:00
tobi
1ded58b34b
add bindAddress configuration option (#320)
* add bindAddress configuration option

* clarify that bindAddress can be a hostname
2021-11-22 10:55:52 +01:00
tobi
f8630348b4
Enable stricter linting with golangci-lint (#316)
* update golangci-lint

* add golangci config file w/ more linters

* correct issues flagged by stricter linters

* add more generous timeout for golangci-lint

* add some style + formatting guidelines

* move timeout to config file

* go fmt
2021-11-22 08:46:19 +01:00
tobi
2aaec82732
smtp + email confirmation (#285)
* add smtp configuration

* add email confirm + reset templates

* add email sender to testrig

* flesh out the email sender interface

* go fmt

* golint

* update from field with more clarity

* tidy up the email formatting

* fix tests

* add email sender to processor

* tidy client api processing a bit

* further tidying in fromClientAPI

* pin new account to user

* send msg to processor on new account creation

* generate confirm email uri

* remove emailer from account processor again

* add processCreateAccountFromClientAPI

* move emailer accountprocessor => userprocessor

* add email sender to user processor

* SendConfirmEmail function

* add noop email sender

* use noop email sender in tests

* only assemble message if callback is not nil

* use noop email sender if no smtp host is defined

* minify email html before sending

* fix wrong email address

* email confirm test

* fmt

* serve web hndler

* add email confirm handler

* init test log properly on testrig

* log emails that *would* have been sent

* go fmt ./...

* unexport confirm email handler

* updatedAt

* test confirm email function

* don't allow tokens older than 7 days

* change error message a bit

* add basic smtp docs

* add a few more snippets

* typo

* add email sender to outbox tests

* don't use dutch wikipedia link

* don't minify email html
2021-10-31 15:46:23 +01:00
R. Aidan Campbell
083099a957
reference global logrus (#274)
* reference logrus' global logger instead of passing and storing a logger reference everywhere

* always directly use global logrus logger instead of referencing an instance

* test suites should also directly use the global logrus logger

* rename gin logging function to clarify that it's middleware

* correct comments which erroneously referenced removed logger parameter

* setting log level for tests now uses logrus' exported type instead of the string value, to guarantee error isn't possible
2021-10-11 14:37:33 +02:00
tobi
231075f28d
Golint (#255) 2021-09-30 11:16:23 +02:00
tobi
555ea8edfb
Import export (#194)
* start with export/import code

* messing about with decoding/encoding

* some more fiddling

* stuff is WORKING

* working pretty alright!

* go fmt

* fix up tests, add docs

* start backup/restore doc

* tweaks

* credits

* update advancedVisibility settings

* update bun library -> v1.0.4

Signed-off-by: kim (grufwub) <grufwub@gmail.com>

* update oauth library -> v4.3.1-SSB

Signed-off-by: kim (grufwub) <grufwub@gmail.com>

* handle oauth token scope, fix user.SigninCount + token.UserID

Signed-off-by: kim (grufwub) <grufwub@gmail.com>

* update oauth library --> v4.3.2-SSB

Signed-off-by: kim (grufwub) <grufwub@gmail.com>

* update sqlite library -> v1.13.0

Signed-off-by: kim (grufwub) <grufwub@gmail.com>

* review changes

* start with export/import code

* messing about with decoding/encoding

* some more fiddling

* stuff is WORKING

* working pretty alright!

* go fmt

* fix up tests, add docs

* start backup/restore doc

* tweaks

* credits

* update advancedVisibility settings

* review changes

Co-authored-by: kim (grufwub) <grufwub@gmail.com>
Co-authored-by: kim <89579420+NyaaaWhatsUpDoc@users.noreply.github.com>
2021-09-09 16:15:25 +02:00
Tobi Smethurst
58dddd86e0
Swagger (#124)
* start experimenting with swagger documentation

* further adventures in swagger

* do a few more api paths

* account paths documented

* go fmt

* fix up some models

* bit o lintin'
2021-07-31 17:49:59 +02:00
Tobi Smethurst
eb13faf54f
Password change (#123)
* add password change command

* document cli commands

* go fmt
2021-07-31 13:57:23 +02:00
Tobi Smethurst
e2757ae676
add trusted proxy for parsing client IPs (#115) 2021-07-26 16:15:36 +02:00
tsmethurst
318e377359 allow custom ports for webserver and le 2021-07-24 18:55:24 +02:00
Tobi Smethurst
05e9af089c
Oidc (#109)
* add oidc config

* inching forward with oidc idp

* lil webfingy fix

* bit more progress

* further oidc

* oidc now working

* document dex config

* replace broken images

* add additional credits

* tiny doc update

* update

* add oidc config

* inching forward with oidc idp

* bit more progress

* further oidc

* oidc now working

* document dex config

* replace broken images

* add additional credits

* tiny doc update

* update

* document

* docs + comments
2021-07-23 10:36:28 +02:00
Tobi Smethurst
b1a4f38e38
allow different host + accountDomain (#103)
* allow different host + accountDomain

* use accountDomain in tags
2021-07-19 18:42:08 +02:00
Tobi Smethurst
677490bc4e
Db tls (#102)
* go mod tidy

* complete example config

* add tls support for db connection

* add certpool to tlsConfig

* add some lil docker scripts
2021-07-19 18:03:07 +02:00
Tobi Smethurst
4f3b3f5c0b
put version in binary properly (#73)
Addresses #71 :

    Set version on the CLI framework.
    Add a build.sh script that injects variables into the build tooling using git and a version file.
    Set version in config.
2021-06-28 12:17:20 +02:00
f0x52
f9bc305aca
new styling for frontpage, update login and authorize templates (#46)
* new styling for frontpage, update login and authorize templates

* run go fmt

* add AssetBaseDir to command flag parsing

* untested: move landing page to it's own router

* go fmt, fix typo

* fix package, adapt to proper Route structure
2021-06-21 19:46:10 +02:00
Tobi Smethurst
efbd839181
Testrig fixes (#50)
* testrig is runnable again
* little fixes, add some more test models
* address https://github.com/superseriousbusiness/gotosocial/issues/44
2021-06-21 12:27:23 +02:00
Tobi Smethurst
3d77f81c7f
Move a lot of stuff + tidy stuff (#37)
Lots of renaming and moving stuff, some bug fixes, more lenient parsing of notifications and home timeline.
2021-05-30 13:12:00 +02:00
Tobi Smethurst
ee65d19ff3
status deletes, profile updates (#30)
1. Proper DELETE of federated statuses (not yet deleting all the media and stuff -- i still have to implement this -- but the actual status is toast).
2. Proper UPDATE of profiles. When you change your profile picture on your remote instance, that will now register properly in GoToSocial.
3. Scrolling down the home timeline - it no longer just sort of ends, and will keep loading older statuses as you scroll.
4. Little bugfixes -- still had some nil pointer errors when dereferencing remote accounts.
2021-05-23 18:07:04 +02:00
Tobi Smethurst
43c3a47773
Admin cli (#29)
Now you can use the CLI tool to:

* Create a new account with the given username, email address and password (which will be hashed of course).
* Confirm the account's so that it can log in and post.
* Promote the account to admin.
* Demote the account from admin.
* Disable the account.
* Suspend the account.
2021-05-22 14:26:45 +02:00
Tobi Smethurst
0cbab627c7
Letsencrypt (#17) 2021-05-09 11:25:13 +02:00
Tobi Smethurst
32c5fd987a
Api/v1/statuses (#11)
This PR adds:
Statuses

    New status creation.
    View existing status
    Delete a status
    Fave a status
    Unfave a status
    See who's faved a status

Media

    Upload media attachment and store/retrieve it
    Upload custom emoji and store/retrieve it

Fileserver

    Serve files from storage

Testing

    Test models, testrig -- run a GTS test instance and play around with it.
2021-04-19 19:42:19 +02:00
Tobi Smethurst
71a49e2b43
Api/v1/accounts (#8)
* start work on accounts module

* plodding away on the accounts endpoint

* groundwork for other account routes

* add password validator

* validation utils

* require account approval flags

* comments

* comments

* go fmt

* comments

* add distributor stub

* rename api to federator

* tidy a bit

* validate new account requests

* rename r router

* comments

* add domain blocks

* add some more shortcuts

* add some more shortcuts

* check email + username availability

* email block checking for signups

* chunking away at it

* tick off a few more things

* some fiddling with tests

* add mock package

* relocate repo

* move mocks around

* set app id on new signups

* initialize oauth server properly

* rename oauth server

* proper mocking tests

* go fmt ./...

* add required fields

* change name of func

* move validation to account.go

* more tests!

* add some file utility tools

* add mediaconfig

* new shortcut

* add some more fields

* add followrequest model

* add notify

* update mastotypes

* mock out storage interface

* start building media interface

* start on update credentials

* mess about with media a bit more

* test image manipulation

* media more or less working

* account update nearly working

* rearranging my package ;) ;) ;)

* phew big stuff!!!!

* fix type checking

* *fiddles*

* Add CreateTables func

* account registration flow working

* tidy

* script to step through auth flow

* add a lil helper for generating user uris

* fiddling with federation a bit

* update progress

* Tidying and linting
2021-04-01 20:46:45 +02:00
Tobi Smethurst
aa9ce272dc
Oauth/token (#7)
* add host and protocol options

* some fiddling

* tidying up and comments

* tick off /oauth/token

* tidying a bit

* tidying

* go mod tidy

* allow attaching middleware to server

* add middleware

* more user friendly

* add comments

* comments

* store account + app

* tidying

* lots of restructuring

* lint + tidy
2021-03-22 22:26:54 +01:00
tsmethurst
1b11884121 auth flow working for code 2021-03-18 23:27:43 +01:00