Commit graph

23 commits

Author SHA1 Message Date
kim 5d55e8d920
[performance] add account block DB cache and remove block query joins (#1085)
* add account block DB cache and remove reliance on relational joins

* actually include cache key arguments...

* add a PutBlock() method which also updates the block cache, update tests accordingly

* use `PutBlock` instead of `Put(ctx, block)`

* add + use functions for deleting + invalidating blocks

Signed-off-by: kim <grufwub@gmail.com>
Co-authored-by: tsmethurst <tobi.smethurst@protonmail.com>
2022-11-20 16:33:49 +00:00
tobi 70d65b683f
[feature] Refetch emojis when they change on remote instances (#905)
* select emoji using image_static_url

* use updated on AP emojis

* allow refetch of updated emojis

* cheeky workaround for test

* clean up old files for refreshed emoji

* check error for originalPostData

* shorten GetEmojiByStaticImageURL

* delete kirby (sorry nintendo)
2022-10-13 15:16:24 +02:00
kim 2f22780800
[chore] simplify generating log entry caller information (#863)
* vastly simplify logging caller information

Signed-off-by: kim <grufwub@gmail.com>

* fix failing test due to multiple calls to processor.Start()

Signed-off-by: kim <grufwub@gmail.com>

Signed-off-by: kim <grufwub@gmail.com>
2022-09-29 12:02:12 +02:00
kim a156188b3e
[chore] update dependencies, bump to Go 1.19.1 (#826)
* update dependencies, bump Go version to 1.19

* bump test image Go version

* update golangci-lint

* update gotosocial-drone-build

* sign

* linting, go fmt

* update swagger docs

* update swagger docs

* whitespace

* update contributing.md

* fuckin whoopsie doopsie

* linterino, linteroni

* fix followrequest test not starting processor

* fix other api/client tests not starting processor

* fix remaining tests where processor not started

* bump go-runners version

* don't check last-webfingered-at, processor may have updated this

* update swagger command

* update bun to latest version

* fix embed to work the same as before with new bun

Signed-off-by: kim <grufwub@gmail.com>
Co-authored-by: tsmethurst <tobi.smethurst@protonmail.com>
2022-09-28 18:30:40 +01:00
tobi c4a08292ee
[feature] Show + federate emojis in accounts (#837)
* Start adding account emoji

* get emojis serialized + deserialized nicely

* update tests

* set / retrieve emojis on accounts

* show account emojis in web view

* fetch emojis from db based on ids

* fix typo in test

* lint

* fix pg migration

* update tests

* update emoji checking logic

* update comment

* clarify comments + add some spacing

* tidy up loops a lil (thanks kim)
2022-09-26 11:56:01 +02:00
tobi 0245c606d7
[chore] Test fixes (#788)
* use 'test' value for testrig storage backend

* update test dependency

* add WaitFor func in testrig

* use WaitFor function instead of time.Sleep

* tidy up tests

* make SentMessages a sync.map

* go fmt
2022-08-31 17:31:21 +02:00
tobi ac6ed3d939
[chore] Update bun / sqlite versions; update gtsmodels (#754)
* upstep bun and sqlite versions

* allow specific columns to be updated in the db

* only update necessary columns for user

* bit tidier

* only update necessary fields of media_attachment

* only update relevant instance fields

* update tests

* update only specific account columns

* use bool pointers on gtsmodels
includes attachment, status, account, user

* update columns more selectively

* test all default fields on new account insert

* updating remaining bools on gtsmodels

* initialize pointer fields when extracting AP emoji

* copy bools properly

* add copyBoolPtr convenience function + test it

* initialize false bool ptrs a bit more neatly
2022-08-15 11:35:05 +01:00
kim 6934ae378a
[chore] improved router logging, recovery and error handling (#705)
* move panic recovery to logging middleware, improve logging + panic recovery logic

Signed-off-by: kim <grufwub@gmail.com>

* remove dead code

Signed-off-by: kim <grufwub@gmail.com>

* remove skip paths code

Signed-off-by: kim <grufwub@gmail.com>

* re-enable log quoting

Signed-off-by: kim <grufwub@gmail.com>

* use human-friendly bytesize in logging body size

Signed-off-by: kim <grufwub@gmail.com>

* only disable quoting in debug builds

Signed-off-by: kim <grufwub@gmail.com>

* use logrus level instead of debug.DEBUG() to enable/disable quoting

Signed-off-by: kim <grufwub@gmail.com>

* shutup linter

Signed-off-by: kim <grufwub@gmail.com>

* fix instance tests

Signed-off-by: kim <grufwub@gmail.com>

* fix gin test contexts created with missing engine HTML renderer

Signed-off-by: kim <grufwub@gmail.com>

* add note regarding not logging query parameters

Signed-off-by: kim <grufwub@gmail.com>

* better explain 'DisableQuoting' logic

Signed-off-by: kim <grufwub@gmail.com>

* add license text

Signed-off-by: kim <grufwub@gmail.com>
2022-07-12 08:32:20 +01:00
tobi dfdc473cef
[chore] Webfinger rework (#627)
* move finger to dereferencer

* totally break GetRemoteAccount

* start reworking finger func a bit

* start reworking getRemoteAccount a bit

* move mention parts to namestring

* rework webfingerget

* use util function to extract webfinger parts

* use accountDomain

* rework finger again, final form

* just a real nasty commit, the worst

* remove refresh from account

* use new ASRepToAccount signature

* fix incorrect debug call

* fix for new getRemoteAccount

* rework GetRemoteAccount

* start updating tests to remove repetition

* break a lot of tests
Move shared test logic into the testrig,
rather than having it scattered all over
the place. This allows us to just mock
the transport controller once, and have
all tests use it (unless they need not to
for some other reason).

* fix up tests to use main mock httpclient

* webfinger only if necessary

* cheeky linting with the lads

* update mentionName regex
recognize instance accounts

* don't finger instance accounts

* test webfinger part extraction

* increase default worker count to 4 per cpu

* don't repeat regex parsing

* final search for discovered accountDomain

* be more permissive in namestring lookup

* add more extraction tests

* simplify GetParseMentionFunc

* skip long search if local account

* fix broken test
2022-06-11 11:01:34 +02:00
kim 223025fc27
[security] transport.Controller{} and transport.Transport{} security and performance improvements (#564)
* cache transports in controller by privkey-generated pubkey, add retry logic to transport requests

Signed-off-by: kim <grufwub@gmail.com>

* update code comments, defer mutex unlocks

Signed-off-by: kim <grufwub@gmail.com>

* add count to 'performing request' log message

Signed-off-by: kim <grufwub@gmail.com>

* reduce repeated conversions of same url.URL object

Signed-off-by: kim <grufwub@gmail.com>

* move worker.Worker to concurrency subpackage, add WorkQueue type, limit transport http client use by WorkQueue

Signed-off-by: kim <grufwub@gmail.com>

* fix security advisories regarding max outgoing conns, max rsp body size

- implemented by a new httpclient.Client{} that wraps an underlying
  client with a queue to limit connections, and limit reader wrapping
  a response body with a configured maximum size
- update pub.HttpClient args passed around to be this new httpclient.Client{}

Signed-off-by: kim <grufwub@gmail.com>

* add httpclient tests, move ip validation to separate package + change mechanism

Signed-off-by: kim <grufwub@gmail.com>

* fix merge conflicts

Signed-off-by: kim <grufwub@gmail.com>

* use singular mutex in transport rather than separate signer mus

Signed-off-by: kim <grufwub@gmail.com>

* improved useragent string

Signed-off-by: kim <grufwub@gmail.com>

* add note regarding missing test

Signed-off-by: kim <grufwub@gmail.com>

* remove useragent field from transport (instead store in controller)

Signed-off-by: kim <grufwub@gmail.com>

* shutup linter

Signed-off-by: kim <grufwub@gmail.com>

* reset other signing headers on each loop iteration

Signed-off-by: kim <grufwub@gmail.com>

* respect request ctx during retry-backoff sleep period

Signed-off-by: kim <grufwub@gmail.com>

* use external pkg with docs explaining performance "hack"

Signed-off-by: kim <grufwub@gmail.com>

* use http package constants instead of string method literals

Signed-off-by: kim <grufwub@gmail.com>

* add license file headers

Signed-off-by: kim <grufwub@gmail.com>

* update code comment to match new func names

Signed-off-by: kim <grufwub@gmail.com>

* updates to user-agent string

Signed-off-by: kim <grufwub@gmail.com>

* update signed testrig models to fit with new transport logic (instead uses separate signer now)

Signed-off-by: kim <grufwub@gmail.com>

* fuck you linter

Signed-off-by: kim <grufwub@gmail.com>
2022-05-15 11:16:43 +02:00
kim 420e2fb22b
replace async client API / federator msg processing with worker pools (#497)
* replace async client API / federator msg processing with worker pools
* appease our lord-and-saviour, the linter
2022-04-28 13:23:11 +01:00
tobi 26683b3d49
[feature] Web profile pages for accounts (#449)
* add default avatars

* allow webModule to error

* return errWithCode from account get

* add AccountGetLocalByUsername

* check nil requesting account

* add timestampShort function for just month/year

* move loading logic to New + add default avatars

* add profile page view

* update swagger docs

* add excludeReblogs to GetAccountStatuses

* ignore casing when selecting local account by username

* appropriate redirects

* css fiddling

* add 'about' heading

* adjust thread page to work with routing

* return AP representation if requested + authorized

* simplify auth check

* go fmt

* golangci-lint ignore math/rand
2022-04-15 14:33:01 +02:00
tsmethurst e0f9323b9a test the media manager a bit, add shutdown logic 2022-01-10 18:36:09 +01:00
tobi ef5a9256a8
Extend license notices to 2022 (#354) 2021-12-20 18:42:19 +01:00
tobi 0884f89431
Implement Cobra CLI tooling, Viper config tooling (#336)
* start pulling out + replacing urfave and config

* replace many many instances of config

* move more stuff => viper

* properly remove urfave

* move some flags to root command

* add testrig commands to root

* alias config file keys

* start adding cli parsing tests

* reorder viper init

* remove config path alias

* fmt

* change config file keys to non-nested

* we're more or less in business now

* tidy up the common func

* go fmt

* get tests passing again

* add note about the cliparsing tests

* reorganize

* update docs with changes

* structure cmd dir better

* rename + move some files around

* fix dangling comma
2021-12-07 13:31:39 +01:00
tobi 09ef9e639e
move to ssb gofed fork (#298) 2021-11-13 17:29:43 +01:00
tobi 2aaec82732
smtp + email confirmation (#285)
* add smtp configuration

* add email confirm + reset templates

* add email sender to testrig

* flesh out the email sender interface

* go fmt

* golint

* update from field with more clarity

* tidy up the email formatting

* fix tests

* add email sender to processor

* tidy client api processing a bit

* further tidying in fromClientAPI

* pin new account to user

* send msg to processor on new account creation

* generate confirm email uri

* remove emailer from account processor again

* add processCreateAccountFromClientAPI

* move emailer accountprocessor => userprocessor

* add email sender to user processor

* SendConfirmEmail function

* add noop email sender

* use noop email sender in tests

* only assemble message if callback is not nil

* use noop email sender if no smtp host is defined

* minify email html before sending

* fix wrong email address

* email confirm test

* fmt

* serve web hndler

* add email confirm handler

* init test log properly on testrig

* log emails that *would* have been sent

* go fmt ./...

* unexport confirm email handler

* updatedAt

* test confirm email function

* don't allow tokens older than 7 days

* change error message a bit

* add basic smtp docs

* add a few more snippets

* typo

* add email sender to outbox tests

* don't use dutch wikipedia link

* don't minify email html
2021-10-31 15:46:23 +01:00
tobi 4b1d9d3780
Serve outbox for Actor (#289)
* add statusesvisible convenience function

* add minID + onlyPublic to account statuses get

* move swagger collection stuff to common

* start working on Outbox GETting

* move functions into federationProcessor

* outboxToASCollection

* add statusesvisible convenience function

* add minID + onlyPublic to account statuses get

* move swagger collection stuff to common

* start working on Outbox GETting

* move functions into federationProcessor

* outboxToASCollection

* bit more work on outbox paging

* wrapNoteInCreate function

* test + hook up the processor functions

* don't do prev + next links on empty reply

* test get outbox through api

* don't fail on no status entries

* add outbox implementation doc

* typo
2021-10-24 11:57:39 +02:00
R. Aidan Campbell 083099a957
reference global logrus (#274)
* reference logrus' global logger instead of passing and storing a logger reference everywhere

* always directly use global logrus logger instead of referencing an instance

* test suites should also directly use the global logrus logger

* rename gin logging function to clarify that it's middleware

* correct comments which erroneously referenced removed logger parameter

* setting log level for tests now uses logrus' exported type instead of the string value, to guarantee error isn't possible
2021-10-11 14:37:33 +02:00
tobi 3dc7644ae6
Derive visibility fixes (#271)
* use pub public const

* don't error on no summary

* move extract visibility to separate function

* extract visibility test

* add addressable interface
2021-10-06 18:18:02 +02:00
tobi 36a09dd0df
handle remote account deletion more systematically (#254) 2021-09-30 10:56:02 +02:00
tobi b5a7e1ba32
Account update issue (#250)
* start poking around

* tests

* notes and fiddling
2021-09-28 15:21:59 +02:00
tobi c7cfbe2702
Unblock fix (#247)
* start tests for inbox posts

* go mod tidy

* rename transferContext

* test block/unblock

* improve logging

* improve logging

* fix comment typo
2021-09-24 17:56:48 +02:00