Commit graph

639 commits

Author SHA1 Message Date
tobi
bfccf4e450
[bugfix] add ON CONFLICT statements to status updates (#1580) 2023-03-02 16:58:23 +01:00
Vyr Cossont
e6cde25466
[feature] Advertise rich text formats, support content_type field (#1370)
* Advertise rich text formats, support content_type field

* Update JSON in instance patch tests

* Replace format with content_type everywhere

* update migration to work with both pg and sqlite

* regenerate swagger docs

* update instance serialization + tests

* fix up

* learn to code tobi please, i'm begging you

---------

Co-authored-by: tsmethurst <tobi.smethurst@protonmail.com>
2023-03-02 12:06:40 +01:00
kim
baf933cb9f
[chore] move client/federator workerpools to Workers{} (#1575)
* replace concurrency worker pools with base models in State.Workers, update code and tests accordingly

* improve code comment

* change back testrig default log level

* un-comment-out TestAnnounceTwice() and fix

---------

Signed-off-by: kim <grufwub@gmail.com>
Reviewed-by: tobi
2023-03-01 18:26:53 +00:00
tobi
24cec4e7aa
[feature] Federate pinned posts (aka featuredCollection) in and out (#1560)
* start fiddling

* the ol' fiddle + update

* start working on fetching statuses

* poopy doopy doo where r u uwu

* further adventures in featuring statuses

* finishing up

* fmt

* simply status unpin loop

* move empty featured check back to caller function

* remove unnecessary log.WithContext calls

* remove unnecessary IsIRI() checks

* add explanatory comment about status URIs

* change log level to error

* better test names
2023-03-01 17:52:44 +00:00
kim
87c5c42972
[chore/performance] simplify storage driver to use storage.Storage directly (#1576)
* simply use storage.Storage, removing wrapping KVStore as we don't need KV store locking functionality

Signed-off-by: kim <grufwub@gmail.com>

* fix missing unwrapped function

Signed-off-by: kim <grufwub@gmail.com>

* add code comment

Signed-off-by: kim <grufwub@gmail.com>

* linter, please take my offering in peace

Signed-off-by: kim <grufwub@gmail.com>

---------

Signed-off-by: kim <grufwub@gmail.com>
2023-03-01 10:44:54 +01:00
Daenney
e4c5f9adfd
[chore] Improve unsupported_grant_type error (#1572)
This attempts to provide a slightly more comprehensive error message for
the end user when an incorrect grant type is used. This is not something
the user can typically resolve but should hopefully be informative for
the (client) developer.
2023-02-28 11:38:34 +01:00
Daenney
9cfb69f75d
[feature] Make OIDC admin groups configurable (#1555)
This removes the current default of checking for membership of the admin
or admins group and makes it required to explicitly configure which
groups should grant admin access, if any.

Relying on the implicit default of admin or admins is potentially
dangerous as that group may contain a different subset of people that we
may wish to grant admin access to GtS. This is probably not an issue for
a single-person instance, but for a community instance different admin
groups may exist in an OIDC provider for different applications.

I'm explicitly opting for not defaulting the value of oidc-admin-groups
to admin,admins because I think it's better for those things to be
explicitly configured.
2023-02-25 16:37:39 +00:00
tobi
c27b4d7ed0
[feature] Client API endpoints + v. basic web view for pinned posts (#1547)
* implement status pin client api + web handler

* make test names + comments more descriptive

* don't use separate table for status pins

* remove unused add + remove checking

* tidy up + add some more tests
2023-02-25 12:16:30 +00:00
tobi
689a10fe17
[bugfix] Fix deleted status causing issues when getting bookmark (#1551)
* [bugfix] Delete bookmark when status deleted

* [chore] Give bookmark processing func some love

* fix paging + embetter tests
2023-02-23 22:11:10 +00:00
Daenney
074f352709
[chore] improve opengraph descripiton tag (#1550)
This changes parseDescription to properly encode things to be safe for
usage without removing things like backslashes that may be relevant.

* text.SanitizePlaintext already calls html.UnescapeString so we don't
  have to do that
* Replace \n with space early
* Remove duplicate white-space by splitting on fields and joining
* HTML-escape the string we have
* For extra certainty, encode the backslash as &bsol;

Fixes #1549
2023-02-22 21:36:18 +00:00
tobi
b6fbdc66c1
[chore] Deinterface processor and subprocessors (#1501)
* [chore] Deinterface processor and subprocessors

* expose subprocessors via function calls

* missing license header
2023-02-22 16:05:26 +01:00
tobi
adb596600b
[bugfix] Remove initial storage cleanup (#1545) 2023-02-21 10:49:57 +01:00
tobi
e8a04b7ce1
[bugfix] Fix account roles (#1542)
* Change account role from string to object

* Update tests

* small fixes + swagger docs

---------

Co-authored-by: zowhoey <11893985+zowhoey@users.noreply.github.com>
2023-02-20 17:00:44 +01:00
f0x52
b6143c9ab8
[feature] About page (#1495)
* about page basics

* more info, styling

* update emoji sizing on about page contact card
2023-02-20 16:29:29 +01:00
Sam Lade
f559d46261
[bugfix] Fix failure to look up remote profiles with duplicate emojis in some cases (#1534)
* Tidy up emoji parsing on profile submission

Don't bother reparsing for emoji unless one of the fields that can have
emoji in it has changed.
Deduplicate emoji between the display name and profile note - I'm not
sure whether this was hurting anything, but better safe.

* Deduplicate emoji when parsing remote accounts

Some servers - Misskey at least - don't deduplicate emoji, so it's
possible to get an account which has the same emoji used in both the
display name and note and therefore includes that emoji twice in its
metadata. When we start trying to put those into our database, we run
into a uniqueness constraint and fall over.

This change just deduplicates at the point of construction of an
account.
2023-02-20 16:27:41 +01:00
Daenney
70398891b8
[chore] Move request validation earlier in client (#1531)
This moves checking if the request is valid as early as possible in the
chain. This should ensure that for an invalid request we never bother
acquiring the wait queue and taking up a spot in it.
2023-02-19 11:01:15 +00:00
tobi
a0068e8915
[bugfix] In Postgres, drop shortcodedomain constraint before creating new emoji table (#1528) 2023-02-18 17:54:51 +01:00
kim
a684fc4628
[chore] transport improvements (#1524)
* improve error readability, mark "bad hosts" as fastFail

Signed-off-by: kim <grufwub@gmail.com>

* pull in latest go-byteutil version with byteutil.Reader{}

Signed-off-by: kim <grufwub@gmail.com>

* use rewindable body reader for post requests

Signed-off-by: kim <grufwub@gmail.com>

---------

Signed-off-by: kim <grufwub@gmail.com>
2023-02-18 17:02:19 +01:00
mushus
fe66a2aed2
[bugfix] Keep png transparency (#1522)
* keep png transparency

* rewrite to switch case
2023-02-18 16:52:15 +01:00
f0x52
59b2e10787
[bugfix] fix oob token route, update templates+css for oob and errors (#1519) 2023-02-18 16:47:42 +01:00
Daenney
b4d18887d3
[chore] Update more log calls to include context (#1517)
In #1476 we updated log.WithFields() but we forgot about
log.WithField(). Also updates a few explicit log.Entry{} creations.
2023-02-17 19:05:43 +00:00
Daenney
d39280ec33
[bug] Pass context in logging middleware (#1514)
This updates the middleware log.WithField calls that create new loggers
to include the context the first time around. Without it the requestID
does not get logged.

Fixup from #1476
2023-02-17 12:37:57 +01:00
Daenney
68e6d08c76
[feature] Add a request ID and include it in logs (#1476)
This adds a lightweight form of tracing to GTS. Each incoming request is
assigned a Request ID which we then pass on and log in all our log
lines. Any function that gets called downstream from an HTTP handler
should now emit a requestID=value pair whenever it logs something.

Co-authored-by: kim <grufwub@gmail.com>
2023-02-17 12:02:29 +01:00
tobi
b5993095fa
[bugfix] Set 'discoverable' properly on API accounts (#1511) 2023-02-16 13:20:23 +00:00
tobi
6ee0dc8c7d
[bugfix] Set cache-control max-age dynamically for s3 (#1510)
* [bugfix] set cache-control max-age dynamically for s3

* woops

* double whoops

* time until, thank you linter, bless you, you're the best, no matter what kim says

* aa
2023-02-16 13:18:53 +00:00
Sam Lade
40b584c219
Fix 410 Gone race on account deletes (#1507) 2023-02-15 19:41:16 +00:00
tobi
6c6f042290
[bugfix] Return empty result rather than 500 error when searching for blocked domains (#1498)
* [bugfix] Return empty result when searching for blocked domains

* add tests
2023-02-14 11:55:02 +01:00
tobi
561ad71e58
[bugfix] Fix up error getting account avatar/header errors, other small fixes (#1496)
* start fiddling with media + account queries a little

* initialize state when pruning

* allow for unsetting remote media
make sure to wait til media loaded
fix silly tiny bug

* move comment a bit for readability

* slight reformat of fetchRemoteAccount{Avatar,Header}

* fix issue after rebase

* slightly neaten up logic of avatar/header media handling

* remove log prefix (callername log field handles this)

---------

Signed-off-by: kim <grufwub@gmail.com>
Co-authored-by: kim <grufwub@gmail.com>
2023-02-13 20:19:51 +00:00
kim
acc95923da
[performance] processing media and scheduled jobs improvements (#1482)
* replace media workers with just runners.WorkerPool, move to state structure, use go-sched for global task scheduling

* improved code comment

* fix worker tryUntil function, update go-runners/go-sched

* make preprocess functions package public, use these where possible to stop doubled up processing

* remove separate emoji worker pool

* limit calls to time.Now() during media preprocessing

* use Processor{} to manage singular runtime of processing media

* ensure workers get started when media manager is used

* improved error setting in processing media, fix media test

* port changes from processingmedia to processing emoji

* finish code commenting

* finish code commenting and comment-out client API + federator worker pools until concurrency worker pools replaced

* linterrrrrrrrrrrrrrrr

---------

Signed-off-by: kim <grufwub@gmail.com>
2023-02-13 18:40:48 +00:00
tobi
041c8e695e
[chore] Do cache-control in a less silly way to avoid writing header twice (#1481)
* do cache-control in a less silly way to avoid writing header twice

* add comment back in
2023-02-13 12:58:22 +01:00
tobi
c223c7598d
[bugfix] Set appropriate cache-control when using presigned s3 links (#1480) 2023-02-12 14:42:28 +01:00
tobi
40bc03e717
[chore/performance] Update media prune logic, add extra CLI command (#1474)
* start updating media prune stuff a wee bit

* continue prune / uncache work

* more tidying + consistency stuff

* add prune CLI command

* docs

* arg
2023-02-11 12:48:38 +01:00
kim
70739d32cc
[performance] remove throttling timers (#1466)
* remove throttling timers, support setting retry-after, use retry-after in transport

* remove unused variables

* add throttling-retry-after to cmd flags

* update envparsing to include new throttling-retry-after

* update example config to include retry-after documentation

* also support retry-after formatted as date-time, ensure max backoff time

---------

Signed-off-by: kim <grufwub@gmail.com>
2023-02-10 20:16:01 +00:00
kim
6ac1dda96f
[chore] small changes missed in previous dereferencer.GetAccount() PRs (#1467)
* small formatting changes, rewrite fetchRemoteMedia to use separate funcs + use mutex lock correctly

* move url parsing before acquiring mutex locks

* use wrapped mutexes to allow safe unlocking. (previously i did a fucky and passed mutex by value...)

* remove unused code

* use consistent map keying for dereferencing headers/avatars

---------

Signed-off-by: kim <grufwub@gmail.com>
2023-02-10 20:15:23 +00:00
tobi
e5e257c259
[bugfix] Fix error on searching for account w/accountDomain by host (#1465) 2023-02-09 10:34:44 +01:00
tobi
95715f9251
[performance] Don't fetch avatar + header if uri hasn't changed (#1463) 2023-02-09 09:27:07 +01:00
tobi
27e95fd123
[chore/bugfix] Serve + throttle publickey separately from rest of ActivityPub API (#1461)
* serve publickey separately from AP, don't throttle it

* update nginx cache documentation, cache main-key too

* throttle public key, but separately from other endpoints
2023-02-08 15:10:56 +01:00
tobi
4e4da19720
[bugfix] Use SignatureCheck middleware for web profile endpoints too (#1451) 2023-02-07 14:57:09 +01:00
Sam Lade
ad6ab037e4
[bugfix] don't trash emoji in profile fields on edit (#1440) 2023-02-07 09:58:36 +01:00
kim
ac2bdbbc62
[bugfix] fix file range length calculation being off by 1 (#1448)
* small formatting change

* fix range handling new length calculation

---------

Signed-off-by: kim <grufwub@gmail.com>
2023-02-07 08:51:15 +00:00
Sam Lade
6a6647d68b
Ignore missing files when cleaning up media (#1435) 2023-02-06 14:58:54 +00:00
kim
02767bfc7d
[performance] remove local copying of file for satisfying range headers (#1421)
* remove local copying of file for satisfying range headers

Signed-off-by: kim <grufwub@gmail.com>

* remove unused import

Signed-off-by: kim <grufwub@gmail.com>

* fix range header reader limit calculation

Signed-off-by: kim <grufwub@gmail.com>

---------

Signed-off-by: kim <grufwub@gmail.com>
2023-02-06 09:50:16 +01:00
tobi
75e1b9c896
[bugfix] fix old password hash staying in cache (#1432) 2023-02-06 09:44:13 +01:00
tobi
7f3245738d
[chore] stub /api/v1/featured_tags endpoint (#1420) 2023-02-05 10:50:09 +01:00
tobi
80c26d61f7
[bugfix] Allow instance thumbnail description to be set separately from image (#1417) 2023-02-04 15:53:11 +01:00
kim
abe9447d28
fix cache startup (#1414)
Signed-off-by: kim <grufwub@gmail.com>
2023-02-03 20:16:11 +00:00
kim
33aee1b1e9
[chore] reformat GetAccount() functionality, support updating accounts based on last_fetch (#1411)
* reformat GetAccount() functionality, and add UpdateAccount() function.

* use fetched_at instead of last_webfingered_at

* catch local "not found" errors. small formatting / error string changes

* remove now unused error type

* return nil when wrapping nil error

* update expected error messages

* return correct url for foss satan webfinger

* add AP model for Some_User

* normalize local domain

* return notretrievable where appropriate

* expose NewErrNotRetrievable

* ensure webfinger for new accounts searched by uri

* update local account short circuit

* allow enrich to fail for already-known accounts

* remove unused LastWebfingeredAt

* expose test maps on mock http client

* update Update test

* reformat GetAccount() functionality, and add UpdateAccount() function.

* use fetched_at instead of last_webfingered_at

* catch local "not found" errors. small formatting / error string changes

* remove nil error checks (we shouldn't be passing nil errors to newError() initializers)

* remove mutex unlock on transport init fail (it hasn't yet been locked!)

* woops add back the error wrapping to use ErrNotRetrievable

* caches were never being started... 🙈

---------

Signed-off-by: kim <grufwub@gmail.com>
Co-authored-by: tsmethurst <tobi.smethurst@protonmail.com>
2023-02-03 20:03:05 +00:00
Autumn!
49beb17a8f
[chore] Text formatting overhaul (#1406)
* Implement goldmark debug print for hashtags and mentions

* Minify HTML in FromPlain

* Convert plaintext status parser to goldmark

* Move mention/tag/emoji finding logic into formatter

* Combine mention and hashtag boundary characters

* Normalize unicode when rendering hashtags
2023-02-03 11:58:58 +01:00
tobi
271da016b9
[bugfix] Read Bookwyrm Articles more thoroughly (#1410) 2023-02-02 16:41:02 +01:00
tobi
382512a5a6
[feature] Implement /api/v2/instance endpoint (#1409)
* interim: start adding /api/v2/instance

* finish up
2023-02-02 14:08:13 +01:00