Commit graph

19 commits

Author SHA1 Message Date
kim
2cee8f2dd8
[bugfix] fix slow accounts / statuses using emojis lookups (#2056)
* update DeleteEmoji to use faster relational tables for status / account finding

Signed-off-by: kim <grufwub@gmail.com>

* update Get{Accounts,Statuses}UsingEmoji() to also use relational tables

Signed-off-by: kim <grufwub@gmail.com>

* remove the now unneeded tags relation from newStatusQ()

Signed-off-by: kim <grufwub@gmail.com>

* fix table names

Signed-off-by: kim <grufwub@gmail.com>

* fix account and status selects using emojis

Signed-off-by: kim <grufwub@gmail.com>

---------

Signed-off-by: kim <grufwub@gmail.com>
2023-08-02 16:11:23 +02:00
kim
ed2477ebea
[performance] cache follow, follow request and block ID lists (#2027) 2023-07-31 11:25:29 +01:00
kim
5f3e095717
[performance] retry db queries on busy errors (#2025)
* catch SQLITE_BUSY errors, wrap bun.DB to use our own busy retrier, remove unnecessary db.Error type

Signed-off-by: kim <grufwub@gmail.com>

* remove dead code

Signed-off-by: kim <grufwub@gmail.com>

* remove more dead code, add missing error arguments

Signed-off-by: kim <grufwub@gmail.com>

* update sqlite to use maxOpenConns()

Signed-off-by: kim <grufwub@gmail.com>

* add uncommitted changes

Signed-off-by: kim <grufwub@gmail.com>

* use direct calls-through for the ConnIface to make sure we don't double query hook

Signed-off-by: kim <grufwub@gmail.com>

* expose underlying bun.DB better

Signed-off-by: kim <grufwub@gmail.com>

* retry on the correct busy error

Signed-off-by: kim <grufwub@gmail.com>

* use longer possible maxRetries for db retry-backoff

Signed-off-by: kim <grufwub@gmail.com>

* remove the note regarding max-open-conns only applying to postgres

Signed-off-by: kim <grufwub@gmail.com>

* improved code commenting

Signed-off-by: kim <grufwub@gmail.com>

* remove unnecessary infof call (just use info)

Signed-off-by: kim <grufwub@gmail.com>

* rename DBConn to WrappedDB to better follow sql package name conventions

Signed-off-by: kim <grufwub@gmail.com>

* update test error string checks

Signed-off-by: kim <grufwub@gmail.com>

* shush linter

Signed-off-by: kim <grufwub@gmail.com>

* update backoff logic to be more transparent

Signed-off-by: kim <grufwub@gmail.com>

---------

Signed-off-by: kim <grufwub@gmail.com>
2023-07-25 10:34:05 +02:00
kim
9eff0d46e4
[feature/performance] support uncaching remote emoji + scheduled cleanup functions (#1987) 2023-07-24 13:14:13 +01:00
kim
9a22102fa8
[bugfix/chore] oauth entropy fix + media cleanup tasks rewrite (#1853) 2023-06-22 20:46:36 +01:00
kim
6c9d8e78eb
[feature] status refetch support (#1690)
* revamp http client to not limit requests, instead use sender worker

Signed-off-by: kim <grufwub@gmail.com>

* remove separate sender worker pool, spawn 2*GOMAXPROCS batch senders each time, no need for transport cache sweeping

Signed-off-by: kim <grufwub@gmail.com>

* improve batch senders to keep popping recipients until remote URL found

Signed-off-by: kim <grufwub@gmail.com>

* fix recipient looping issue

Signed-off-by: kim <grufwub@gmail.com>

* move request id ctx key to gtscontext, finish filling out more code comments, add basic support for not logging client IP

Signed-off-by: kim <grufwub@gmail.com>

* first draft of status refetching logic

Signed-off-by: kim <grufwub@gmail.com>

* fix testrig to use new federation alloc func signature

Signed-off-by: kim <grufwub@gmail.com>

* fix log format directive

Signed-off-by: kim <grufwub@gmail.com>

* add status fetched_at migration

Signed-off-by: kim <grufwub@gmail.com>

* remove unused / unchecked for error types

Signed-off-by: kim <grufwub@gmail.com>

* add back the used type...

Signed-off-by: kim <grufwub@gmail.com>

* add separate internal getStatus() function for derefThread() that doesn't recurse

Signed-off-by: kim <grufwub@gmail.com>

* improved mention and media attachment error handling

Signed-off-by: kim <grufwub@gmail.com>

* fix log and error format directives

Signed-off-by: kim <grufwub@gmail.com>

* update account deref to match status deref changes

Signed-off-by: kim <grufwub@gmail.com>

* very small code formatting change to make things clearer

Signed-off-by: kim <grufwub@gmail.com>

* add more code comments

Signed-off-by: kim <grufwub@gmail.com>

* improved code commenting

Signed-off-by: kim <grufwub@gmail.com>

* only check for required further derefs if needed

Signed-off-by: kim <grufwub@gmail.com>

* improved cache invalidation

Signed-off-by: kim <grufwub@gmail.com>

* tweak cache restarting to use a (very small) backoff

Signed-off-by: kim <grufwub@gmail.com>

* small readability changes and fixes

Signed-off-by: kim <grufwub@gmail.com>

* fix account sync issues

Signed-off-by: kim <grufwub@gmail.com>

* fix merge conflicts + update account enrichment to accept already-passed accountable

Signed-off-by: kim <grufwub@gmail.com>

* remove secondary function declaration

Signed-off-by: kim <grufwub@gmail.com>

* normalise dereferencer get status / account behaviour, fix remaining tests

Signed-off-by: kim <grufwub@gmail.com>

* fix remaining rebase conflicts, finish commenting code

Signed-off-by: kim <grufwub@gmail.com>

* appease the linter

Signed-off-by: kim <grufwub@gmail.com>

* add source file header

Signed-off-by: kim <grufwub@gmail.com>

* update to use TIMESTAMPTZ column type instead of just TIMESTAMP

Signed-off-by: kim <grufwub@gmail.com>

* don't pass in 'updated_at' to UpdateEmoji()

Signed-off-by: kim <grufwub@gmail.com>

* use new ap.Resolve{Account,Status}able() functions

Signed-off-by: kim <grufwub@gmail.com>

* remove the somewhat confusing rescoping of the same variable names

Signed-off-by: kim <grufwub@gmail.com>

* update migration file name, improved database delete error returns

Signed-off-by: kim <grufwub@gmail.com>

* formatting

Signed-off-by: kim <grufwub@gmail.com>

* improved multi-delete database functions to minimise DB calls

Signed-off-by: kim <grufwub@gmail.com>

* remove unused type

Signed-off-by: kim <grufwub@gmail.com>

* fix delete statements

Signed-off-by: kim <grufwub@gmail.com>

---------

Signed-off-by: kim <grufwub@gmail.com>
2023-05-12 11:15:54 +02:00
Daenney
5e2bf0bdca
[chore] Improve copyright header handling (#1608)
* [chore] Remove years from all license headers

Years or year ranges aren't required in license headers. Many projects
have removed them in recent years and it avoids a bit of yearly toil.

In many cases our copyright claim was also a bit dodgy since we added
the 2021-2023 header to files created after 2021 but you can't claim
copyright into the past that way.

* [chore] Add license header check

This ensures a license header is always added to any new file. This
avoids maintainers/reviewers needing to remember to check for and ask
for it in case a contribution doesn't include it.

* [chore] Add missing license headers

* [chore] Further updates to license header

* Use the more common // indentend comment format
* Remove the hack we had for the linter now that we use the // format
* Add SPDX license identifier
2023-03-12 16:00:57 +01:00
Daenney
68e6d08c76
[feature] Add a request ID and include it in logs (#1476)
This adds a lightweight form of tracing to GTS. Each incoming request is
assigned a Request ID which we then pass on and log in all our log
lines. Any function that gets called downstream from an HTTP handler
should now emit a requestID=value pair whenever it logs something.

Co-authored-by: kim <grufwub@gmail.com>
2023-02-17 12:02:29 +01:00
tobi
0dbe6c514f
[chore] Update/add license headers for 2023 (#1304) 2023-01-05 12:43:00 +01:00
kim
e58d2d8122
[chore] move caches to a separate State{} structure (#1078)
* move caches to a separate State{} structure

Signed-off-by: kim <grufwub@gmail.com>

* fix call to log.Panic not using formatted call

Signed-off-by: kim <grufwub@gmail.com>

* move caches to use interfaces, to make switchouts easier in future

Signed-off-by: kim <grufwub@gmail.com>

* fix rebase issue

Signed-off-by: kim <grufwub@gmail.com>

* improve code comment

Signed-off-by: kim <grufwub@gmail.com>

* fix further issues after rebase

Signed-off-by: kim <grufwub@gmail.com>

* heh

Signed-off-by: kim <grufwub@gmail.com>

* add missing license text

Signed-off-by: kim <grufwub@gmail.com>

Signed-off-by: kim <grufwub@gmail.com>
2022-12-08 18:35:14 +01:00
tobi
34716d7d7d
[bugfix] Add missing continues in emoji get funcs (#1200) 2022-12-02 16:41:05 +01:00
kim
8598dea98b
[chore] update database caching library (#1040)
* convert most of the caches to use result.Cache{}

* add caching of emojis

* fix issues causing failing tests

* update go-cache/v2 instances with v3

* fix getnotification

* add a note about the left-in StatusCreate comment

* update EmojiCategory db access to use new result.Cache{}

* fix possible panic in getstatusparents

* further proof that kim is not stinky
2022-11-15 18:45:15 +00:00
tobi
4cd00d546c
[feature] Allow newly uploaded emojis to be placed in categories (#939)
* [feature] Add emoji categories GET
Serialize emojis in appropriate categories; make it possible to get categories via the admin API

* [feature] Create (or use existing) category for new emoji uploads

* fix lint issue

* update misleading line in swagger docs
2022-11-14 22:47:27 +00:00
tobi
f7416d6e94
[feature] Add emoji DELETE handler at /api/v1/admin/custom_emojis (#913)
* add emoji DELETE handler

* no need to process error (thanks kim)

* don't double check if user is admin

* add missing security annotation
2022-10-14 17:30:04 +02:00
tobi
70d65b683f
[feature] Refetch emojis when they change on remote instances (#905)
* select emoji using image_static_url

* use updated on AP emojis

* allow refetch of updated emojis

* cheeky workaround for test

* clean up old files for refreshed emoji

* check error for originalPostData

* shorten GetEmojiByStaticImageURL

* delete kirby (sorry nintendo)
2022-10-13 15:16:24 +02:00
tobi
eb85ef7325
[feature] Add /api/v1/admin/custom_emojis endpoint (#902)
* add admin emojis get path + model + docs

* stub admin emojis get processor function

* add id + disabled fields to admin emoji

* add emoji -> api admin emoji converter

* tidy up a bit

* add GetEmojis function

* finish up get emojis function

* order by shortcodedomain

* ASC

* tidy up + explain

* update to allow paging

* make admin emojis pageable

* fix mixed case paging

* normalize emoji queries a bit better

* test emoji get paging

* make limit optional

* fix incorrect path in media cleanup tests

* i have bad coder syndrome

* don't trimspace

* rename -> GetUseableEmojis

* wrap emoji query in subquery
avoid selecting more than we need

* fix a bit of sillyness teehee

* fix subquery postgres woes
2022-10-12 15:01:42 +02:00
tobi
aa07750bdb
[chore] Standardize database queries, use bun.Ident() properly (#886)
* use bun.Ident for user queries

* use bun.Ident for account queries

* use bun.Ident for media queries

* add DeleteAccount func

* remove CaseInsensitive in Where+use Ident ipv Safe

* update admin db

* update domain, use ident

* update emoji, use ident

* update instance queries, use bun.Ident

* fix media

* update mentions, use bun ident

* update relationship + tests

* use tableexpr

* add test follows to bun db test suite

* update notifications

* updatebyprimarykey => updatebyid

* fix session

* prefer explicit ID to pk

* fix little fucky wucky

* remove workaround

* use proper db func for attachment selection

* update status db

* add m2m entries in test rig

* fix up timeline

* go fmt

* fix status put issue

* update GetAccountStatuses
2022-10-08 13:50:48 +02:00
tobi
a872ddebe6
[feature] Custom emoji updates (serve emoji via s2s api, tune db models) (#805)
* migrate emojis

* add get emoji to s2s (federation) API

* add new emoji db + cache functions

* add shortcodeDomain lookup for emojis

* check existing emojis w/cache, not w/constraints

* go fmt

* add putEmoji func

* use new db emoji funcs instead of where

* remove emojistringstotags func

* add unique constraint back in

* fix up broken migration

* update index
2022-09-06 12:42:55 +02:00
Adelie Paull
caa0cde0e0
[feature] implement custom_emojis endpoint (#563)
* implement custom_emojis api endpoint

* add tests for getting custom emoji out of the database and converting to api emoji

* change sort direction of emoji query

* change logging level and initialize array with known length as per kim's suggestions

* add continue to lessen risk of making a malformed struct during conversion from db to api emojis
2022-05-20 10:34:36 +02:00