Commit graph

998 commits

Author SHA1 Message Date
kim
8ebb7775a3
[feature] request blocking by http headers (#2409) 2023-12-18 14:18:25 +00:00
Sam Lade
07bd848028
fix poll total vote double count (#2464) 2023-12-16 19:12:25 +00:00
Sam Lade
285d55dda8
[feature] Push status edit messages into open streams (#2418)
* push status edit messages into open streams

* fix a few comments

* test++

* commented out code? moi?
2023-12-16 12:55:49 +01:00
Daenney
fbe4e60232
[feature] Run ANALYZE after migrations on SQLite (#2428)
* [feature] Run ANALYZE after migrations on SQLite

This ensures that at the end of migrations, we run ANALYZE if we're
using SQLite. This should be relatively quick and guarantees that the
table and index statistics have been updated. This helps to ensure the
query planner makes better choices when it comes to picking which
indexes are used when running queries.

* [chore] use ExecContext

Uses ExecContext so we pass the context through, this is helpful for
anyone running with tracing enabled
2023-12-16 12:54:53 +01:00
kim
d56a8d095e
[performance] simpler throttling logic (#2407)
* reduce complexity of throttling logic to use 1 queue and an atomic int

* use atomic add instead of CAS, add throttling test
2023-12-16 12:53:42 +01:00
kim
f4fcffc8b5
[bugfix] use a much shorter refresh limit for statuses with polls (#2453)
* specifically use a much shorter refresh limit for statuses with polls

* allow specifying whether status must be upToDate in calls to Get(Visible)?TargetStatusBy_(), limit force refresh to 5 minute cooldown

* remove the PollID check from statusUpToDate()

* remove unnecessary force flag checks

* remove unused field

* check refresh status error

* use argument name 'refresh' instead of 'upToDate' to better fit with the codebase

* add statuses_poll_id_idx

* remove the definitely-not copy-pasted comment i accidentally typed out in full

* only synchronously refresh if the refresh flag is provided, otherwise do async

* fix wrong force value being provided for async

---------

Co-authored-by: tobi <tobi.smethurst@protonmail.com>
2023-12-15 15:24:39 +01:00
tobi
d0bb8f0973
[bugfix] Let templates deref pointers, as a treat (#2448) 2023-12-12 14:44:54 +00:00
kim
ac48192562
[bugfix] poll vote count fixes (#2444)
* don't drop all vote counts if hideCounts is set, refactors poll option extraction slightly

* omit voters_count when not set

* make voters_count a ptr to ensure it is omit unless definitely needed

* handle case of expires_at, voters_count and option.votes_count being nilable

* faster isNil check

* remove omitempty tags since mastodon API marks things as nullable but still sets them in outgoing json
2023-12-12 13:47:07 +00:00
kim
2191c7dee5
[bugfix] ensure the 'Closing' flag doesn't get cached (#2443) 2023-12-11 12:05:54 +00:00
tobi
3f070a442a
[bugfix] Narrow search scope for accounts starting with '@'; don't LOWER SQLite text searches (#2435) 2023-12-10 14:15:41 +01:00
tobi
d60edf7ec6
[bugfix] Ensure pre renders as expected, fix orderedCollectionPage (#2434) 2023-12-10 12:36:00 +01:00
tobi
cc91ea057d
[bugfix] Fix web media not showing as sensitive (#2433)
* [bugfix] Fix web media not showing as sensitive

* test

* go fmt
2023-12-09 16:54:38 +01:00
tobi
c6d6fec17d
[bugfix] Fix wrong notification type sent for poll end (#2429) 2023-12-08 23:33:42 +01:00
tobi
4779aecdec
[chore] Run ANALYZE for SQLite after latest migrations (#2427) 2023-12-08 16:59:20 +01:00
Daenney
455064fec7
[bug] Fix an import statement in the gen template (#2426)
The package was renamed from langs to language.
2023-12-08 13:07:34 +01:00
tobi
dacfd413dc
[chore/frontend] Refactor status templates slightly, put polls behind CWs if present (#2419)
* [chore/frontend] Reorder templates, allow polls to be inside content warnings

* show when multiple-choice
2023-12-05 12:43:07 +01:00
tobi
0e2c342191
[bugfix/chore] Announce reliability updates (#2405)
* [bugfix/chore] `Announce` updates

* test update

* fix tests

* TestParseAnnounce

* update comments

* don't lock/unlock, change function signature

* naming stuff

* don't check domain block twice

* UnwrapIfBoost

* beep boop
2023-12-01 15:27:15 +01:00
kim
d1cac53cbb
[bugfix] in fedi API CreateStatus(), handle case of data-race and return early (#2403) 2023-12-01 11:53:53 +01:00
kim
eb170003b8
[bugfix] return 400 Bad Request on more cases of malformed AS data (#2399) 2023-11-30 16:22:34 +00:00
kim
5fd2e427bb
[bugfix] always go through status parent dereferencing on isNew, even on data-race (#2402)
* no need to deref status author account, will already be deref'd during previous getStatusByAP{IRI,Model}()

* don't unset the isNew flag on dereference data race

* improved code comment
2023-11-30 12:32:45 +01:00
tobi
f9ba0df726
[chore] Re-add indexes, rename account actions indexes (#2401) 2023-11-30 11:22:35 +00:00
tobi
0108463e7b
[bugfix] Update exif-terminator (fix png issue) (#2391)
* [bugfix] Update exif-terminator (fix png issue)

* bump exif terminator

* fix tests
2023-11-30 10:50:28 +01:00
tobi
6abe91ceb2
[performance/postgres] Rename constraints, remove duplicate indexes (#2392)
* [performance/postgres] Rename constraints, remove duplicate indexes

* remove duplicate indexes from sqlite as well
2023-11-29 11:28:23 +01:00
Jadeiin
2cc264584e
[bugfix] Correctly handle range > content-length (#2395) 2023-11-29 11:25:52 +01:00
dependabot[bot]
2eb8b8eeb4
[chore]: Bump github.com/gorilla/websocket from 1.5.0 to 1.5.1 (#2335)
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kim <grufwub@gmail.com>
2023-11-28 11:05:07 +00:00
tobi
33ee61575f
[bugfix] Don't copy ptr fields in caches (#2386) 2023-11-27 15:39:44 +00:00
tobi
5eddef6c9b
[feature] Add /api/v1/admin/debug/apurl endpoint (#2359) 2023-11-27 14:02:52 +00:00
kim
74700cc803
[performance] http response encoding / writing improvements (#2374) 2023-11-27 14:00:57 +00:00
tobi
e4e0a5e3f6
[bugfix] Add Actor to outgoing poll vote Create; other fixes (#2384) 2023-11-27 13:14:28 +00:00
tobi
2b9cf56f56
[chore/bugfix] Fix double gzip on prometheus endpoint (#2383)
* [chore] Move "/metrics" into separate API module

* use our own gzip middleware for prom
2023-11-23 19:10:51 +01:00
tobi
71e0aaee3c
[bugfix/chore] id poll options properly (#2379)
* aria-labels for polls, id poll options properly

* my bad, aria labels are for interactive elements
2023-11-22 16:27:32 +01:00
tobi
8c2d94c168
[feature] Poll web view (#2377)
* [feature] Render polls nicely on the web view

* use figure for poll, other small tweaks

* reverse share + count (lines up better)

* poll options list entries

* fix up some remaining things
2023-11-22 12:17:42 +01:00
tobi
cfefbc08d8
[feature] Federate status language in and out (#2366)
* [feature] Federate status language in + out

* go fmt

* tests, little fix

* improve comments

* unnest a bit

* avoid unnecessary nil check

* use more descriptive variable for contentMap

* prefer instance languages when selecting from contentMap

* update docs to reflect lang selection

* rename rdfLangString -> rdfLangs

* update comments to mention Pollable

* iter through slice instead of map
2023-11-21 15:13:30 +01:00
kim
42d8011ff4
[chore/security] refactor AuthenticateFederatedRequest() to handle account deref + suspension checks (#2371)
* refactor AuthenticateFederatedRequest() to handle account suspension + fetching of owner

* small fixups

* small changes

* revert to 'IsEitherBlocked' instead of just 'IsBlocked" :grimace:

* update code comment to indicate that AuthenticateFederatedRequest() will handle account + instance dereferencing
2023-11-21 11:35:30 +01:00
Tsuribori
1ba3e14b36
[feature] Initial Prometheus metrics implementation (#2334)
* feat: Initial OTEL metrics

* docs: add metrics documentation

* fix: metrics endpoint conditional check

* feat: metrics endpoint basic auth

* fix: make metrics-auth-enabled default false

* fix: go fmt helpers.gen.go

* fix: add metric-related env vars to envparsing.sh

* fix: metrics docs

* fix: metrics related stuff in envparsing.sh

* fix: metrics docs

* chore: metrics docs wording

* fix: metrics stuff in envparsing?

* bump otel versions

---------

Co-authored-by: Tsuribori <user@acertaindebian>
Co-authored-by: Tsuribori <none@example.org>
Co-authored-by: tsmethurst <tobi.smethurst@protonmail.com>
2023-11-20 16:43:55 +01:00
kim
16275853eb
[bugfix] self-referencing collection pages for status replies (#2364) 2023-11-20 12:22:28 +00:00
tobi
9b6d972c74
[chore/whoops] re-add missing languages flag (#2365) 2023-11-17 12:29:03 +01:00
tobi
fc02d3c6f7
[feature] Set/show instance language(s); show post language on frontend (#2362)
* update go text, include text/display

* [feature] Set instance langs, show post lang on frontend

* go fmt

* WebGet

* set language for whole article, don't use FA icon

* mention instance languages + other optional config vars

* little tweak

* put languages in config properly

* warn log language parse

* change some naming around

* tidy up validate a bit

* lint

* rename LanguageTmpl in template
2023-11-17 11:35:28 +01:00
tobi
4ee436e98a
[bugfix] process account delete side effects in serial, not in parallel (#2360)
* [bugfix] process account delete side effects in serial, not in parallel

* StartWorkers / StartNoopWorkers for tests

* undo testrig trace logging

* log errors instead of immediately returning
2023-11-14 15:57:25 +01:00
tobi
0b99f14d64
[bugfix] Update poll delete/update db queries (#2361) 2023-11-14 12:43:27 +00:00
tobi
8d0c017cf2
[feature/performance] Wrap incoming HTTP requests in timeout handler (#2353)
* deinterface router, start messing about with deadlines

* weeeee

* thanks linter (thinter)

* write Connection: close when timing out requests

* update wording

* don't replace req

* don't bother with fancy Cause functions (I'll use them one day...)
2023-11-13 19:48:51 +01:00
tobi
7753f42132
[chore] update otel -> v1.20.0 (#2358) 2023-11-13 11:08:02 +01:00
kim
deaea100c3
[bugfix] support endless polls, and misskey's' method of inferring expiry in closed polls (#2349) 2023-11-11 10:15:04 +00:00
tobi
ba9d6b467a
[feature] Media attachment placeholders (#2331)
* [feature] Use placeholders for unknown media types

* fix read of underreported small files

* switch to reduce nesting

* simplify cleanup
2023-11-10 19:29:26 +01:00
tobi
c7ecab9e6f
[chore/bugfix/horror] Allow expires_in and poll choices to be parsed from strings (#2346) 2023-11-10 16:42:48 +00:00
tobi
7ce3a1e6f3
[bugfix] Don't try to update suspended accounts (#2348)
* [bugfix] Don't try to update suspended accounts

* bail early if requesting account suspended
2023-11-10 17:16:58 +01:00
tobi
42a19cf390
[bugfix/docs] Poll api fixups + swagger docs (#2345) 2023-11-09 12:06:37 +00:00
kim
b1c65ed9ac
[bugfix] actually decrement votes during poll vote delete ... (#2344) 2023-11-09 09:08:46 +01:00
kim
34d0879c16
[bugfix] fix poll vote count responses on client and fedi API vote creation (#2343)
* increment poll votes *before* enqueuing vote to client API worker

* increment vote counts before federating status update after vote in local poll

* improved vote count calculation during backend -> frontend model conversion
2023-11-08 23:37:35 +01:00
kim
e9e5dc5a40
[feature] add support for polls + receiving federated status edits (#2330) 2023-11-08 14:32:17 +00:00
kim
41435a6c4e
[feature] support canceling scheduled tasks, some federation API performance improvements (#2329) 2023-11-04 20:21:20 +00:00
Sam Lade
145dc515a2
Remove account_suspended_at_idx to resolve slow query issues (#2310) 2023-10-31 13:43:33 +00:00
kim
ce71a5a790
[feature] add per-uri dereferencer locks (#2291) 2023-10-31 11:12:22 +00:00
tobi
51d0a0bba5
[bugfix] Relax Mention parsing, allowing either href or name (#2320) 2023-10-31 11:05:17 +00:00
tobi
dd4b0241ea
[bugfix] Allow blocked accounts to show in precise search (#2321) 2023-10-30 19:01:00 +01:00
tobi
4dc0547dc0
[feature] Customizable media cleaner schedule (#2304) 2023-10-30 17:35:11 +00:00
tobi
0b978f2c56
[bugfix] Extract description as summary first, fall back to name (#2303) 2023-10-26 11:59:10 +02:00
kim
ece2e795e0
[feature] attach any request errors if found, only set level=ERROR if code >= 500 (#2300) 2023-10-25 16:11:40 +01:00
tobi
c7b6cd7770
[feature] Status thread mute/unmute functionality (#2278)
* add db models + functions for keeping track of threads

* give em the old linty testy

* create, remove, check mutes

* swagger

* testerino

* test mute/unmute via api

* add info log about new index creation

* thread + allow muting of any remote statuses that mention a local account

* IsStatusThreadMutedBy -> IsThreadMutedByAccount

* use common processing functions in status processor

* set = NULL

* favee!

* get rekt darlings, darlings get rekt

* testrig please, have mercy muy liege
2023-10-25 15:04:53 +01:00
KEINOS
27f4659139
[bugfix] allow store smaller PNG image than 261 bytes (#2263) (#2298)
* chore: add test of golden cases before fix of #2263

* chore: add test case to reproduce error of #2263

* [bugfix] allow store smaller PNG image than 261 bytes (#2263)
2023-10-25 14:58:45 +02:00
tobi
cc313f58cb
[bugfix] serialize instance terms via API (#2293) 2023-10-24 14:23:22 +02:00
tobi
013a17f5e8
[chore] bump go version -> 1.21.x (#2287)
* [chore] bump go version -> 1.21.x

* versions

* lint
2023-10-23 13:07:31 +01:00
kim
69ba9a79a1
[chore] de-interface{} the federator and dereferencer structs (#2285)
* de-interface{} the federator and dereferencer structs

* fix broken type signatures
2023-10-23 10:58:13 +01:00
tobi
9114c5ca1b
[bugfix/frontend] fix typo and other oddness in patchRemoteEmojis (#2281)
* fix emoji test model

* found the bug!

* remove unused 'current' import

* comment useChecklistReducer

* wah

* lint

* fix cleaner tests
2023-10-21 17:23:05 +02:00
tobi
805c67b759
[bugfix] add missing "local" param in public timeline link headers (#2277) 2023-10-18 12:32:40 +02:00
tobi
77c5d74113
[bugfix] Postgres error wrapping fix (#2276) 2023-10-18 12:17:26 +02:00
tobi
637f188ebe
[feature] Allow import/export/creation of domain allows via admin panel (#2264)
* it's happening!

* aaa

* fix silly whoopsie

* it's working pa! it's working ma!

* model report parameters

* shuffle some more stuff around

* getting there

* oo hoo

* finish tidying up for now

* aaa

* fix use form submit errors

* peepee poo poo

* aaaaa

* ffff

* they see me typin', they hatin'

* boop

* aaa

* oooo

* typing typing tappa tappa

* almost done typing

* weee

* alright

* push it push it real good doo doo doo doo doo doo

* thingy no worky

* almost done

* mutation modifers not quite right

* hmm

* it works

* view blocks + allows nicely

* it works!

* typia install

* the old linterino

* linter plz
2023-10-17 12:46:06 +02:00
kim
6e508830e1
updates markdown parsing to reduce allocations in the same way as the plain text formatter (#2252) 2023-10-05 13:22:40 +01:00
恐咖兵糖
e0f0d320f6
[docs] statuses-query-pinned #2250 (#2251) 2023-10-04 14:19:24 +02:00
kim
c6e00afc7c
[feature] tentatively start adding polls support (#2249) 2023-10-04 13:09:42 +01:00
kim
297b6eeaaa
[chore] internal/ap: add pollable AS types, code reformatting, general niceties (#2248) 2023-10-03 14:59:30 +01:00
Daenney
0cce2c0838
[feature] Block a bunch of "AI" crawlers (#2239)
* [feature] Block Google Bard/AI crawlers

* [feature] Block the other OpenAI crawler

* [feature] Block Common Crawl crawler

This is used in research, but also gleefully advertises itself as the
training source used in all LLMs and GPT-3.

Fixes: #2240

* [feature] Block Omgilikebot

Used by some shady big web data engine company.

* [feature] Block Meta's language model crawler

* [feature] Block well-known.dev crawler
2023-09-30 20:44:57 +01:00
tobi
2b6b9cdf83
[bugfix] Fix paging for empty items (#2236)
* use minID properly for public timeline

* return paged response properly even when 0 items

* use gtserror

* page more consistently (for now)

* test

* aaa
2023-09-29 15:31:10 +02:00
tobi
536d9e482d
[chore/bugfix] Deinterface text.Formatter, allow underscores in hashtags (#2233) 2023-09-29 10:39:56 +02:00
tobi
b6b8f82c87
[bugfix] Move follow.show_reblogs check further up to avoid showing unwanted reblogs in home timeline (#2234) 2023-09-29 10:39:35 +02:00
Daenney
2a9927dfdc
[chore] Enable S256 code challenge (#2224)
For some reason we only had plain enabled. This adds S256.
2023-09-28 11:21:19 +02:00
kim
9f9fcf743d
[bugfix] support both CollectionPage AND OrderedCollectionPage in status replies (#2220) 2023-09-23 18:28:12 +01:00
kim
8f67dd583d
[chore] deinterface the typeutils.Converter and update to use state structure (#2217)
* update typeconverter to use state structure

* deinterface the typeutils.TypeConverter -> typeutils.Converter

* finish copying over old type converter code comments

* fix cherry-pick merge issues, fix tests pointing to old typeutils interface type still
2023-09-23 18:44:11 +02:00
K
1b848aa1e5
[chore] fix typo in slice.go (#2219) 2023-09-23 18:42:34 +02:00
tobi
183eaa5b29
[feature] Implement explicit domain allows + allowlist federation mode (#2200)
* love like winter! wohoah, wohoah

* domain allow side effects

* tests! logging! unallow!

* document federation modes

* linty linterson

* test

* further adventures in documentation

* finish up domain block documentation (i think)

* change wording a wee little bit

* docs, example

* consolidate shared domainPermission code

* call mode once

* fetch federation mode within domain blocked func

* read domain perm import in streaming manner

* don't use pointer to slice for domain perms

* don't bother copying blocks + allows before deleting

* admonish!

* change wording just a scooch

* update docs
2023-09-21 12:12:04 +02:00
kim
fc11deeb83
[feature] add paging to AP following / followers endpoints (#2198) 2023-09-20 16:49:46 +01:00
kim
ba0a31d224
[bugfix] fix flakey paging test (#2210) 2023-09-18 13:45:10 +01:00
kim
7293d6029b
[feature] add paging to account follows, followers and follow requests endpoints (#2186) 2023-09-12 14:00:35 +01:00
tobi
4b594516ec
[feature] Allow admins to expire remote public keys; refetch expired keys on demand (#2183) 2023-09-12 10:43:12 +01:00
Daenney
2cac5a4613
[feature] Support Actor URIs for webfinger queries (#2187)
* [feature] Support Actor URIs for webfinger queries

It's now possible to pass an Actor URI as the resource to query for when
doing a webfinger query. The code now extracts the username and domain
from the URI. The URI needs to be fully qualified, including having a
scheme of http or https to be recognised as such.

The acct scheme is handled as we used to, including dealing with an
erroneous leading @ on the username. We retain the ability to handle
resources without a scheme by parsing them again with the acct scheme if
the original parse failed. This can happen due to parsing ambiguities
when dealing with a string like user@domain.tld:port.

* [bugfix] Remove debugging changes

* [chore] Make TestExtractNamestring table-driven

* [chore] Unnest Trim and Split for readability
2023-09-11 18:38:31 +02:00
kim
b093947d84
[chore] much improved paging package (#2182) 2023-09-07 15:58:37 +01:00
Daenney
14ef098099
[feature] Support OTLP HTTP, drop Jaeger (#2184)
* [feature] Add http trace exporter, drop Jaeger

Jaeger supports ingesting traces using the OpenTelemetry gRPC or HTTP
methods. The Jaeger project has deprecated the old jaeger transport.

* Add support for submitting traces over HTTP
* Drop support for the old Jaeger protocol
* Upgrade the trace libraries to v1.17

Fixes: #2176
Fixes: #2179
2023-09-07 13:20:37 +02:00
kim
916c6d07ba
[bugfix] fix checks for deref the same status descendants / ascendants (#2181) 2023-09-05 11:22:02 +01:00
Daenney
5f10885dde
[feature] Don't trace non-existing routes (#2172)
c.FullPath() is the empty string if a request doesn't match any route on
our mux. In those cases, there's no value in emitting a trace. The trace
will be empty, containing no other information beyond the fact that we
didn't match a route. Since Gin breaks off the processing early we don't
need to trace this request as it won't do anything and consumes no
further resources.

The 404 will still be emitted by our logs and will be visible from a
reverse proxy too.
2023-09-04 17:15:14 +02:00
tobi
3ed1ca68e5
[feature] Store admin actions in the db, prevent conflicting actions (#2167) 2023-09-04 14:55:17 +01:00
kim
4eb77ff5d7
[bugfix] move SQLite pragmas into connection string (#2171)
* move SQLite pragmas into connection string

Signed-off-by: kim <grufwub@gmail.com>

* use url.Values type for SQLite connection preferences

Signed-off-by: kim <grufwub@gmail.com>

* set SQLite URI prefs properly using _pragma query key

Signed-off-by: kim <grufwub@gmail.com>

* add notes on SQLite connection preferences

Signed-off-by: kim <grufwub@gmail.com>

* fix typo

Signed-off-by: kim <grufwub@gmail.com>

* add one extra line regarding connection pooling

Signed-off-by: kim <grufwub@gmail.com>

---------

Signed-off-by: kim <grufwub@gmail.com>
2023-09-01 15:13:33 +02:00
kim
1ee99fc165
[bugfix] wrap bun.Tx to add our own error processing (#2169)
* wrap bun.Tx to add our own error processing

Signed-off-by: kim <grufwub@gmail.com>

* add compile-time check for updateRowError() compatibility with sql.Row, fix wrapTx() not being used properly

Signed-off-by: kim <grufwub@gmail.com>

---------

Signed-off-by: kim <grufwub@gmail.com>
2023-08-31 11:46:15 +02:00
rdelaage
7b48437f17
[feature] list commands for both attachment and emojis (#2121)
* [feature] list commands for both attachment and emojis

* use fewer commands, provide `local-only` and `remote-only` as filters

* envparsing

---------

Co-authored-by: Romain de Laage <romain.delaage@rdelaage.ovh>
Co-authored-by: tsmethurst <tobi.smethurst@protonmail.com>
2023-08-23 18:01:16 +02:00
tobi
8f38dc2e7f
[feature] Add rate limit exceptions option, use ISO8601 for rate limit reset (#2151)
* start updating rate limiting, add exceptions

* tests, comments, tidying up

* add rate limiting exceptions to example config

* envparsing

* nolint

* apply kimbediff

* add examples
2023-08-23 14:32:27 +02:00
tobi
94d16631bc
[performance] Rework home timeline query to use cache more (#2148) 2023-08-22 15:41:51 +02:00
Daenney
4ae16bce8c
[feature] Make log format configurable (#2130)
* [feature] Don't emit timestamp in log lines

When running gotosocial with a service manager like systemd, or a
container runtime, the associated log driver usually emits timestamps
itself. In those cases, having the extra timestamp from our own log
lines ends up being a bit noisy and when centrally ingesting logs is
duplicate information.

This introduces a configuration flag that allows disabling emitting the
timestamp. It's only wired up for "daemonised" processes, meaning server
and testrig.

* [chore] Add docs for log-timestamp

* [feature] Simplify timestamp handling

Co-Authored-By: kim <89579420+NyaaaWhatsUpDoc@users.noreply.github.com>

* [chore] Less escaped double-quotes

* [chore] Fix help string

---------

Co-authored-by: kim <89579420+NyaaaWhatsUpDoc@users.noreply.github.com>
2023-08-21 19:07:55 +01:00
tobi
638f023a1c
[performance] Tweak media attachment cleanup; replace stale index (#2143) 2023-08-21 14:03:04 +01:00
tobi
1e2db7a32f
[feature/bugfix] Probe S3 storage for CSP uri, add config flag for extra URIs (#2134)
* [feature/bugfix] Probe S3 storage for CSP uri, add config flag for extra URIs

* env parsing tests, my coy mistress
2023-08-20 13:35:55 +02:00
f0x52
92de8fb396
[feature] Instance rules (#2125)
* init instance rules database model, admin api

* expose instance rules in public instance api

* public /api/v1/instance/rules route

* GET ruleById

* createRule route

* createRule auth check

* updateRule

* deleteRule

* list rules on about page

* ruleGet auth

* add about page ids for anchors

* process and store adding violated rules to reports

* admin api models for instance rules

* instance rule edit frontend

* change rule inputs to textareas

* database fixes after rebase (#2124)

* remove unused imports

* fix db migration column name

* fix tests

* fix more tests

* fix postgres error with wrongly used Ident

* add some tests, fiddle with rule model a bit, fix postgres migration

* swagger docs

---------

Co-authored-by: tsmethurst <tobi.smethurst@protonmail.com>
2023-08-19 14:33:15 +02:00
kim
d5d6ad406f
[bugfix] fix double firing bun.DB query hooks (#2124)
* improve bun.DB wrapping readability + comments, fix double-firing query hooks

* fix incorrect code comment placement

* fix linter issues

* Update internal/db/basic.go

* do as the linter commmands ...

---------

Signed-off-by: kim <grufwub@gmail.com>
Co-authored-by: Daenney <daenney@users.noreply.github.com>
2023-08-17 17:26:21 +01:00
kim
e9c3663cce
[chore] ensure worker contexts have request ID (#2120) 2023-08-15 17:01:01 +01:00
kim
815b5291e0
[bugfix] fix inconsistent calculated cache sizes (#2115)
* use calculated exampleTime instead of `time.Now()` to ensure no locale data, retweak cache ratios

* update envparsing test

* update default cache memory to 100MiB

* fix envparsing with latest cache target default

---------

Signed-off-by: kim <grufwub@gmail.com>
2023-08-14 17:08:19 +01:00
f0x52
912a104aed
[fix] Update CSP header for blob images (upload preview) and dev livereload (#2109)
* update CSP header for blob images (upload preview) and dev livereload websocket

* update csp for s3, update csp tests
2023-08-14 12:30:09 +02:00
Daenney
5e368d3089
[bugfix] CSP policy fixes for S3/object storage (#2104)
* [bugfix] CSP policy fixes for S3 in non-proxied mode

* It should be img-src
* In both img-src and media-src we still need to include 'self'
2023-08-12 12:21:48 +02:00
tobi
b7274545e0
[bugfix] Add s3 endpoint as image-src and media-src for CSP (#2103)
* [bugfix] Add s3 endpoint as image-src and media-src for CSP

* use https if secure

* reorder comment
2023-08-11 17:49:17 +02:00
tobi
a1768a83e0
[bugfix] Suppress 'errNoEntries' warnings from InboxForwarding function call (#2102) 2023-08-11 14:17:36 +01:00
tobi
dc99e9e10b
[bugfix] Fix using wrong key for clientID during oauth callback (#2101) 2023-08-11 14:58:47 +02:00
tobi
dc96562b40
[bugfix] Use custom bluemonday policy to disallow inline img tags (#2100) 2023-08-11 14:40:11 +02:00
Daenney
3aedd937c3
[feature] Set Content-Security-Policy header (#2095)
This adds the CSP header with a policy of only loading from the same
domain. We don't make use of external media, CSS, JS, fonts, so we don't
ever need external data loaded in our context.

When building a DEBUG build, the policy gets extended to include
localhost:*, i.e localhost on any port. This keeps the live-reloading
flow for JS development working. localhost and 127.0.0.1 are considered
to be the same so mixing and matching those doesn't result in a CSP
violation.
2023-08-11 13:20:56 +02:00
tobi
a26af1310f
[bugfix] Populate followReq before accessing targetaccount pointer (#2099) 2023-08-11 11:05:49 +02:00
tobi
5588d4e88e
[bugfix] Use length in runes when trimming for RSS (#2094) 2023-08-10 18:26:56 +02:00
tobi
992c7ce4c2
[chore] Add test to ensure show_reblogs on follow works as expected (#2093) 2023-08-10 17:10:27 +02:00
kim
91cbcd589e
[performance] remove last of relational queries to instead rely on caches (#2091) 2023-08-10 15:08:41 +01:00
tobi
9770d54237
[feature] List replies policy, refactor async workers (#2087)
* Add/update some DB functions.

* move async workers into subprocessor

* rename FromFederator -> FromFediAPI

* update home timeline check to include check for current status first before moving to parent status

* change streamMap to pointer to mollify linter

* update followtoas func signature

* fix merge

* remove errant debug log

* don't use separate errs.Combine() check to wrap errs

* wrap parts of workers functionality in sub-structs

* populate report using new db funcs

* embed federator (tiny bit tidier)

* flesh out error msg, add continue(!)

* fix other error messages to be more specific

* better, nicer

* give parseURI util function a bit more util

* missing headers

* use pointers for subprocessors
2023-08-09 19:14:33 +02:00
Daenney
dbf487effb
[bugfix] Fix incorrect per-loop variable capture (#2092)
These should be per iteration, not per loop. This was caught by running
a build with the loopvar experiment: go build -gcflags=-d=loopvar=2.
2023-08-09 18:40:32 +02:00
kim
0ddc2edf19
[bugfix] only set content-length AFTER rewinding body bytes (#2086) 2023-08-08 12:45:29 +01:00
kim
3920bc87d1
[bugfix] don't accept unrelated statuses (#2078)
Co-authored-by: Daenney <daenney@users.noreply.github.com>
Co-authored-by: tsmethurst <tobi.smethurst@protonmail.com>
2023-08-08 12:26:34 +01:00
tobi
4b05dcde43
[chore] Update robots.txt, give chatgpt the middle finger (#2085) 2023-08-08 13:16:34 +02:00
Daenney
be3718f6e4
[chore] Use generic pointer function (#2080)
This replaces the different $TypePtr functions with a generic
implementation.
2023-08-07 18:38:11 +01:00
Vyr Cossont
0f812746b7
[feature] Allow full BCP 47 in language inputs (#2067)
* Allow full BCP 47 in language inputs

Fixes #2066

* Fuse validation and normalization for languages

* Remove outdated comment line

* Move post language canonicalization test
2023-08-07 10:25:54 +02:00
Daenney
c1375ca5c1
[chore] Remove go-playground/validator (#2069)
* [chore] Remove go-playground/validator

It turns out we're not actually using the validator code. This is a
remnant from when we intended to use it, but the presence of it and its
struct tags creates the illusion we're validating a lot of things we're
not. It resulted in some confusion when we were trying to figure out
language valdiation.

Remove all this code, so that only the validation functions from the
validate package we actually use remain. I'm not touching the struct
tags in the migrations in order to avoid things potentially thinking
migrations need to be re-run.

* [chore] Bring back a struct tag on api

The validate on internal/api is Gin doing form validation, not the
validator from go-playground/validator.
2023-08-06 12:22:40 +02:00
kim
6f4ae8f58d
[bugfix] update cache library with nil ptr fix (#2070)
Signed-off-by: kim <grufwub@gmail.com>
2023-08-06 11:55:04 +02:00
kim
9a291dea84
[performance] add caching of status fave, boost of, in reply to ID lists (#2060) 2023-08-04 12:28:33 +01:00
kim
00adf18c24
[feature] simpler cache size configuration (#2051)
* add automatic cache max size generation based on ratios of a singular fixed memory target

Signed-off-by: kim <grufwub@gmail.com>

* remove now-unused cache max-size config variables

Signed-off-by: kim <grufwub@gmail.com>

* slight ratio tweak

Signed-off-by: kim <grufwub@gmail.com>

* remove unused visibility config var

Signed-off-by: kim <grufwub@gmail.com>

* add secret little ratio config trick

Signed-off-by: kim <grufwub@gmail.com>

* fixed a word

Signed-off-by: kim <grufwub@gmail.com>

* update cache library to remove use of TTL in result caches + slice cache

Signed-off-by: kim <grufwub@gmail.com>

* update other cache usages to use correct interface

Signed-off-by: kim <grufwub@gmail.com>

* update example config to explain the cache memory target

Signed-off-by: kim <grufwub@gmail.com>

* update env parsing test with new config values

Signed-off-by: kim <grufwub@gmail.com>

* do some ratio twiddling

Signed-off-by: kim <grufwub@gmail.com>

* add missing header

* update envparsing with latest defaults

Signed-off-by: kim <grufwub@gmail.com>

* update size calculations to take into account result cache, simple cache and extra map overheads

Signed-off-by: kim <grufwub@gmail.com>

* tweak the ratios some more

Signed-off-by: kim <grufwub@gmail.com>

* more nan rampaging

Signed-off-by: kim <grufwub@gmail.com>

* fix envparsing script

Signed-off-by: kim <grufwub@gmail.com>

* update cache library, add sweep function to keep caches trim

Signed-off-by: kim <grufwub@gmail.com>

* sweep caches once a minute

Signed-off-by: kim <grufwub@gmail.com>

* add a regular job to sweep caches and keep under 80% utilisation

Signed-off-by: kim <grufwub@gmail.com>

* remove dead code

Signed-off-by: kim <grufwub@gmail.com>

* add new size library used to libraries section of readme

Signed-off-by: kim <grufwub@gmail.com>

* add better explanations for the mem-ratio numbers

Signed-off-by: kim <grufwub@gmail.com>

* update go-cache

Signed-off-by: kim <grufwub@gmail.com>

* library version bump

Signed-off-by: kim <grufwub@gmail.com>

* update cache.result{} size model estimation

Signed-off-by: kim <grufwub@gmail.com>

---------

Signed-off-by: kim <grufwub@gmail.com>
2023-08-03 11:34:35 +02:00
tobi
e8a20f587c
[bugfix] Rework MultiError to wrap + unwrap errors properly (#2057)
* rework multierror a bit

* test multierror
2023-08-02 17:21:46 +02:00
kim
2cee8f2dd8
[bugfix] fix slow accounts / statuses using emojis lookups (#2056)
* update DeleteEmoji to use faster relational tables for status / account finding

Signed-off-by: kim <grufwub@gmail.com>

* update Get{Accounts,Statuses}UsingEmoji() to also use relational tables

Signed-off-by: kim <grufwub@gmail.com>

* remove the now unneeded tags relation from newStatusQ()

Signed-off-by: kim <grufwub@gmail.com>

* fix table names

Signed-off-by: kim <grufwub@gmail.com>

* fix account and status selects using emojis

Signed-off-by: kim <grufwub@gmail.com>

---------

Signed-off-by: kim <grufwub@gmail.com>
2023-08-02 16:11:23 +02:00
kim
24516b84c2
[bugfix] handle HEAD requests more elegantly (#2055)
Signed-off-by: kim <grufwub@gmail.com>
2023-08-02 10:28:20 +02:00
tobi
cec29e2a8d
[bugfix] Allow instance accounts to be shown in search results in certain circumstances (#2053) 2023-08-02 08:31:09 +01:00
tobi
2be83fdca5
[feature] Allow users to skip http client tls verification for testing purposes (with appropriately loud warnings) (#2052) 2023-08-01 18:50:17 +01:00
tobi
2796a2e82f
[feature] Hashtag federation (in/out), hashtag client API endpoints (#2032)
* update go-fed

* do the things

* remove unused columns from tags

* update to latest lingo from main

* further tag shenanigans

* serve stub page at tag endpoint

* we did it lads

* tests, oh tests, ohhh tests, oh tests (doo doo doo doo)

* swagger docs

* document hashtag usage + federation

* instanceGet

* don't bother parsing tag href

* rename whereStartsWith -> whereStartsLike

* remove GetOrCreateTag

* dont cache status tag timelineability
2023-07-31 15:47:35 +02:00
kim
ed2477ebea
[performance] cache follow, follow request and block ID lists (#2027) 2023-07-31 11:25:29 +01:00
Vyr Cossont
b874e9251e
[feature] Implement markers API (#1989)
* Implement markers API

Fixes #1856

* Correct import grouping in markers files

* Regenerate Swagger for markers API

* Shorten names for readability

* Cache markers for 6 hours

* Update DB ref

* Update envparsing.sh
2023-07-29 12:49:14 +02:00
tobi
e747c14685
[chore] fix merge issue (#2029) 2023-07-27 12:54:27 +02:00
Vyr Cossont
22ac4607a1
[feature] Support setting private notes on accounts (#1982)
* Support setting private notes on accounts

* Reformat comment whitespace

* Add missing license headers

* Use apiutil.ParseID

* Rename Note model and cache to AccountNote

* Update golden cache config in test/envparsing.sh

* Rename gtsmodel/note.go to gtsmodel/accountnote.go

* Update AccountNote uniqueness constraint name

Now has same prefix as other indexes on this table.

---------

Co-authored-by: tobi <31960611+tsmethurst@users.noreply.github.com>
2023-07-27 10:30:39 +02:00
kim
5f3e095717
[performance] retry db queries on busy errors (#2025)
* catch SQLITE_BUSY errors, wrap bun.DB to use our own busy retrier, remove unnecessary db.Error type

Signed-off-by: kim <grufwub@gmail.com>

* remove dead code

Signed-off-by: kim <grufwub@gmail.com>

* remove more dead code, add missing error arguments

Signed-off-by: kim <grufwub@gmail.com>

* update sqlite to use maxOpenConns()

Signed-off-by: kim <grufwub@gmail.com>

* add uncommitted changes

Signed-off-by: kim <grufwub@gmail.com>

* use direct calls-through for the ConnIface to make sure we don't double query hook

Signed-off-by: kim <grufwub@gmail.com>

* expose underlying bun.DB better

Signed-off-by: kim <grufwub@gmail.com>

* retry on the correct busy error

Signed-off-by: kim <grufwub@gmail.com>

* use longer possible maxRetries for db retry-backoff

Signed-off-by: kim <grufwub@gmail.com>

* remove the note regarding max-open-conns only applying to postgres

Signed-off-by: kim <grufwub@gmail.com>

* improved code commenting

Signed-off-by: kim <grufwub@gmail.com>

* remove unnecessary infof call (just use info)

Signed-off-by: kim <grufwub@gmail.com>

* rename DBConn to WrappedDB to better follow sql package name conventions

Signed-off-by: kim <grufwub@gmail.com>

* update test error string checks

Signed-off-by: kim <grufwub@gmail.com>

* shush linter

Signed-off-by: kim <grufwub@gmail.com>

* update backoff logic to be more transparent

Signed-off-by: kim <grufwub@gmail.com>

---------

Signed-off-by: kim <grufwub@gmail.com>
2023-07-25 10:34:05 +02:00
kim
9eff0d46e4
[feature/performance] support uncaching remote emoji + scheduled cleanup functions (#1987) 2023-07-24 13:14:13 +01:00
Daenney
994d5e80d9
[chore] Disable the right webfinger test (#2018)
We don't talk about 8868889 / #1983.
2023-07-23 16:33:16 +02:00
tobi
5a29a031ad
[chore] Admin CLI + new account creation refactoring (#2008)
* set maxPasswordLength to 72 bytes, rename validate function

* refactor NewSignup

* refactor admin account CLI commands

* refactor oidc create user

* refactor processor create

* tweak password change, check old != new password
2023-07-23 12:33:17 +02:00
Daenney
f8f0312042
[feature] Report Masto version in /api/v1/instance (#1977) 2023-07-21 18:49:13 +01:00
Daenney
da8282081c
[feature] Lower remote media cache config duration (#2007)
The old default of 30d can lead to a lot of media getting cached and
significant disk usage, even on small or single person instances. A lot
of deployments decrease this value, to 15 or even less. This is less of
an issue when using object storage, but for local storage which is the
more popular deployment option running out of disk space is unpleasant.

With GoToSocial's aim to fit in small places, this changes the default
to a much more conservative 7 days. In all likelihood people aren't
scrolling that far back in their timeline so this change shouldn't
result in any issue. Existing deployments will only be affected by
this change if the admin hasn't already configured this value, or didn't
bootstrap from the example configuration.
2023-07-21 16:22:59 +02:00
tobi
89ee9d5004
[bugfix] Return all accounts when list accounts limit <= 0 (#2014) 2023-07-21 13:56:38 +01:00
tobi
d6fa74e5dc
[bugfix] Change maximumPasswordLength to 72 bytes (#2012) 2023-07-21 10:29:18 +01:00
kim
f4319740ab
[bugfix] more robust list timeline invalidation (#1995) 2023-07-18 09:43:17 +01:00
tobi
346ecabd07
[bugfix] Add missing continue statement in prepareXBetweenIDs (#1996) 2023-07-18 09:39:16 +01:00
tobi
12b6cdcd8c
[bugfix] Set Vary header correctly on cache-control (#1988)
* [bugfix] Set Vary header correctly on cache-control

* Prefer activitypub types on AP endpoints

* use immutable on file server, vary by range

* vary auth on Accept
2023-07-13 21:27:25 +02:00
Daenney
88688899aa
[chore] Skip webfinger test on CI (#1983) 2023-07-12 16:09:49 +01:00
tobi
1951e6c840
[bugfix] Update account Update logic (#1984) 2023-07-12 12:20:15 +01:00
firescry
8d92b2479f
[bugfix] Align default values in the configuration file with the code (#1971)
* Set default value of SMTPFrom to empty string

This parameter should contain proper e-mail address (to be provided by user during configuration).

* Update default values in example/config.yaml

Default values and related comments in example/config.yaml are aligned
with values defined in internal/config/defaults.go.
Small improvements to foramting of config.yaml file.

* Add default value for AdvancedThrottlingRetryAfter to internal/config/defaults.go

AdvancedThrottlingRetryAfter was introduced in 70739d3 (superseriousbusiness/gotosocial#1466).

* Update config.yaml snippets in documentation
2023-07-12 10:28:41 +02:00
MaeIsBad
0d267fd598
[bugfix] Properly handle range > content-length (#1979)
This makes the serveFileRange function return the entire file
if suffix-range is larger than content-length in compliance with RFC9110

Co-authored-by: mae <git@badat.dev>
2023-07-12 09:51:51 +02:00
tobi
ca5492b65f
[bugfix] Tidy up rss feed serving; don't error on empty feed (#1970)
* [bugfix] Tidy up rss feed serving; don't error on empty feed

* fall back to account creation time as rss feed update time

* return feed early when account has no eligible statuses
2023-07-10 17:05:59 +02:00