* rewrite Stream{} to use much less mutex locking, update related code
* use new context for the stream context
* ensure stream gets closed on return of writeTo / readFrom WSConn()
* ensure stream write timeout gets cancelled
* remove embedded context type from Stream{}, reformat log messages for consistency
* use c.Request.Context() for context passed into Stream().Open()
* only return 1 boolean, fix tests to expect multiple stream types in messages
* changes to ping logic
* further improved ping logic
* don't export unused function types, update message sending to only include relevant stream type
* ensure stream gets closed 🤦
* update to error log on failed json marshal (instead of panic)
* inverse websocket read error checking to _ignore_ expected close errors
On outgoing `GET` requests that are signed (e.g. authorized fetch),
if the initial request fails with `401`, try again, but _without_
the query parameters included in the HTTP signature.
This is primarily useful for compatibility with Mastodon; though
hopefully this can be removed in the not-too-distant future, as
they've started changing their behavior here.
Signed-off-by: Milas Bowman <devnull@milas.dev>
* [bugfix] Pass `latest` to dereferenceThread instead of barebones status
* only mark status orphaned if visibility suggests parent is really deleted
* tone down "not deref'd" warnings, since they represent a legit visibility situation
* remove FAQ entry for "status not deref'd yet"
* [chore] Disable Move API endpoints for now until Move is fully implemented in the backend
* disable other form fields
* clarify that moving to GtS also isn't implemented yet
* [chore] Simplify the User-Agent string
RFC 9110[1] includes a definition for the format of a user-agent header:
User-Agent = product *( RWS ( product / comment ) )
product = token ["/" product-version]
product-version = token
comment = "(" *( ctext / quoted-pair / comment ) ")"
ctext = HTAB / SP / %x21-27 / %x2A-5B / %x5D-7E / obs-text
An example given in the RFC: User-Agent: CERN-LineMode/2.15 libwww/2.17b3
The idea is typically start with the most important product/version, add
a (comment) if necessary and then include any auxilliary products.
However, the RFC warns against including too many auxiliary products as
those can be unnecessarily revealing.
For automated systems (i.e not a browser), the common and recommended
format is <product></version> (+uri-for-contact), followed with any
additional <product>/<version> pairs that are relevant.
This changes our UA to match that convention more closely. This makes it
easier for administrators who do user-agent parsing for statistics or
other purposes to correctly identify the version of GoToSocial.
Currently tools tend to get confused by the lack of a /<version> on the
start of our string.
[1]: https://www.rfc-editor.org/rfc/rfc9110.html#name-user-agents
* [chore] Don't use app name in UA
From all the GtS UAs I've collected, nobody seems to set/change this,
so we might as well use the static string.
The main usefulness for this is when you have multilpe GtS instances
connecting to the same DB, so they can identify as different instances
by changing the application name (though it should already be obvious
from having different usernames).
* temporarily cache account status counts to reduce no. account counts
* whoops, forgot to initAccountCounts()
* use already fetched cache capacity value
* make cache a ptr type
* whoops, use count instead of just select
* fix to correctly use the transaction
* properly wrap that tx 😇
* correctly wrap both tx types
* outline retryOnBusy() to allow the fast path to be inlined
* return err on context cancelled
* remove unnecessary storage of context in stmt, fix Exec and Query interface implementations
* shutup linter
* [feature] serdes for moved/also_known_as
* document `alsoKnownAs` and `movedTo` properties
* only implicitly populate AKA uris from DB for local accounts
* don't let remotes store more than 20 AKA uris to avoid shenanigans
* Improve context descendant sorting
Topologically sort replies, then move self-replies to top of list
* Unify descendant sort passes
* Correct test package name
* Preallocate maps
* update media / emoji cleaner funcs to use new paging package, check for same returned maxID
* fix other calls of getattachments and getmojis not using paging
* use alternative order-by function
* much simplified DereferenceStatusAncestors(), also handles edge cases now
* perform status acceptibility check before handling even as forward
* don't further dereference ancestors if they're up to date
* call enrichStatusSafely() directly to ensure we get error messages
* change getStatusByURI() semantics to return error + old model on failed update, fix deref ancestor to check for staleness before refetch
* perform a nil-check on the status.Local variable, in case it hasn't been set on new status attempting refresh
* more consistently set returned parent status, don't check if updated
* only home-timeline statuses if explicitly visible AND not explicitly invisible!
* fix broken test now that status acceptibility checks happen on forwarded statuses
* Fix EmptyJSONObject/EmptyJSONArray
These are meant to be the bytes representing an empty object and array in JSON: `{}` and `[]`. They are actually the strings `"{}"` and `"[]"`. This causes clients expecting an object or array to not be able to parse the response.
* Use json.RawMessage instead of []byte
* tidy up account, status, webfingering logic a wee bit
* go fmt
* invert published check
* alter resp initialization
* get Published from account in typeutils
* don't instantiate error for no darn good reason
* shadow err
* don't repeat error codes in wrapped errors
* don't wrap error unnecessarily
For some reason httpconv seems to have disappeared from 1.21, which
results in a 1.21 runtime with 1.20 semconv/httpconv which seems to
break some things.
For now, this rolls the OTEL dependencies back to 1.20 which should fix
the observability issues. We'll need to take a look at how to upgrade
safely and correctly in the future.
Relates to #2503.
* rewrite cache library as codeberg.org/gruf/go-structr, implement in gotosocial
* use actual go-structr release version (not just commit hash)
* revert go toolchain changes (damn you go for auto changing this)
* fix go mod woes
* ensure %w is used in calls to errs.Appendf()
* fix error checking
* fix possible panic
* remove unnecessary start/stop functions, move to main Cache{} struct, add note regarding which caches require start/stop
* fix copy-paste artifact... 😇
* fix all comment copy-paste artifacts
* remove dropID() function, now we can just use slices.DeleteFunc()
* use util.Deduplicate() instead of collate(), move collate to util
* move orderByIDs() to util package and "generify"
* add a util.DeleteIf() function, use this to delete entries on failed population
* use slices.DeleteFunc() instead of util.DeleteIf() (i had the logic mixed up in my head somehow lol)
* add note about how collate differs from deduplicate
* some small code fixups and changes
* add check in ResolveIncomingActivity for transient activity types (i.e. activity ID is nil)
* update test to handle new transient behaviour
* [feature] Account alias / move API + db models
* go fmt
* fix little cherry-pick issues
* update error checking, formatting
* add and use new util functions to simplify alias logic
* [bugfix] Replace named unique constraint on header filter header with generic unique directive
* add migration retry
* the old fixie uppie
* fix constraint name
* my goodness
* fix the sort direction of domain cache child nodes ...
* add more domain cache test cases
* add specific test for this bug to database domain test suite (thanks for writing this @tsmethurst!)
* remove unused field (this was a previous attempt at a fix)
* remove debugging println statements 😇
* only perform status-up-to-date checks if no statusable has been provided
* copy over the same style of freshness checking from status deref -> accounts
* change some var names
* check for empty account domain
* [chore] Refactor HTML templates and CSS
* eslint
* ignore "Local"
* rss tests
* fiddle with OG just a tiny bit
* dick around with polls a bit more so SR stops saying "clickable"
* remove break
* oh lord
* don't lazy load avatar
* fix ogmeta tests
* clean up some cruft
* catch remaining calls to c.HTML
* fix error rendering + stack overflow in tag
* allow templating attributes
* fix indent
* set aria-hidden on status complementary content, since it's already present in the label anyway
* tidy up templating calls a little
* try to make styling a bit more consistent + readable
* fix up some remaining CSS issues
* fix up reports
* [feature] Run ANALYZE after migrations on SQLite
This ensures that at the end of migrations, we run ANALYZE if we're
using SQLite. This should be relatively quick and guarantees that the
table and index statistics have been updated. This helps to ensure the
query planner makes better choices when it comes to picking which
indexes are used when running queries.
* [chore] use ExecContext
Uses ExecContext so we pass the context through, this is helpful for
anyone running with tracing enabled
* specifically use a much shorter refresh limit for statuses with polls
* allow specifying whether status must be upToDate in calls to Get(Visible)?TargetStatusBy_(), limit force refresh to 5 minute cooldown
* remove the PollID check from statusUpToDate()
* remove unnecessary force flag checks
* remove unused field
* check refresh status error
* use argument name 'refresh' instead of 'upToDate' to better fit with the codebase
* add statuses_poll_id_idx
* remove the definitely-not copy-pasted comment i accidentally typed out in full
* only synchronously refresh if the refresh flag is provided, otherwise do async
* fix wrong force value being provided for async
---------
Co-authored-by: tobi <tobi.smethurst@protonmail.com>
* don't drop all vote counts if hideCounts is set, refactors poll option extraction slightly
* omit voters_count when not set
* make voters_count a ptr to ensure it is omit unless definitely needed
* handle case of expires_at, voters_count and option.votes_count being nilable
* faster isNil check
* remove omitempty tags since mastodon API marks things as nullable but still sets them in outgoing json
* no need to deref status author account, will already be deref'd during previous getStatusByAP{IRI,Model}()
* don't unset the isNew flag on dereference data race
* improved code comment
* [feature] Render polls nicely on the web view
* use figure for poll, other small tweaks
* reverse share + count (lines up better)
* poll options list entries
* fix up some remaining things
* [feature] Federate status language in + out
* go fmt
* tests, little fix
* improve comments
* unnest a bit
* avoid unnecessary nil check
* use more descriptive variable for contentMap
* prefer instance languages when selecting from contentMap
* update docs to reflect lang selection
* rename rdfLangString -> rdfLangs
* update comments to mention Pollable
* iter through slice instead of map
* refactor AuthenticateFederatedRequest() to handle account suspension + fetching of owner
* small fixups
* small changes
* revert to 'IsEitherBlocked' instead of just 'IsBlocked" :grimace:
* update code comment to indicate that AuthenticateFederatedRequest() will handle account + instance dereferencing
* update go text, include text/display
* [feature] Set instance langs, show post lang on frontend
* go fmt
* WebGet
* set language for whole article, don't use FA icon
* mention instance languages + other optional config vars
* little tweak
* put languages in config properly
* warn log language parse
* change some naming around
* tidy up validate a bit
* lint
* rename LanguageTmpl in template
* [bugfix] process account delete side effects in serial, not in parallel
* StartWorkers / StartNoopWorkers for tests
* undo testrig trace logging
* log errors instead of immediately returning
* deinterface router, start messing about with deadlines
* weeeee
* thanks linter (thinter)
* write Connection: close when timing out requests
* update wording
* don't replace req
* don't bother with fancy Cause functions (I'll use them one day...)
* increment poll votes *before* enqueuing vote to client API worker
* increment vote counts before federating status update after vote in local poll
* improved vote count calculation during backend -> frontend model conversion
* add db models + functions for keeping track of threads
* give em the old linty testy
* create, remove, check mutes
* swagger
* testerino
* test mute/unmute via api
* add info log about new index creation
* thread + allow muting of any remote statuses that mention a local account
* IsStatusThreadMutedBy -> IsThreadMutedByAccount
* use common processing functions in status processor
* set = NULL
* favee!
* get rekt darlings, darlings get rekt
* testrig please, have mercy muy liege
* chore: add test of golden cases before fix of #2263
* chore: add test case to reproduce error of #2263
* [bugfix] allow store smaller PNG image than 261 bytes (#2263)
* it's happening!
* aaa
* fix silly whoopsie
* it's working pa! it's working ma!
* model report parameters
* shuffle some more stuff around
* getting there
* oo hoo
* finish tidying up for now
* aaa
* fix use form submit errors
* peepee poo poo
* aaaaa
* ffff
* they see me typin', they hatin'
* boop
* aaa
* oooo
* typing typing tappa tappa
* almost done typing
* weee
* alright
* push it push it real good doo doo doo doo doo doo
* thingy no worky
* almost done
* mutation modifers not quite right
* hmm
* it works
* view blocks + allows nicely
* it works!
* typia install
* the old linterino
* linter plz
* [feature] Block Google Bard/AI crawlers
* [feature] Block the other OpenAI crawler
* [feature] Block Common Crawl crawler
This is used in research, but also gleefully advertises itself as the
training source used in all LLMs and GPT-3.
Fixes: #2240
* [feature] Block Omgilikebot
Used by some shady big web data engine company.
* [feature] Block Meta's language model crawler
* [feature] Block well-known.dev crawler
* use minID properly for public timeline
* return paged response properly even when 0 items
* use gtserror
* page more consistently (for now)
* test
* aaa
* update typeconverter to use state structure
* deinterface the typeutils.TypeConverter -> typeutils.Converter
* finish copying over old type converter code comments
* fix cherry-pick merge issues, fix tests pointing to old typeutils interface type still
* love like winter! wohoah, wohoah
* domain allow side effects
* tests! logging! unallow!
* document federation modes
* linty linterson
* test
* further adventures in documentation
* finish up domain block documentation (i think)
* change wording a wee little bit
* docs, example
* consolidate shared domainPermission code
* call mode once
* fetch federation mode within domain blocked func
* read domain perm import in streaming manner
* don't use pointer to slice for domain perms
* don't bother copying blocks + allows before deleting
* admonish!
* change wording just a scooch
* update docs
* [feature] Support Actor URIs for webfinger queries
It's now possible to pass an Actor URI as the resource to query for when
doing a webfinger query. The code now extracts the username and domain
from the URI. The URI needs to be fully qualified, including having a
scheme of http or https to be recognised as such.
The acct scheme is handled as we used to, including dealing with an
erroneous leading @ on the username. We retain the ability to handle
resources without a scheme by parsing them again with the acct scheme if
the original parse failed. This can happen due to parsing ambiguities
when dealing with a string like user@domain.tld:port.
* [bugfix] Remove debugging changes
* [chore] Make TestExtractNamestring table-driven
* [chore] Unnest Trim and Split for readability
* [feature] Add http trace exporter, drop Jaeger
Jaeger supports ingesting traces using the OpenTelemetry gRPC or HTTP
methods. The Jaeger project has deprecated the old jaeger transport.
* Add support for submitting traces over HTTP
* Drop support for the old Jaeger protocol
* Upgrade the trace libraries to v1.17
Fixes: #2176Fixes: #2179
c.FullPath() is the empty string if a request doesn't match any route on
our mux. In those cases, there's no value in emitting a trace. The trace
will be empty, containing no other information beyond the fact that we
didn't match a route. Since Gin breaks off the processing early we don't
need to trace this request as it won't do anything and consumes no
further resources.
The 404 will still be emitted by our logs and will be visible from a
reverse proxy too.
* move SQLite pragmas into connection string
Signed-off-by: kim <grufwub@gmail.com>
* use url.Values type for SQLite connection preferences
Signed-off-by: kim <grufwub@gmail.com>
* set SQLite URI prefs properly using _pragma query key
Signed-off-by: kim <grufwub@gmail.com>
* add notes on SQLite connection preferences
Signed-off-by: kim <grufwub@gmail.com>
* fix typo
Signed-off-by: kim <grufwub@gmail.com>
* add one extra line regarding connection pooling
Signed-off-by: kim <grufwub@gmail.com>
---------
Signed-off-by: kim <grufwub@gmail.com>
* wrap bun.Tx to add our own error processing
Signed-off-by: kim <grufwub@gmail.com>
* add compile-time check for updateRowError() compatibility with sql.Row, fix wrapTx() not being used properly
Signed-off-by: kim <grufwub@gmail.com>
---------
Signed-off-by: kim <grufwub@gmail.com>
