Commit graph

561 commits

Author SHA1 Message Date
tobi
13e4bbdbfa
[chore] Duplicated media cleanup (#649)
* add migration to clean up duplicated media

* use /tmp/gotosocial for testrig storage path

* defer remove storage tempdir

* skip if not attached to status or status not found

* log errors at error level

* only log delete as else clause if successful

* just return nil on down

* reword delete logic a little bit

* check if storage base path is defined

* check for status id more thoroughly

* don't log error if just no rows

* go fmt

* break statusIDLoop when found

* break currentlyUsedLoop when found
2022-06-14 17:00:57 +01:00
tobi
da2386bab1
[chore] Fix testrig emoji bug (#646)
* fix teeny tiny bug in testrig

* Update test case
2022-06-11 16:41:34 +02:00
tobi
7f9925afe5
[chore] Refactor thread dereference a bit for clarity (#647)
* refactor thread dereference a bit for clarity

* lint for the lint gods
2022-06-11 16:25:41 +02:00
tobi
cf5c6d724d
[chore] Validate/set account domain (#619)
* add miekg/dns dependency

* set/validate accountDomain
2022-06-11 11:09:31 +02:00
tobi
dfdc473cef
[chore] Webfinger rework (#627)
* move finger to dereferencer

* totally break GetRemoteAccount

* start reworking finger func a bit

* start reworking getRemoteAccount a bit

* move mention parts to namestring

* rework webfingerget

* use util function to extract webfinger parts

* use accountDomain

* rework finger again, final form

* just a real nasty commit, the worst

* remove refresh from account

* use new ASRepToAccount signature

* fix incorrect debug call

* fix for new getRemoteAccount

* rework GetRemoteAccount

* start updating tests to remove repetition

* break a lot of tests
Move shared test logic into the testrig,
rather than having it scattered all over
the place. This allows us to just mock
the transport controller once, and have
all tests use it (unless they need not to
for some other reason).

* fix up tests to use main mock httpclient

* webfinger only if necessary

* cheeky linting with the lads

* update mentionName regex
recognize instance accounts

* don't finger instance accounts

* test webfinger part extraction

* increase default worker count to 4 per cpu

* don't repeat regex parsing

* final search for discovered accountDomain

* be more permissive in namestring lookup

* add more extraction tests

* simplify GetParseMentionFunc

* skip long search if local account

* fix broken test
2022-06-11 11:01:34 +02:00
tobi
694a490589
[feature] Add created_at and error_description to /oauth/token endpoint (#645)
* start fiddling about with oauth server

* start returning more helpful errors from oauth

* test helpful(ish) token errors

* add missing license header
2022-06-11 10:39:39 +02:00
tobi
2385b51d58
[bugfix] Make accounts media_only query also work with pg (#643) 2022-06-10 10:56:49 +02:00
tobi
5864954e2e
[bugfix] Fix domain blocks get regression (#642)
* fix domain blocks get regression

* add missing license text headers
2022-06-09 13:28:15 +02:00
f0x52
b43f9ceca9
[frontend] Restructure Frontend Sources (#634)
* 🐸restructure frontend stuff, include admin and future user panel in main repo, properly deduplicate bundles for css+js across uses

* rename bundled to dist, caught by gitignore

* re-include status.css for profile template

* default to localhost

* serve frontend panels

* add todo message for abstraction

* refactor oauth registration flow

* oauth restructure

* update footer template

* change panel routes

* remove superfluous css imports

* write bundle to disk from test server, use forked budo-express

* wrap all page content in container

for robustness with addons etc injection other elements in body

* update documentation, goreleaser, Dockerfile

* update template meta tags

* add AGPL-3.0+ license header everywhere

* only attach update listener on EventEmitter

* cleaner config for various frontend bundles

* fix bundler script paths

* Merge commit 'd191931932b9293ce1be44ed08a1e69b9fcc1e25'

* fix up dockerfile, goreleaser

* go mod tidy

* add uglifyify

* move status hide/show js to frontend bundle

* fix stylesheet color( func regressions

* update contributing docs for new build path

* update goreleaser + docker building

* resolve dependency paths properly

* update package name

* use api errorhandler

Co-authored-by: tsmethurst <tobi.smethurst@protonmail.com>
2022-06-09 12:51:19 +02:00
tobi
1ede54ddf6
[feature] More consistent API error handling (#637)
* update templates

* start reworking api error handling

* update template

* return AP status at web endpoint if negotiated

* start making api error handling much more consistent

* update account endpoints to new error handling

* use new api error handling in admin endpoints

* go fmt ./...

* use api error logic in app

* use generic error handling in auth

* don't export generic error handler

* don't defer clearing session

* user nicer error handling on oidc callback handler

* tidy up the sign in handler

* tidy up the token handler

* use nicer error handling in blocksget

* auth emojis endpoint

* fix up remaining api endpoints

* fix whoopsie during login flow

* regenerate swagger docs

* change http error logging to debug
2022-06-08 20:38:03 +02:00
kim
91c0ed863a
[bugfix] #621: add weak type handing to mapstructure decode (#625)
* Drone sig (#623)

* accept weakly typed input on mapstructure decode i.e. .UnmarshalMap()

Signed-off-by: kim <grufwub@gmail.com>

* add envparsing script to test for panics during environment variable parsing

Signed-off-by: kim <grufwub@gmail.com>

* add envparsing.sh script to drone commands

Signed-off-by: kim <grufwub@gmail.com>

* update drone signature

Co-authored-by: kim <grufwub@gmail.com>

* compare expected with output

* update expected output of envparsing

* update expected output to correct value

* use viper's unmarshal function instead
There were problems with marshalling
string slices from viper into the st.config
struct with the other function. Now, we
can use viper's unmarshal function and pass
in the custom decoder config that we need
as a hook. This ensures that we marshal
string slices from viper into our config
struct correctly.

Co-authored-by: tobi <31960611+tsmethurst@users.noreply.github.com>
Co-authored-by: tsmethurst <tobi.smethurst@protonmail.com>
2022-06-08 20:28:28 +02:00
tobi
6f6e89e271
[feature] Add paging via Link header for notifications and account statuses (#629)
* test link headers

* page get account statuses properly

* page get notifications

* add util func for packaging timeline responses

* return timelined stuff from accountstatusesget

* rename timeline response

* use new convenience function

* go fmt
2022-06-08 20:22:49 +02:00
tobi
f3b44426f4
[bugfix] Update time marshalling format to provide 3 digits of ms (#630) 2022-06-04 20:03:01 +02:00
tobi
327d3f001f
[feature] Start adding advanced configuration options, starting with samesite (#628)
* fix incorrect port being used for db

* start adding advanced config flags

* use samesite lax by default
2022-06-03 15:40:38 +02:00
kim
43ac0cdb9c
[chore] Global server configuration overhaul (#575)
* move config flag names and usage to config package, rewrite config package to use global Configuration{} struct

Signed-off-by: kim <grufwub@gmail.com>

* improved code comment

Signed-off-by: kim <grufwub@gmail.com>

* linter

Signed-off-by: kim <grufwub@gmail.com>

* fix unmarshaling

Signed-off-by: kim <grufwub@gmail.com>

* remove kim's custom go compiler changes

Signed-off-by: kim <grufwub@gmail.com>

* generate setter and flag-name functions, implement these in codebase

Signed-off-by: kim <grufwub@gmail.com>

* update deps

Signed-off-by: kim <grufwub@gmail.com>

* small change

Signed-off-by: kim <grufwub@gmail.com>

* appease the linter...

Signed-off-by: kim <grufwub@gmail.com>

* move configuration into ConfigState structure, ensure reloading to/from viper settings to keep in sync

Signed-off-by: kim <grufwub@gmail.com>

* lint

Signed-off-by: kim <grufwub@gmail.com>

* update code comments

Signed-off-by: kim <grufwub@gmail.com>

* fix merge issue

Signed-off-by: kim <grufwub@gmail.com>

* fix merge issue

Signed-off-by: kim <grufwub@gmail.com>

* improved version string (removes time + go version)

Signed-off-by: kim <grufwub@gmail.com>

* fix version string build to pass test script + consolidate logic in func

Signed-off-by: kim <grufwub@gmail.com>

* add license text, update config.Defaults comment

Signed-off-by: kim <grufwub@gmail.com>

* add license text to generated config helpers file

Signed-off-by: kim <grufwub@gmail.com>

* defer unlock on config.Set___(), to ensure unlocked on panic

Signed-off-by: kim <grufwub@gmail.com>

* make it more obvious which cmd flags are being attached

Signed-off-by: kim <grufwub@gmail.com>
2022-05-30 14:41:24 +02:00
tobi
ae5402ada6
[chore] Mastodon api fixups (#617)
* don't omitempty on description

* don't omitempty on any fields

* add ms to timestamp format

* don't omitempty on text_url

* rearrange attachment fields a bit

* just give URL again as attachment text url

* update tests

* fix accidental replace
2022-05-28 18:59:55 +01:00
tobi
2b11d4b7b0
[bugfix] Add accept: application/activity+json to dereferencer (#611)
* add Accept application/activity+json to transport

* add application/activity+json comma-separated
2022-05-27 20:09:57 +01:00
tobi
73d5766572
[chore] Debug failed account get (#612) 2022-05-27 16:44:33 +02:00
tobi
dc8cc7e364
[chore] Add test for dereferencing Owncast service account (#613) 2022-05-27 16:35:35 +02:00
tobi
1cdc163276
[performance] Don't retry/backoff invalid http requests that will never succeed (#609)
* add httpguts (ew)

* add ValidateRequest err wrapping logic

* don't retry on unrecoverable errors

* i am very clever
2022-05-26 13:38:41 +02:00
tobi
0f01f72db0
[performance] Bump default workers to CPUs * 2 (#608)
* add license text to workers.go
* bump default workers to maxprocs*2
2022-05-26 11:51:59 +02:00
tobi
5668ce1ec7
[bugfix] Fix HTML escaping in instance title (#607)
* move caption sanitization -> sanitize.go

* use sanitizeplaintext rather than removehtml

* rename sanitizecaption to sanitizeplaintext

* avoid removing html twice from statuses

* unexport remoteHTML
it's no longer used outside the text package so this
makes it less confusing

* test instance PATCH
2022-05-26 11:37:13 +02:00
tobi
f848aaa81f
[security] Set SameSite to strict instead of browser default (#606) 2022-05-25 18:08:12 +02:00
tobi
a54efa09f9
[chore] Serialize times as UTC ISO8601 instead of RFC3339 (#602)
* add time util to mimic utc ISO8601

* use ISO8601 when serializing to frontend

* update test notification
2022-05-24 17:21:27 +01:00
tobi
196d542e4a
[bugfix] Don't serialize instance account if not set (#603)
* omit contactAccount from json if not set

* test instance serialization
2022-05-24 17:20:13 +01:00
tobi
2d748a68ae
[bugfix] Only search remote if protocol is http(s) (#601) 2022-05-24 13:38:11 +02:00
tobi
21557c92d9
[bugfix] Set refetch to true in iterateDescendants (#600) 2022-05-24 11:00:37 +02:00
tobi
f0c9f4169b
[bugfix] Fix multiple dereferences of boosted status causing media duplication (#589)
* add some announces to test models

* start on announce test logic

* test federatingDB.Announce

* change signature of GetRemoteStatus

* remove 'refresh' logic and replace it with refetch

* go fmt

* remove timeline manager from processor test

* make zork created at determinate

* test get account statuses

* test get + serialize zork

* make account keys determinate

* make admin accountCreate time determinate

* test account to as

* init test config before test log

* test status to frontend

* remove daft Within check

* hack around a bit

* use index of slice
2022-05-23 16:40:03 +01:00
tobi
f5a4f4321a
[bugfix] Fix error extracting status content: no content found (#598)
* don't return error if no content found in Activity

* add test for content extraction

* go fmt
2022-05-23 16:12:46 +01:00
tobi
a09e101931
[bugfix] If status URL is empty, use URI instead and don't log unnecessary error (#597)
* test parse status with no URL

* if no status URL is available, use the URI instead
2022-05-23 16:10:48 +01:00
tobi
469da93678
[security] Check all involved IRIs during block checking (#593)
* tidy up context keys, add otherInvolvedIRIs

* add ReplyToable interface

* skip block check if we own the requesting domain

* add block check for other involved IRIs

* use cacheable status fetch

* remove unused ContextActivity

* remove unused ContextActivity

* add helper for unique URIs

* check through CCs and clean slice

* add GetAccountIDForStatusURI

* add GetAccountIDForAccountURI

* check blocks on involved account

* add statuses to tests

* add some blocked tests

* go fmt

* extract Tos as well as CCs

* test PostInboxRequestBodyHook

* add some more testActivities

* deduplicate involvedAccountIDs

* go fmt

* use cacheable db functions, remove new functions
2022-05-23 11:46:50 +02:00
Adelie Paull
caa0cde0e0
[feature] implement custom_emojis endpoint (#563)
* implement custom_emojis api endpoint

* add tests for getting custom emoji out of the database and converting to api emoji

* change sort direction of emoji query

* change logging level and initialize array with known length as per kim's suggestions

* add continue to lessen risk of making a malformed struct during conversion from db to api emojis
2022-05-20 10:34:36 +02:00
tobi
62d4d756d3
[bugfix] Stop some statuses from being home timelined when they shouldn't be (#585)
* recursively check timelineability of parent status

* check following status creator

* add tests for hometimelineability (whew)

* add test with mix of public + unlocked vis
2022-05-18 22:23:49 +01:00
tobi
b2810fedf2
[bugfix] Clean up boosts of status when the status itself is deleted (#579)
* move status wiping logic to fromcommon.go

* delete reblogs of status when a status is deleted

* add admin boost of zork to test model

* update tests to make them more determinate

* Merge branch 'main' into status_reblog_cleanup

* move status wiping logic to fromcommon.go

* delete reblogs of status when a status is deleted

* add admin boost of zork to test model

* update tests to make them more determinate

* Merge branch 'main' into status_reblog_cleanup

* test status delete via client api

* go fmt
2022-05-18 22:13:03 +01:00
tobi
f4b0d76cd4
[performance] Add further indexes to mitigate laggy queries (#586)
* start adding more indexes as a migration

* update sqlite version
2022-05-18 15:58:26 +01:00
tobi
5ef41ba3f2
[chore] Timeline test updates (#578)
* add admin boost of zork to test model

* update tests to make them more determinate

* remove printf call
2022-05-16 18:48:59 +02:00
tobi
b915a41811
[feature] Basic config validation (#562)
* add optional config validation

* clarify that host and protocol are required

* add validation for host and protocol

* pass prerunArgs as a struct (validate by default)
2022-05-16 14:13:19 +02:00
tobi
b143877995
[feature] Unused avatar and header cleanup (#574)
* rename + tidy up remote pruning

* fix media attachment account join
see https://bun.uptrace.dev/guide/golang-orm.html#table-relationships

* update logging to new function name

* add get avatars and headers to bun

* add pruneallmeta function

* don't set uncached since we're deleting anyway

* fix totalPruned being in wrong place

* test pruning meta

* go fmt ./...

* rename mediaprune

* add meta pruning to routine mediaprune

* tidy up cleanup job scheduling

* rename adminmediaremoteprune

* update mediacleanup to use renamed prune func

* update swagger docs a little bit

* reuse cancel + context
2022-05-15 15:45:04 +01:00
Sashanoraa
6e947ff266
[feature] Media cleanup endpoint (#560)
Adds an admin endpoint to trigger a remote media cleanup.

Fixed #348

Signed-off-by: Sashanoraa <sasha@noraa.gay>
2022-05-15 14:52:46 +02:00
kim
223025fc27
[security] transport.Controller{} and transport.Transport{} security and performance improvements (#564)
* cache transports in controller by privkey-generated pubkey, add retry logic to transport requests

Signed-off-by: kim <grufwub@gmail.com>

* update code comments, defer mutex unlocks

Signed-off-by: kim <grufwub@gmail.com>

* add count to 'performing request' log message

Signed-off-by: kim <grufwub@gmail.com>

* reduce repeated conversions of same url.URL object

Signed-off-by: kim <grufwub@gmail.com>

* move worker.Worker to concurrency subpackage, add WorkQueue type, limit transport http client use by WorkQueue

Signed-off-by: kim <grufwub@gmail.com>

* fix security advisories regarding max outgoing conns, max rsp body size

- implemented by a new httpclient.Client{} that wraps an underlying
  client with a queue to limit connections, and limit reader wrapping
  a response body with a configured maximum size
- update pub.HttpClient args passed around to be this new httpclient.Client{}

Signed-off-by: kim <grufwub@gmail.com>

* add httpclient tests, move ip validation to separate package + change mechanism

Signed-off-by: kim <grufwub@gmail.com>

* fix merge conflicts

Signed-off-by: kim <grufwub@gmail.com>

* use singular mutex in transport rather than separate signer mus

Signed-off-by: kim <grufwub@gmail.com>

* improved useragent string

Signed-off-by: kim <grufwub@gmail.com>

* add note regarding missing test

Signed-off-by: kim <grufwub@gmail.com>

* remove useragent field from transport (instead store in controller)

Signed-off-by: kim <grufwub@gmail.com>

* shutup linter

Signed-off-by: kim <grufwub@gmail.com>

* reset other signing headers on each loop iteration

Signed-off-by: kim <grufwub@gmail.com>

* respect request ctx during retry-backoff sleep period

Signed-off-by: kim <grufwub@gmail.com>

* use external pkg with docs explaining performance "hack"

Signed-off-by: kim <grufwub@gmail.com>

* use http package constants instead of string method literals

Signed-off-by: kim <grufwub@gmail.com>

* add license file headers

Signed-off-by: kim <grufwub@gmail.com>

* update code comment to match new func names

Signed-off-by: kim <grufwub@gmail.com>

* updates to user-agent string

Signed-off-by: kim <grufwub@gmail.com>

* update signed testrig models to fit with new transport logic (instead uses separate signer now)

Signed-off-by: kim <grufwub@gmail.com>

* fuck you linter

Signed-off-by: kim <grufwub@gmail.com>
2022-05-15 11:16:43 +02:00
kim
4ac508f037
[chore] Update LE server to use copy of main http.Server{} to maintain server timeouts etc (#571)
* update LE server to use copy of main HTTP server to maintain server timeouts etc

Signed-off-by: kim <grufwub@gmail.com>

* shutup linter

Signed-off-by: kim <grufwub@gmail.com>
2022-05-15 11:10:55 +02:00
kim
9f2a2abe42
[bugfix] update syslog tests to listen/connect specifically to ipv4 loopback (#561)
Signed-off-by: kim <grufwub@gmail.com>
2022-05-13 13:35:41 +02:00
tobi
898d256511
[bugfix] Fix server trying to listen twice on same address when l.e. enabled (#557) 2022-05-12 11:35:36 +02:00
tobi
8e30671a62
[bugfix] Add account raw note fix (#556)
* hack the latest migration to fix psql issue

* add new migration to fix previous migration

* adjust query
2022-05-11 19:27:25 +02:00
Eamonn O'Brien-Strain
b24b71c0a4
[feature] Include password strength in error message when password strength is too low (#550)
* When password validation fails, return how close to enough entropy it has.

* Shorter version of low-strength password error message
2022-05-09 10:31:46 +02:00
tobi
5004e0a9da
[bugfix] Fix remote media pruning failing if media already gone (#548)
* fix error check of prune to allow missing files

* update go-store library, add test for pruning item with db entry but no file

Signed-off-by: kim <grufwub@gmail.com>

* remove now-unneccessary error check

Signed-off-by: kim <grufwub@gmail.com>

Co-authored-by: kim <grufwub@gmail.com>
2022-05-08 18:49:45 +01:00
kim
26b74aefaf
[bugfix] Fix existing bio text showing as HTML (#531)
* fix existing bio text showing as HTML

- updated replaced mentions to include instance
- strips HTML from account source note in Verify handler
- update text formatter to use buffers for string writes

Signed-off-by: kim <grufwub@gmail.com>

* go away linter

Signed-off-by: kim <grufwub@gmail.com>

* change buf reset location, change html mention tags

Signed-off-by: kim <grufwub@gmail.com>

* reduce FindLinks code complexity

Signed-off-by: kim <grufwub@gmail.com>

* fix HTML to text conversion

Signed-off-by: kim <grufwub@gmail.com>

* Update internal/regexes/regexes.go

Co-authored-by: Mina Galić <mina.galic@puppet.com>

* use improved html2text lib with more options

Signed-off-by: kim <grufwub@gmail.com>

* fix to produce actual plaintext from html

Signed-off-by: kim <grufwub@gmail.com>

* fix span tags instead written as space

Signed-off-by: kim <grufwub@gmail.com>

* performance improvements to regex replacements, fix link replace logic for un-html-ing in the future

Signed-off-by: kim <grufwub@gmail.com>

* fix tag/mention replacements to use input string, fix link replace to not include scheme

Signed-off-by: kim <grufwub@gmail.com>

* use matched input string for link replace href text

Signed-off-by: kim <grufwub@gmail.com>

* remove unused code (to appease linter :sobs:)

Signed-off-by: kim <grufwub@gmail.com>

* improve hashtagFinger regex to be more compliant

Signed-off-by: kim <grufwub@gmail.com>

* update breakReplacer to include both unix and windows line endings

Signed-off-by: kim <grufwub@gmail.com>

* add NoteRaw field to Account to store plaintext account bio, add migration for this, set for sensitive accounts

Signed-off-by: kim <grufwub@gmail.com>

* drop unnecessary code

Signed-off-by: kim <grufwub@gmail.com>

* update text package tests to fix logic changes

Signed-off-by: kim <grufwub@gmail.com>

* add raw note content testing to account update and account verify

Signed-off-by: kim <grufwub@gmail.com>

* remove unused modules

Signed-off-by: kim <grufwub@gmail.com>

* fix emoji regex

Signed-off-by: kim <grufwub@gmail.com>

* fix replacement of hashtags

Signed-off-by: kim <grufwub@gmail.com>

* update code comment

Signed-off-by: kim <grufwub@gmail.com>

Co-authored-by: Mina Galić <mina.galic@puppet.com>
2022-05-07 17:55:27 +02:00
kim
08eb271a4c
[performance] improved logrus output switching performance (#544)
* improved logrus output switching performance

Signed-off-by: kim <grufwub@gmail.com>

* add license to test file

Signed-off-by: kim <grufwub@gmail.com>
2022-05-07 17:46:59 +02:00
kim
a561ef3541
[feature] Update media manager to use internal/worker package (#543)
* update media manager to use internal/worker package, update worker with better logging

Signed-off-by: kim <grufwub@gmail.com>

* fix Queue() trace log message format operators

Signed-off-by: kim <grufwub@gmail.com>

* update media manager comment to match updated worker implementation

Signed-off-by: kim <grufwub@gmail.com>
2022-05-07 17:36:01 +02:00
tobi
d7b46a4b63
[chore] Move initial MediaRemotePrune to cmd.server function (#536)
* move initial prune up to the server start function

* don't do initial prune when starting media manager

* add admin/mediaremoteprune in processor
2022-05-03 10:56:18 +01:00
kim
e09958e7f1
[feature] Run initial cache cleanup on gotosocial start in case max-age changed (#533)
Signed-off-by: kim <grufwub@gmail.com>
2022-05-03 11:05:22 +02:00
tobi
b28fd7433a
[chore] Fix inbox iris test to use 'contains' instead of index (#529) 2022-05-02 16:12:56 +02:00
tobi
3112949b6e
[bugfix] Fix panic when relative tmpl path given to router (#528) 2022-05-02 16:06:03 +02:00
Adelie Paull
9265a09a65
[bugfix] Allow self-boosting for any visibility but direct (#510)
* create visibility filter for boostability and allow self-boosting for any visbility but direct messages

* add a followers-only status to local_account_2

* fix typo in comment

* add license header, unwrap errors, be explicit about non-boostable visibility settings to avoid rogue boosting from miscoded clients, use ID compare for checking if self-boosting

* add tests for statusboostable filter

* fix tests that were affected by adding a new status to the test data

* fix the rest of tests affected by adding a status to the textrig data
2022-05-02 15:23:37 +02:00
kim
b56dae8120
[chore] Update all but bun libraries (#526)
* update all but bun libraries

Signed-off-by: kim <grufwub@gmail.com>

* remove my personal build script changes

Signed-off-by: kim <grufwub@gmail.com>
2022-05-02 15:05:18 +02:00
tobi
e06bf9cc9a
add extra indexes as a migration (#527) 2022-05-02 13:55:48 +01:00
tobi
a5852fd7e4
[performance] Speed up some of the slower db queries (#523)
* remove unnecessary LOWER() db calls

* warn during slow db queries

* use bundb built-in exists function

* add db block test

* update account block query

* add domain block db test

* optimize domain block query

* fix implementing wrong test

* exclude most columns when checking block

* go fmt

* remote more unnecessary use of LOWER()
2022-05-02 11:53:46 +01:00
kim
faae2505c0
Add logging to the new generic worker package (#516)
* add logging to generic worker type
2022-05-01 15:45:15 +01:00
tobi
d93e8ddf75
[chore] Add Federatingactor.Send() tests and log call (#509)
* expose testrig util functions

* add tests for federatingActor

* rename some suite vars
2022-04-29 15:53:04 +02:00
tobi
63f84da3e4
[chore] Inboxes for iri test (#508)
* tidy up some federation tests

* add missing license to following.go

* give zork some followers, as a treat

* test InboxesForIRI

* Go fmt

* update timeline tests
2022-04-29 15:05:13 +02:00
Terin Stock
f9b351438b
[feature] allow absolute path to templates (#507)
Only attempt to join the `WebTemplateBaseDir` with the current working
directory if the user has not configured an absolute path to the
template directory.

This changeset also makes a similar change to the testrig, allowing
tests to configure an alternative location for the templates directory.

Fixes #411

Signed-off-by: Terin Stock <terinjokes@gmail.com>
2022-04-29 11:00:25 +02:00
kim
69011d4901
Add support for running profiling when debug build-tags provided (#491)
* wrap root HTTP handler in debug.WithPprof(), rearrange router.Start() to support this
* remove unused code
* set debug buildtag in build script when $DEBUG set
* update go-debug version with fixed handler
* use clone of router.srv for LE cert manager, reset server timeouts in debug
* add kim's other libraries to README
2022-04-28 13:32:53 +01:00
kim
420e2fb22b
replace async client API / federator msg processing with worker pools (#497)
* replace async client API / federator msg processing with worker pools
* appease our lord-and-saviour, the linter
2022-04-28 13:23:11 +01:00
kim
cc5f2e98b7
[bugfix] Fix possible race condition in federatingdb (#490)
Signed-off-by: kim <grufwub@gmail.com>
2022-04-28 11:18:27 +02:00
Terin Stock
8e80f983b3
[bugfix] use Exec to rename media_attachments (#498)
The `database/sql` package in the Go stdlib has two methods that can run
queries against the database driver: `Query` and `Exec`. When the query
returns rows, such as "SELECT", the package expects the use of `Query`,
and returns `*sql.Rows`. When the query does not return rows, the
package expects the use of `Exec`, which returns `sql.Result`.

This changeset corrects the "media_attachments" migration to using
`ExecContext`, as "ALTER TABLE" does not return rows.

Signed-off-by: Terin Stock <terinjokes@gmail.com>
2022-04-28 08:43:41 +01:00
tobi
9cf66bf298
[chore] Return more useful errors from auth failure (#494)
* try rsa_sha256 sig algo first

* return more informative errors from auth

* adapt to reworked auth function
2022-04-26 18:10:11 +02:00
tobi
728c4a5e38
[bugfix] Trim log entries to 1700 chars before they enter syslog (#493)
* start implementing trimming hook

* add test with very long test

* test syslog w/ unix socket + long (trimmed) msg

* trim long entries with trimhook

* trim to 1700 chars instead
2022-04-26 17:55:24 +02:00
tobi
2259838108
[bugfix] Fix CWs not showing sometimes (#488)
* allow summaries that are parsed as iris

* test parsing a status with iri summary
2022-04-26 10:47:21 +02:00
tobi
eeb78bd141
[bugfix] Allow processing of .png files where checksum is not correct (#487)
* add png stripping code from google/wuffs

* experiment with stripping data from pngs

* add test images

* use StrippedPngDecode for pngs

* add StrippedPngDecode func

* update tests for (no)alphachannel pngs

* nolint on copied function
2022-04-25 14:45:44 +02:00
0hlov3
d5fe729cf4
[chore] Add two new test images for media processing (#424) 2022-04-24 14:49:37 +02:00
tobi
0d05bf473b
[bug] Fix status API / status web API being case sensitive (#481)
* make getStatus by id case-insensitive

* test get status case insensitive

* init config before log
2022-04-24 14:24:43 +02:00
tobi
9813a044c0
[feature] Implement media v2 endpoint to accommodate Tusky 17 (#480)
* serve v2 media api

* go fmt
2022-04-24 13:11:52 +02:00
tobi
8d34d5af3c
Revert "[chore] Tidy up federating db locks a tiny bit (#472)" (#479)
This reverts commit 31c1067a54.
2022-04-24 11:56:06 +02:00
tobi
31c1067a54
[chore] Tidy up federating db locks a tiny bit (#472) 2022-04-19 16:31:37 +02:00
tobi
1e3b38573d
[bugfix] Fix infinite domain block database loop (#467)
This fixes an issue where the domain block logic would go into an infinite loop.
2022-04-18 17:44:36 +02:00
tobi
c365863ea9
[bugfix] Use our own (Batch)Deliver implementation for federated messages (#466) 2022-04-18 17:17:05 +02:00
tobi
094f032f74
[feature] Add log-db-queries config option (#465) 2022-04-18 16:47:11 +02:00
tobi
7883dd5499
[bugfix] Convert IDNs to punycode before using as session name (#458)
* convert hostname to punycode for session name

* test punycode
2022-04-16 13:09:42 +02:00
tobi
af97d6bb7e
[bugfix] Use background ctx for domain block side effects (#457) 2022-04-16 12:56:41 +02:00
tobi
26683b3d49
[feature] Web profile pages for accounts (#449)
* add default avatars

* allow webModule to error

* return errWithCode from account get

* add AccountGetLocalByUsername

* check nil requesting account

* add timestampShort function for just month/year

* move loading logic to New + add default avatars

* add profile page view

* update swagger docs

* add excludeReblogs to GetAccountStatuses

* ignore casing when selecting local account by username

* appropriate redirects

* css fiddling

* add 'about' heading

* adjust thread page to work with routing

* return AP representation if requested + authorized

* simplify auth check

* go fmt

* golangci-lint ignore math/rand
2022-04-15 14:33:01 +02:00
tobi
a7e9dee33d
[bugfix] Fix broken only_media and only_public flags on /api/v1/accounts/:id/statuses (#448)
* bump superseriousbusiness/activity version

* fetch list of follower inbox iris

* add test for getting account statuses

* fix incorrect publicOnly parsing

* update media not null query
2022-04-10 15:11:12 +02:00
tobi
707de094a0
[dependency] Update superseriousbusiness/activity dependency (#446)
* bump superseriousbusiness/activity version

* fetch list of follower inbox iris
2022-04-05 18:05:24 +02:00
tobi
03d7c75ebf
[chore] Update Go version to 1.18 (#444)
* linting with new golangci-lint version

* update go to 1.18

* bump versions in drone.yml

* use new runtime/debug package for version info

* remove Commit build flag from goreleaser

* remove mock commit + version from build script

* go fmt

* add dummy version env flag to test container

* install git in golang container for testing

* only set versionString if Version is defined
2022-04-02 15:40:09 +02:00
tobi
37d310f981
[feature] Dereference remote mentions when the account is not already known (#442)
* remove mention util function from db

* add ParseMentionFunc to gtsmodel

* add parseMentionFunc to processor

* refactor search to simplify it a bit

* add parseMentionFunc to account

* add parseMentionFunc to status

* some renaming for clarity

* test dereference of unknown mentioned account
2022-03-29 11:54:56 +02:00
tobi
8cc70a14d6
[bugfix] Fix media uploads blocking after a failed upload #438
safer in case of failure
2022-03-22 12:42:34 +01:00
tobi
25cab0e1f4
[bugfix] Fix images not being processed correctly sometimes (#437)
* bump exif-terminator to latest version

* add and test giant turnip from turnip.farm

* don't error if content property is nil
2022-03-21 19:46:51 +01:00
tobi
36b2f2b4e6
don't error if content property is nil (#436) 2022-03-21 17:10:31 +01:00
tobi
73e9cca701
[bugfix] Close ReadClosers properly in the media package (#434)
* defer lock reader

* close readers when finished with them

* close the reader in the teereader when finished
2022-03-21 13:41:44 +01:00
tobi
55ad6dee71
[feature] Admin account actions (#432)
* add accountAction to the admin API

* model admin account action

* add admin account action to the processor

* add migration for new AdminAccountActions table

* fix accounts admin path

* Update swagger docs
2022-03-19 12:01:40 +01:00
tobi
532c4cc697
[feature] Federate local account deletion (#431)
* add account delete to API

* model account delete request

* add AccountDeleteLocal

* federate local account deletes

* add DeleteLocal

* update transport (controller) to allow shortcuts

* delete logic + testing

* update swagger docs

* more tests + fixes
2022-03-15 16:12:35 +01:00
tobi
e63b653199
[performance] Add dereference shortcuts to avoid making http calls to self (#430)
* update transport (controller) to allow shortcuts

* go fmt

* expose underlying sig transport to allow test sigs
2022-03-15 15:01:19 +01:00
tobi
4b4c935e02
[bugfix] Fix bug where admin panel could not be accessed at /admin (#427)
* clarify comments

* tidy up static serving + add /admin redirect
2022-03-13 18:35:26 +01:00
tobi
e306233166
[bugfix] Fix html-escaped characters in content warnings (#426)
* test status create with odd CWs

* use SanitizeCaption for content warning escaping
2022-03-13 16:37:45 +01:00
tobi
8de928b5e9
[performance] Database optimizations (#419)
* create first index on notifications

* tidy up + add tests

* log queries for trace, ops for debug

* index commonly used fields

* rearrange query

* add a few more indexes

* remove schema-breaking index
(add this back in later)

* re-add cleanup query index
2022-03-07 11:33:18 +01:00
tobi
07727753b9
[feature] Clean up/uncache remote media (#407)
* Add whereNotEmptyAndNotNull

* Add GetRemoteOlderThanDays

* Add GetRemoteOlderThanDays

* Add PruneRemote to Manager interface

* Start implementing PruneRemote

* add new attachment + status to tests

* fix up and test GetRemoteOlderThan

* fix bad import

* PruneRemote: return number pruned

* add Cached column to mediaattachment

* update + test pruneRemote

* update mediaTest

* use Cached column

* upstep bun to latest version

* embed structs in mediaAttachment

* migrate mediaAttachment to new format

* don't default cached to true

* select only remote media

* update db dependencies

* step bun back to last working version

* update pruneRemote to use Cached field

* fix storage path of test attachments

* add recache logic to manager

* fix trimmed aspect ratio

* test prune and recache

* return errwithcode

* tidy up different paths for emoji vs attachment

* fix incorrect thumbnail type being stored

* expose TransportController to media processor

* implement tee-ing recached content

* add thoughts of dog to test fedi attachments

* test get remote files

* add comment on PruneRemote

* add postData cleanup to recache

* test thumbnail fetching

* add incredible diagram

* go mod tidy

* buffer pipes for recache streaming

* test for client stops reading after 1kb

* add media-remote-cache-days to config

* add cron package

* wrap logrus so it's available to cron

* start and stop cron jobs gracefully
2022-03-07 11:08:26 +01:00
tobi
55b83bea7c
[feature] Add postDataCallbackFunc to allow cleanup (#408) 2022-02-22 13:50:33 +01:00
tobi
15d1e6b3a1
[bugfix] Fix thumbnail image type (#406)
* fix thumbnail content-type

* test fix thumbnail content-type
2022-02-21 11:26:26 +01:00
tobi
a089a98ea9
[feature] Gin enable gzip encoding (#405)
* add gin gzip dependency

* add gzip middleware to router

* go mod tidy
2022-02-19 12:12:41 +01:00
tobi
23034ec145
[feature] Stream files via reader (#404)
* serve files via reader rather than byte slice

* close readcloser when we're done with it

* cast reader to readcloser
2022-02-19 11:44:56 +01:00
Shadowfacts
09d6478d72
[bugfix] Only mark cookies as Secure on https (#398)
Fixes cookies not being stored/sent by Safari when serving over plain http
2022-02-15 16:00:07 +01:00
tsmethurst
8c0141d103 store and retrieve processState atomically 2022-02-08 13:38:44 +01:00
tsmethurst
dba9ad4348 hopefully fix potential race condition 2022-02-08 13:17:10 +01:00
tsmethurst
c4ece24654 log error on header/avatar fetch fail 2022-02-08 12:37:58 +01:00
tsmethurst
b42b0a667e go fmt 2022-02-08 12:19:54 +01:00
tsmethurst
3301148bb7 merge fixup 2022-02-08 12:17:53 +01:00
tsmethurst
4c294a596a Merge branch 'main' into media_refactor 2022-02-08 12:17:18 +01:00
tsmethurst
e811d03cc9 update outdated comment 2022-02-08 10:08:27 +01:00
Forest Johnson
6ed368cbeb
[feature] add authorization to the already-existing authentication (#365)
* add ensureUserIsAuthorizedOrRedirect to /oauth/authorize

* adding authorization (email confirm, account approve, etc) to TokenCheck

* revert un-needed changes to signin.go

* oops what happened here

* error css

* add account.SuspendedAt check

* remove redundant checks from oauth util Authed function

* wip tests

* tests passing

* stop stripping useful information from ErrAlreadyExists

* that feeling of scraping the dryer LINT off the screen

* oops I didn't mean to get rid of this NewTestRouter function

* make tests work with recorder

* re-add ConfigureTemplatesWithGin to handle template path err

Co-authored-by: tsmethurst <tobi.smethurst@protonmail.com>
2022-02-07 12:04:31 +01:00
tobi
5c9d20cea3
don't bother negotiating Accept for webfinger (#381) 2022-02-06 15:00:57 +01:00
tobi
85b4f96263
if accountDomain isn't set, use Host value instead (#380) 2022-02-06 14:07:38 +01:00
tobi
1b36e85840
[feature] Rework timeline code to make it useful for more than just statuses (#373)
* add preparable and timelineable interfaces

* initialize timeline manager within the processor

* generic renaming

* move status-specific timeline logic into the processor

* refactor timeline to make it useful for more than statuses
2022-02-05 12:47:38 +01:00
tobi
18e7537393
[bug] Fix OIDC users requiring second approval (#371)
* tidy up NewSignup
* pre-approve users created via OIDC
2022-01-31 16:03:47 +01:00
Forest Johnson
5be8a7a7ea
[bug] Send plaintext emails to fix "message refused: Message is not RFC 2822 compliant" (#366)
* trying to fix "message refused: Message is not RFC 2822 compliant"

* fix "message refused: Message is not RFC 2822 compliant"

550 5.7.1 Delivery not authorized, message refused: Message is not RFC
2822 compliant

* remove silly regex

* lint

* fix tests

* we should use text/template instead of html/template now
2022-01-31 11:46:20 +01:00
tobi
959e38ac5c
[bug] Fix sqlite empty address issue (#370)
* error when empty address has been set for sqlite

* better explain sqlite db-address setting
2022-01-30 17:06:28 +01:00
tobi
66175c8ad9
[bug] Fix minimum description length check (#369)
* add/update tests

* don't check min description length on media create
2022-01-30 16:10:53 +01:00
tobi
e13718148d
[feature] Make Let's Encrypt integration an opt-in feature (#368) 2022-01-30 14:57:33 +01:00
tsmethurst
d9a778a795 put store lock file in base of storage 2022-01-29 12:16:12 +01:00
tsmethurst
c156602c66 ensure blocking calls to getRemoteAccount before showing stuff to client 2022-01-25 13:48:13 +01:00
tsmethurst
5d9e9e0e7f fix up some account conversion logic 2022-01-25 12:03:25 +01:00
tsmethurst
eba66d3a88 only update account in db if changed 2022-01-25 11:21:22 +01:00
tsmethurst
926b37e271 change getaccount function signature 2022-01-24 18:12:42 +01:00
tsmethurst
a6fb93ae2a ctx => innerctx 2022-01-24 18:12:28 +01:00
tsmethurst
c12520167d use background context w/deadline 2022-01-24 18:12:04 +01:00
tsmethurst
8e6ba1de52 start trying to figure out why this test is failing 2022-01-24 18:11:38 +01:00
tsmethurst
0e7f24ff26 test with disk storage as well 2022-01-24 18:11:27 +01:00
tsmethurst
667e7f112c update remote account get/deref logic 2022-01-24 13:12:17 +01:00
tsmethurst
3c1eb155e4 add file size checks 2022-01-23 15:52:18 +01:00
tsmethurst
c157b1b20b rework data function to provide filesize 2022-01-23 14:41:58 +01:00
tsmethurst
589bb9df02 pass reader around instead of []byte 2022-01-16 18:52:55 +01:00
tsmethurst
723bfe8944 lint, fmt 2022-01-15 17:41:18 +01:00
tsmethurst
6bf39d0fc1 emoji code passing muster 2022-01-15 17:36:15 +01:00
tsmethurst
c4a533db72 start fixing up emoji processing code 2022-01-15 14:33:58 +01:00
tsmethurst
33ca5513ad pin instanceAccountID to in-process emoji 2022-01-11 17:51:45 +01:00
tsmethurst
113f9d9ab4 pass a function into the manager, start work on emoji 2022-01-11 17:49:14 +01:00
tsmethurst
e0f9323b9a test the media manager a bit, add shutdown logic 2022-01-10 18:36:09 +01:00
tsmethurst
0ef478584c add async test 2022-01-09 18:52:28 +01:00
tsmethurst
dccf21dd87 tests are passing, but there's still much to be done 2022-01-09 18:41:22 +01:00
tsmethurst
f61c3ddcf7 compiling now 2022-01-08 17:17:01 +01:00
tsmethurst
c2ff8f392b further refinements 2022-01-08 13:45:42 +01:00
tsmethurst
7ebe0f6a15 start working on thumb + full funcs 2022-01-04 17:37:54 +01:00
tsmethurst
8abfa7751a return very partial image on first upload 2022-01-03 17:37:38 +01:00
tsmethurst
2f57eb5ece fiddle around with workers 2022-01-02 15:00:53 +01:00
tsmethurst
c4d63d125b more refactoring, media handler => manager 2021-12-28 16:36:00 +01:00
tsmethurst
6803c1682b start refactor of media package 2021-12-27 18:03:36 +01:00
embr
ed9158fa05
[fix] Make postgres connections magically work in common setups (#352)
* Don't use the system 'postgres' database by default

* Use postgres adapter defaults

The pgx code actually goes to great lengths to make postgres connections
Just Work(tm) out of the box, including supporting `~/.pg_service.conf`,
SSL certificates, UNIX sockets if it can find a socket at a common path,
and falling back to TCP to localhost if not.

(On Windows, it won't try to use UNIX sockets, but will read credentials
from %appdata% as is standard over there.)

By applying our flags as overrides only when they're specified, database
connections should Just Work(tm) anywhere `psql gotosocial` does.
2021-12-21 12:08:27 +01:00
tobi
ef5a9256a8
Extend license notices to 2022 (#354) 2021-12-20 18:42:19 +01:00
tobi
cb8688f429
Remove unnecessary storage config variables (#344)
* rewire config to not use extraneous serve vars

* rename 'file' to 'local' for consistency

* use Type and Size again
2021-12-20 15:19:53 +01:00
tobi
2582515b4d
return first offer when no accept header set (#351) 2021-12-20 11:08:54 +01:00
embr
86e8e7fd21
Log when listening (#350)
* Log when starting letsencrypt endpoint + clearer errors

Running `gotosocial server` with the default configuration will try to
bind to :80 and listen for letsencrypt challenges, which will fail if
running as non-root (w/o capabilities), or if eg. nginx hogs the port.

When that happens, this should make it more obvious what's wrong.

* Log what address/port we're listening on

Always nice not to have to guess. Also feels more consistent than just
doing it for the letsencrypt endpoint.
2021-12-20 10:34:57 +01:00
tobi
c111b239f7
Add optional syslog logrus hook (#343)
* add optional syslog logrus hook

* document syslog
2021-12-12 18:00:20 +01:00
tobi
67ac8db190
Upstep Go dependencies (#340)
* Upstep Go dependencies

* tiny linter fix

* Tidy
2021-12-12 15:47:51 +01:00
tobi
e2daf0f012
Add Accept header negotiation to relevant API endpoints (#337)
* start centralizing negotiation logic for API

* swagger document nodeinfo endpoint

* go fmt

* document negotiate function

* use content negotiation

* tidy up negotiation logic

* negotiate content throughout client api

* swagger

* remove attachment on Content

* add accept header to test requests
2021-12-11 17:50:00 +01:00
tobi
0884f89431
Implement Cobra CLI tooling, Viper config tooling (#336)
* start pulling out + replacing urfave and config

* replace many many instances of config

* move more stuff => viper

* properly remove urfave

* move some flags to root command

* add testrig commands to root

* alias config file keys

* start adding cli parsing tests

* reorder viper init

* remove config path alias

* fmt

* change config file keys to non-nested

* we're more or less in business now

* tidy up the common func

* go fmt

* get tests passing again

* add note about the cliparsing tests

* reorganize

* update docs with changes

* structure cmd dir better

* rename + move some files around

* fix dangling comma
2021-12-07 13:31:39 +01:00
tobi
ce22e03f9d
Require confirmed email when checking oauth token (#332)
* move token checker to security package

* update tests with new security package

* add oauth token checking to security package

* check if user email confirmed when parsing token
2021-11-27 14:53:34 +01:00
Forest Johnson
ab316d2250
Fix broken defaults and broken flags (#314)
* start with a default config, not an empty config.

* some data structures were present on Empty config but not Default config

* the monkey patched CLIContext is working

* remove print debugging log

* make the behaviour of the flags consistent across all data types

Conflicts:
	internal/config/config.go

* try to fix accidentally broken test
2021-11-24 17:01:38 +01:00
tobi
3caae376e7
Fix streamed messages ending up in wrong timeline(s) (#325)
* define timeline consts

* remove double stream of status

* change test stream creation up a bit

* stream messages more selectively

* add test for streaming new status creation via clientAPI

* tidy code + comments a bit

* tidy up tests

* make sure new status isn't streamed to public
2021-11-22 19:03:21 +01:00
tobi
79ccd8fd8a
Fix mentioned accounts visibility bug (#323)
* update other tests

* set test status to followers_only

* add test dm

* fix mentioned accounts not being added to relevantAccounts

* add some visibility tests for statuses
2021-11-22 14:40:23 +01:00
tobi
b46e5fb65d
Fix incorrect target being used in CC prop for mentioning statuses (#322)
* hardcode time on admin_account_status_3

* add test for as status with mention

* fix incorrect target URI being used for mentions
2021-11-22 12:48:46 +01:00
tobi
24f9e11221
Fix image description unnecessarily html-escaping innocent characters (#321)
* implement SanitizeCaption function

* tidy up text test setup
2021-11-22 11:49:11 +01:00
tobi
1ded58b34b
add bindAddress configuration option (#320)
* add bindAddress configuration option

* clarify that bindAddress can be a hostname
2021-11-22 10:55:52 +01:00
tobi
6c0550e76f
properly initialize user client module (#319) 2021-11-22 09:04:16 +01:00
tobi
f8630348b4
Enable stricter linting with golangci-lint (#316)
* update golangci-lint

* add golangci config file w/ more linters

* correct issues flagged by stricter linters

* add more generous timeout for golangci-lint

* add some style + formatting guidelines

* move timeout to config file

* go fmt
2021-11-22 08:46:19 +01:00
tobi
38d73f0316
tidy up database creation logic a bit (#317) 2021-11-21 17:41:51 +01:00
tobi
f3e36de78a
fix broken GetInstanceAccount selector (#311)
* fix broken GetInstanceAccount selector

* patch tests
2021-11-17 14:40:39 +01:00
tobi
ff7c96caa3
Fix instance account not being created (#310)
* Rework 'CreateInstanceAccount' with better checking logic

* add test for create instance account
2021-11-17 13:40:09 +01:00
tobi
aee1932bd4
First db migration (#297)
* store first db schema

* md linting *screams*
2021-11-13 17:30:01 +01:00
tobi
09ef9e639e
move to ssb gofed fork (#298) 2021-11-13 17:29:43 +01:00
tobi
829a934d23
update dependencies (#296) 2021-11-13 12:29:08 +01:00
tobi
2aaec82732
smtp + email confirmation (#285)
* add smtp configuration

* add email confirm + reset templates

* add email sender to testrig

* flesh out the email sender interface

* go fmt

* golint

* update from field with more clarity

* tidy up the email formatting

* fix tests

* add email sender to processor

* tidy client api processing a bit

* further tidying in fromClientAPI

* pin new account to user

* send msg to processor on new account creation

* generate confirm email uri

* remove emailer from account processor again

* add processCreateAccountFromClientAPI

* move emailer accountprocessor => userprocessor

* add email sender to user processor

* SendConfirmEmail function

* add noop email sender

* use noop email sender in tests

* only assemble message if callback is not nil

* use noop email sender if no smtp host is defined

* minify email html before sending

* fix wrong email address

* email confirm test

* fmt

* serve web hndler

* add email confirm handler

* init test log properly on testrig

* log emails that *would* have been sent

* go fmt ./...

* unexport confirm email handler

* updatedAt

* test confirm email function

* don't allow tokens older than 7 days

* change error message a bit

* add basic smtp docs

* add a few more snippets

* typo

* add email sender to outbox tests

* don't use dutch wikipedia link

* don't minify email html
2021-10-31 15:46:23 +01:00
tobi
4b1d9d3780
Serve outbox for Actor (#289)
* add statusesvisible convenience function

* add minID + onlyPublic to account statuses get

* move swagger collection stuff to common

* start working on Outbox GETting

* move functions into federationProcessor

* outboxToASCollection

* add statusesvisible convenience function

* add minID + onlyPublic to account statuses get

* move swagger collection stuff to common

* start working on Outbox GETting

* move functions into federationProcessor

* outboxToASCollection

* bit more work on outbox paging

* wrapNoteInCreate function

* test + hook up the processor functions

* don't do prev + next links on empty reply

* test get outbox through api

* don't fail on no status entries

* add outbox implementation doc

* typo
2021-10-24 11:57:39 +02:00
kim
7e4c3fa5c7
fix mention extracting when no domain exists (usually intra-instance mentions) (#272)
* fix mention extracting when no domain exists (usually when intra-instance mentions)

Signed-off-by: kim <grufwub@gmail.com>

* fix search logic to match new mention matching logic

Signed-off-by: kim <grufwub@gmail.com>

* appease the linter :p

Signed-off-by: kim <grufwub@gmail.com>
2021-10-17 14:19:49 +02:00
tobi
15621f5324
Follow request improvements (#282)
* tiny doc update

* add rejectfollowrequest to db

* add follow request reject to processor

* add reject handler

* tidy up follow request api

* tidy up federation call

* regenerate swagger docs

* api endpoint tests

* processor test

* add reject federatingdb handler

* start writing reject tests

* test reject follow request

* go fmt

* increase sleep for slow test setups

* more relaxed time.sleep
2021-10-16 13:27:43 +02:00
tobi
107685e22e
User password change (#280)
* start passwordChangeHandler

* add user scope

* add user module / api path

* add password change request

* make comment clearer

* add user to processor

* required true

* add processor call to handler

* don't pass tc or channel

* change password func + tests

* add some first docs about password management

* update swagger docs

* add api tests

* go fmt

* test fixes
2021-10-14 14:26:04 +02:00
tobi
a07e62e49e
go fmt (#278) 2021-10-13 13:50:24 +02:00
R. Aidan Campbell
083099a957
reference global logrus (#274)
* reference logrus' global logger instead of passing and storing a logger reference everywhere

* always directly use global logrus logger instead of referencing an instance

* test suites should also directly use the global logrus logger

* rename gin logging function to clarify that it's middleware

* correct comments which erroneously referenced removed logger parameter

* setting log level for tests now uses logrus' exported type instead of the string value, to guarantee error isn't possible
2021-10-11 14:37:33 +02:00
tobi
367bdca250
Handle forwarded messages (#273)
* correct path of foss_satan

* add APIri and notes

* test create forward note

* rename target => receiving account

* split up create into separate funcs

* update extractFromCtx

* tidy up from federator processing

* foss satan => http not https

* check if status in db

* mock dereference of status from IRI

* add forward message deref test

* update test with activities

* add remote_account_2 to test rig
2021-10-10 12:39:25 +02:00
tobi
3dc7644ae6
Derive visibility fixes (#271)
* use pub public const

* don't error on no summary

* move extract visibility to separate function

* extract visibility test

* add addressable interface
2021-10-06 18:18:02 +02:00
tobi
ff77bf1d11
fix logs not working properly (#264) 2021-10-04 16:00:35 +02:00
tobi
e04b187702
Refactor/tidy (#261)
* tidy up streaming

* cut down code duplication

* test get followers/following

* test streaming processor

* fix some test models

* add TimeMustParse

* fix uri / url typo

* make trace logging less verbose

* make logging more consistent

* disable quote on logging

* remove context.Background

* remove many extraneous mastodon references

* regenerate swagger

* don't log query on no rows result

* log latency first for easier reading
2021-10-04 15:24:19 +02:00
tobi
9ce4234b9f
Follow request auto approval (#259)
* start messing about

* fiddle more

* Tests & fiddling
2021-10-01 19:08:50 +02:00
tobi
365c3bf5d7
rework mention replacement func (#258) 2021-10-01 14:10:20 +02:00
tobi
28b6ce59d6
don't catch mentions within links (#257) 2021-09-30 18:11:57 +02:00
tobi
0cd2bd2960
allow dereferencing of groups (#256) 2021-09-30 12:27:42 +02:00
tobi
231075f28d
Golint (#255) 2021-09-30 11:16:23 +02:00
tobi
36a09dd0df
handle remote account deletion more systematically (#254) 2021-09-30 10:56:02 +02:00
tobi
08cb8a3385
skip account on error instead of returning error (#251)
* skip account on error instead of returning error

* still return error on a real error
2021-09-28 16:19:13 +02:00
tobi
b5a7e1ba32
Account update issue (#250)
* start poking around

* tests

* notes and fiddling
2021-09-28 15:21:59 +02:00
tobi
b3fd9c39a3
Weird notif issue (#248)
* start working on weird issue

* go fmt ./...

* more tests
2021-09-27 17:42:20 +02:00
tobi
c7cfbe2702
Unblock fix (#247)
* start tests for inbox posts

* go mod tidy

* rename transferContext

* test block/unblock

* improve logging

* improve logging

* fix comment typo
2021-09-24 17:56:48 +02:00
tobi
ddfd83d0fb
update media processing (#244) 2021-09-23 11:13:11 +02:00
tobi
7e6350b448
remove hardcoded friendica user agent block (#239) 2021-09-21 11:40:16 +02:00
tobi
ffc55e9b15
tweak db settings slightly (#238) 2021-09-20 18:20:21 +02:00
tobi
c7c9fff730
use timestamptz instead of timestamp (#237) 2021-09-20 17:41:52 +02:00
tobi
b315eee43b
Update webfingering a little, add tests (#236)
* Update webfingering a little, add tests

* fix broken tests oops
2021-09-20 16:46:45 +02:00
tobi
92186c8c6f
federated authentication better logging + tidying (#232)
* change trace logging in authenticator

* messing about

* lil changes

* go fmt

* error fix

* Fix broken test
2021-09-16 11:35:09 +02:00
tobi
2e5dcc2929
Fix mentions not notifying (#230)
* set default privacy for new accounts

* teshts

* found it

* tiny change

* aaaa
2021-09-14 12:23:56 +02:00
tobi
69bb56368d
set default privacy for new accounts (#225) 2021-09-13 17:35:33 +02:00
f0x52
026674bc2c
Thread views on the web (#207)
* Webviews for status threads

* fix up templates

* add ForkAwesome and gotosocial-styling into repo

* clean up gotosocial-styling, old styling

* update CONTRIBUTING with new css building, and nodemon recommendation

* update Dockerfile with new css bundling

* those weren't supposed to make it in

* upgrade gotosocial-styling deps

* update authorize template with main wrapper

* update css pipeline

* abstract status from thread to avoid copy-pasting

* basic CW implementation

* fix PR review suggestions

* fix no-image-desc icon alignment

* remove template loading println

* remove println

* remove changes to testmodels

* reset changes to testmodels
2021-09-13 14:45:33 +02:00
kim (grufwub)
6db57d1487 fix keys used to access storage items
Signed-off-by: kim (grufwub) <grufwub@gmail.com>
2021-09-11 20:28:57 +01:00
kim (grufwub)
7f36688643 rename KVStore uses as storage to differentiate between upcoming store package
Signed-off-by: kim (grufwub) <grufwub@gmail.com>
2021-09-11 20:18:06 +01:00
kim (grufwub)
e43a46e982 add git.iim.gay/grufwub/go-store for storage backend, replacing blob.Storage
Signed-off-by: kim (grufwub) <grufwub@gmail.com>
2021-09-11 20:12:47 +01:00
tobi
793f9af6e6
only store statuses in the AS cache, not accounts (#210) 2021-09-11 14:16:00 +02:00
tobi
9dc2255a8f
kim is a reply guy (#208)
* bun debug

* bun trace logging hooks

* more tests

* fix up some stuffffff

* drop the frontend cache until a proper fix is made

* go fmt
2021-09-11 13:19:06 +02:00
tobi
e681aac589
fixes + db changes (#204)
* fixes + db changes

* make duration more lenient
2021-09-10 14:36:10 +02:00
kim
446dbb7a72
remove boosted statuses from public (federated timeline) (#201)
Signed-off-by: kim (grufwub) <grufwub@gmail.com>
2021-09-10 10:09:15 +02:00
kim
4c4a622088
Prune unnecessary nullzeros, fixup db tags (#200)
* prune gtsmodel.Account bun tags, add note to gtsmodel dir

Signed-off-by: kim (grufwub) <grufwub@gmail.com>

* further database tag fixes

Signed-off-by: kim (grufwub) <grufwub@gmail.com>

* more db tag fixups

Signed-off-by: kim (grufwub) <grufwub@gmail.com>

* fix removing nullzero for account timestamps...

Signed-off-by: kim (grufwub) <grufwub@gmail.com>

* add nullzero back to accountid tag

Signed-off-by: kim (grufwub) <grufwub@gmail.com>

* rename gtsmodel readme

Signed-off-by: kim (grufwub) <grufwub@gmail.com>
2021-09-10 10:08:21 +02:00
kim
41ace19e0c
fix up status inreplyto visibility, + small format improvements (#199)
Signed-off-by: kim (grufwub) <grufwub@gmail.com>
2021-09-10 08:37:28 +02:00
tobi
555ea8edfb
Import export (#194)
* start with export/import code

* messing about with decoding/encoding

* some more fiddling

* stuff is WORKING

* working pretty alright!

* go fmt

* fix up tests, add docs

* start backup/restore doc

* tweaks

* credits

* update advancedVisibility settings

* update bun library -> v1.0.4

Signed-off-by: kim (grufwub) <grufwub@gmail.com>

* update oauth library -> v4.3.1-SSB

Signed-off-by: kim (grufwub) <grufwub@gmail.com>

* handle oauth token scope, fix user.SigninCount + token.UserID

Signed-off-by: kim (grufwub) <grufwub@gmail.com>

* update oauth library --> v4.3.2-SSB

Signed-off-by: kim (grufwub) <grufwub@gmail.com>

* update sqlite library -> v1.13.0

Signed-off-by: kim (grufwub) <grufwub@gmail.com>

* review changes

* start with export/import code

* messing about with decoding/encoding

* some more fiddling

* stuff is WORKING

* working pretty alright!

* go fmt

* fix up tests, add docs

* start backup/restore doc

* tweaks

* credits

* update advancedVisibility settings

* review changes

Co-authored-by: kim (grufwub) <grufwub@gmail.com>
Co-authored-by: kim <89579420+NyaaaWhatsUpDoc@users.noreply.github.com>
2021-09-09 16:15:25 +02:00
kim
a027da0ac9
Merge pull request #198 from NyaaaWhatsUpDoc/update/sqlite-library
Update/sqlite library
2021-09-08 21:26:32 +01:00
kim (grufwub)
71a4f8667c update sqlite library -> v1.13.0
Signed-off-by: kim (grufwub) <grufwub@gmail.com>
2021-09-08 21:12:23 +01:00
kim (grufwub)
d2a85f2f5f handle oauth token scope, fix user.SigninCount + token.UserID
Signed-off-by: kim (grufwub) <grufwub@gmail.com>
2021-09-08 20:46:08 +01:00
tobi
2b14b20802
rework media processing a little bit (#191)
* rework media processing a little bit

* review changes
2021-09-04 14:02:01 +02:00
tobi
ff05046df7
tests + announce notification fix (#193) 2021-09-04 13:29:56 +02:00
kim
25edd57eaf
Merge pull request #186 from superseriousbusiness/struct_validation
Struct validation
2021-09-03 10:27:40 +01:00
tsmethurst
32fe2282c4 review changes 2021-09-03 11:12:19 +02:00
tsmethurst
a116676814 Review changes 2021-09-03 10:30:40 +02:00
tsmethurst
5d5327614d lint 2021-09-02 12:24:18 +02:00
tsmethurst
cc7a35ee92 token tests 2021-09-02 12:22:09 +02:00
tsmethurst
42892f91e1 client_test 2021-09-02 12:17:14 +02:00
tsmethurst
6f7da0ecdd account + application tests 2021-09-02 12:10:59 +02:00
tobi
ac7c5e8cd2
session name fix (#185) 2021-09-01 22:12:31 +02:00
tsmethurst
4696e1a7b3 moving stuff around 2021-09-01 18:29:25 +02:00
tsmethurst
684bd56528 move oauth models into gtsmodel 2021-09-01 11:45:01 +02:00
tsmethurst
55a9b8553f lil move 2021-09-01 11:13:01 +02:00
tsmethurst
7b01304dac more updates 2021-09-01 11:12:10 +02:00
tsmethurst
2786b5f887 change muchos things 2021-09-01 11:11:26 +02:00
tsmethurst
fffff93b75 go fmt 2021-09-01 11:11:26 +02:00
tsmethurst
356d28fef9 more work on struct validation 2021-09-01 11:11:26 +02:00
tsmethurst
dc2e1bf9ab more work on struct validation 2021-09-01 11:11:26 +02:00
tsmethurst
8ba816cd83 statusmute annotations 2021-09-01 11:11:26 +02:00
tsmethurst
d2276fc553 start working on struct validation for gtsmodel 2021-09-01 11:11:26 +02:00
kim
7d193de25f
Improve GetRemoteStatus and db.GetStatus() logic (#174)
* only fetch status parents / children if explicity requested when dereferencing

Signed-off-by: kim (grufwub) <grufwub@gmail.com>

* Remove recursive DB GetStatus logic, don't fetch parent unless requested

Signed-off-by: kim (grufwub) <grufwub@gmail.com>

* StatusCache copies status so there are no thread-safety issues with modified status objects

Signed-off-by: kim (grufwub) <grufwub@gmail.com>

* remove sqlite test files

Signed-off-by: kim (grufwub) <grufwub@gmail.com>

* fix bugs introduced by previous commit

Signed-off-by: kim (grufwub) <grufwub@gmail.com>

* fix not continue on error in loop

Signed-off-by: kim (grufwub) <grufwub@gmail.com>

* use our own RunInTx implementation (possible fix for nested tx error)

Signed-off-by: kim (grufwub) <grufwub@gmail.com>

* fix cast statement to work with SQLite

Signed-off-by: kim (grufwub) <grufwub@gmail.com>

* be less strict about valid status in cache

Signed-off-by: kim (grufwub) <grufwub@gmail.com>

* add cache=shared ALWAYS for SQLite db instances

Signed-off-by: kim (grufwub) <grufwub@gmail.com>

* Fix EnrichRemoteAccount when updating account fails

Signed-off-by: kim (grufwub) <grufwub@gmail.com>

* add nolint tag

Signed-off-by: kim (grufwub) <grufwub@gmail.com>

* ensure file: prefixes the filename in sqlite addr

Signed-off-by: kim (grufwub) <grufwub@gmail.com>

* add an account cache, add status author account from db

Signed-off-by: kim (grufwub) <grufwub@gmail.com>

* Fix incompatible SQLite query

Signed-off-by: kim (grufwub) <grufwub@gmail.com>

* *actually* use the new getAccount() function in accountsDB

Signed-off-by: kim (grufwub) <grufwub@gmail.com>

* update cache tests to use test suite

Signed-off-by: kim (grufwub) <grufwub@gmail.com>

* add RelationshipTestSuite, add tests for methods with changed SQL

Signed-off-by: kim (grufwub) <grufwub@gmail.com>
2021-09-01 11:08:21 +02:00
kim
ed46224573
Add SQLite support, fix un-thread-safe DB caches, small performance f… (#172)
* Add SQLite support, fix un-thread-safe DB caches, small performance fixes

Signed-off-by: kim (grufwub) <grufwub@gmail.com>

* add SQLite licenses to README

Signed-off-by: kim (grufwub) <grufwub@gmail.com>

* appease the linter, and fix my dumbass-ery

Signed-off-by: kim (grufwub) <grufwub@gmail.com>

* make requested changes

Signed-off-by: kim (grufwub) <grufwub@gmail.com>

* add back comment

Signed-off-by: kim (grufwub) <grufwub@gmail.com>
2021-08-29 16:41:41 +02:00
tobi
53507ac2a3
Mention fixup (#167)
* rework mention creation a bit

* rework mention creation a bit

* tidy up status dereferencing

* start adding tests for dereferencing

* fixups

* fix

* review changes
2021-08-29 12:03:08 +02:00
kim
79afcdba3f
Replace federating DB locks map, add a cleanup goroutine (#166)
Signed-off-by: kim (grufwub) <grufwub@gmail.com>
2021-08-27 21:14:20 +02:00
tobi
add6eb6e2b
Gin logging (#165)
* start messing around with logger

* more messing about

* fix error with instance not created on startup (#156)

* set logger

* log user agent

* finalize logging

* go fmt

* start messing around with logger

* more messing about

* set logger

* log user agent

* finalize logging

* go fmt
2021-08-27 13:26:45 +02:00
kim
87328b6218
fix new tx within tx (#163)
Signed-off-by: kim (grufwub) <grufwub@gmail.com>
2021-08-27 11:38:24 +02:00
tobi
30f688dbe2
some more little fixes for all to enjoy (#161) 2021-08-26 22:06:34 +02:00
tobi
2162f21636
fix broken db queries in auth (#160) 2021-08-26 19:56:40 +02:00
tobi
1582cf8bad
fix null returned session (#159) 2021-08-26 18:55:39 +02:00
tobi
2eefaa0227
oops (#157) 2021-08-26 17:46:24 +02:00
tobi
14ebc94fd9
fix error with instance not created on startup (#156) 2021-08-26 17:22:41 +02:00
tobi
ddc120d5e6
fix public timeline bug (#150) 2021-08-26 11:28:16 +02:00
tobi
2dc9fc1626
Pg to bun (#148)
* start moving to bun

* changing more stuff

* more

* and yet more

* tests passing

* seems stable now

* more big changes

* small fix

* little fixes
2021-08-25 15:34:33 +02:00
tobi
071eca20ce
Manually approves followers (#146)
* update go-fed

* update go-fed

* manuallyapprovesfollowers

* serialize manuallyApprovesFollowers
2021-08-23 12:46:05 +02:00
tobi
4920229a3b
Database updates (#144)
* start moving some database stuff around

* continue moving db stuff around

* more fiddling

* more updates

* and some more

* and yet more

* i broke SOMETHING but what, it's a mystery

* tidy up

* vendor ttlcache

* use ttlcache

* fix up some tests

* rename some stuff

* little reminder

* some more updates
2021-08-20 12:26:56 +02:00
Tobi Smethurst
ce190d867c
Text/status parsing fixes (#141)
* aaaaaa

* vendor minify

* update + test markdown parsing
2021-08-16 19:17:56 +02:00
Tobi Smethurst
ff406be68f
Timeline loop fix (#140)
* uwu we made a fucky wucky

* uwu we made a fucky wucky

* work on timeline fixes a little

* fiddle with tests some more

* bleep bloop more tests

* more tests

* update drone yml

* update some sturf

* make the timeline code a bit lazier

* go fmt

* fix drone.yml
2021-08-15 18:43:08 +02:00
Tobi Smethurst
98263a7de6
Grand test fixup (#138)
* start fixing up tests

* fix up tests + automate with drone

* fiddle with linting

* messing about with drone.yml

* some more fiddling

* hmmm

* add cache

* add vendor directory

* verbose

* ci updates

* update some little things

* update sig
2021-08-12 21:03:24 +02:00
Tobi Smethurst
329a5e8144
Text duplication fix (#137)
* start testing text duplication

* tests

* fixes + tests
2021-08-11 16:54:54 +02:00