Commit graph

68 commits

Author SHA1 Message Date
tobi
de4d75246c
[chore] Update pruneOneLocal to use ctx (#684)
* update pruneOneLocal to use ctx

* update test
2022-07-03 12:18:59 +02:00
Dominik Süß
9d0df426da
[feature] S3 support (#674)
* feat: vendor minio client

* feat: introduce storage package with s3 support

* feat: serve s3 files directly

this saves a lot of bandwith as the files are fetched from the object
store directly

* fix: use explicit local storage in tests

* feat: integrate s3 storage with the main server

* fix: add s3 config to cli tests

* docs: explicitly set values in example config

also adds license header to the storage package

* fix: use better http status code on s3 redirect

HTTP 302 Found is the best fit, as it signifies that the resource
requested was found but not under its presumed URL

307/TemporaryRedirect would mean that this resource is usually located
here, not in this case

303/SeeOther indicates that the redirection does not link to the
requested resource but to another page

* refactor: use context in storage driver interface
2022-07-03 12:08:30 +02:00
tobi
9e7d022a06
[feature] Cleanup unattached local media (#680)
* add localUnattached db function

* add parseOlderThan util function

* add pruneunusedlocalattachments to media manager

* add unusedlocal pruning to schedule + admin call

* set number of days to keep as a const

* fix test
2022-06-30 12:22:10 +02:00
tobi
68736efd20
[feature] add configuration to /api/v1/instance response (#670)
* add configuration object to api instance model

* regenerate swagger docs

* add func to return all supported mimes for media

* add instance configuration to api serialization

* fix json tags

* update instance endpoint tests

* fix typeutils tests

* final regen of swagger docs

* omitempty instance configuration
2022-06-26 10:58:45 +02:00
kim
43ac0cdb9c
[chore] Global server configuration overhaul (#575)
* move config flag names and usage to config package, rewrite config package to use global Configuration{} struct

Signed-off-by: kim <grufwub@gmail.com>

* improved code comment

Signed-off-by: kim <grufwub@gmail.com>

* linter

Signed-off-by: kim <grufwub@gmail.com>

* fix unmarshaling

Signed-off-by: kim <grufwub@gmail.com>

* remove kim's custom go compiler changes

Signed-off-by: kim <grufwub@gmail.com>

* generate setter and flag-name functions, implement these in codebase

Signed-off-by: kim <grufwub@gmail.com>

* update deps

Signed-off-by: kim <grufwub@gmail.com>

* small change

Signed-off-by: kim <grufwub@gmail.com>

* appease the linter...

Signed-off-by: kim <grufwub@gmail.com>

* move configuration into ConfigState structure, ensure reloading to/from viper settings to keep in sync

Signed-off-by: kim <grufwub@gmail.com>

* lint

Signed-off-by: kim <grufwub@gmail.com>

* update code comments

Signed-off-by: kim <grufwub@gmail.com>

* fix merge issue

Signed-off-by: kim <grufwub@gmail.com>

* fix merge issue

Signed-off-by: kim <grufwub@gmail.com>

* improved version string (removes time + go version)

Signed-off-by: kim <grufwub@gmail.com>

* fix version string build to pass test script + consolidate logic in func

Signed-off-by: kim <grufwub@gmail.com>

* add license text, update config.Defaults comment

Signed-off-by: kim <grufwub@gmail.com>

* add license text to generated config helpers file

Signed-off-by: kim <grufwub@gmail.com>

* defer unlock on config.Set___(), to ensure unlocked on panic

Signed-off-by: kim <grufwub@gmail.com>

* make it more obvious which cmd flags are being attached

Signed-off-by: kim <grufwub@gmail.com>
2022-05-30 14:41:24 +02:00
tobi
b143877995
[feature] Unused avatar and header cleanup (#574)
* rename + tidy up remote pruning

* fix media attachment account join
see https://bun.uptrace.dev/guide/golang-orm.html#table-relationships

* update logging to new function name

* add get avatars and headers to bun

* add pruneallmeta function

* don't set uncached since we're deleting anyway

* fix totalPruned being in wrong place

* test pruning meta

* go fmt ./...

* rename mediaprune

* add meta pruning to routine mediaprune

* tidy up cleanup job scheduling

* rename adminmediaremoteprune

* update mediacleanup to use renamed prune func

* update swagger docs a little bit

* reuse cancel + context
2022-05-15 15:45:04 +01:00
Sashanoraa
6e947ff266
[feature] Media cleanup endpoint (#560)
Adds an admin endpoint to trigger a remote media cleanup.

Fixed #348

Signed-off-by: Sashanoraa <sasha@noraa.gay>
2022-05-15 14:52:46 +02:00
kim
223025fc27
[security] transport.Controller{} and transport.Transport{} security and performance improvements (#564)
* cache transports in controller by privkey-generated pubkey, add retry logic to transport requests

Signed-off-by: kim <grufwub@gmail.com>

* update code comments, defer mutex unlocks

Signed-off-by: kim <grufwub@gmail.com>

* add count to 'performing request' log message

Signed-off-by: kim <grufwub@gmail.com>

* reduce repeated conversions of same url.URL object

Signed-off-by: kim <grufwub@gmail.com>

* move worker.Worker to concurrency subpackage, add WorkQueue type, limit transport http client use by WorkQueue

Signed-off-by: kim <grufwub@gmail.com>

* fix security advisories regarding max outgoing conns, max rsp body size

- implemented by a new httpclient.Client{} that wraps an underlying
  client with a queue to limit connections, and limit reader wrapping
  a response body with a configured maximum size
- update pub.HttpClient args passed around to be this new httpclient.Client{}

Signed-off-by: kim <grufwub@gmail.com>

* add httpclient tests, move ip validation to separate package + change mechanism

Signed-off-by: kim <grufwub@gmail.com>

* fix merge conflicts

Signed-off-by: kim <grufwub@gmail.com>

* use singular mutex in transport rather than separate signer mus

Signed-off-by: kim <grufwub@gmail.com>

* improved useragent string

Signed-off-by: kim <grufwub@gmail.com>

* add note regarding missing test

Signed-off-by: kim <grufwub@gmail.com>

* remove useragent field from transport (instead store in controller)

Signed-off-by: kim <grufwub@gmail.com>

* shutup linter

Signed-off-by: kim <grufwub@gmail.com>

* reset other signing headers on each loop iteration

Signed-off-by: kim <grufwub@gmail.com>

* respect request ctx during retry-backoff sleep period

Signed-off-by: kim <grufwub@gmail.com>

* use external pkg with docs explaining performance "hack"

Signed-off-by: kim <grufwub@gmail.com>

* use http package constants instead of string method literals

Signed-off-by: kim <grufwub@gmail.com>

* add license file headers

Signed-off-by: kim <grufwub@gmail.com>

* update code comment to match new func names

Signed-off-by: kim <grufwub@gmail.com>

* updates to user-agent string

Signed-off-by: kim <grufwub@gmail.com>

* update signed testrig models to fit with new transport logic (instead uses separate signer now)

Signed-off-by: kim <grufwub@gmail.com>

* fuck you linter

Signed-off-by: kim <grufwub@gmail.com>
2022-05-15 11:16:43 +02:00
tobi
5004e0a9da
[bugfix] Fix remote media pruning failing if media already gone (#548)
* fix error check of prune to allow missing files

* update go-store library, add test for pruning item with db entry but no file

Signed-off-by: kim <grufwub@gmail.com>

* remove now-unneccessary error check

Signed-off-by: kim <grufwub@gmail.com>

Co-authored-by: kim <grufwub@gmail.com>
2022-05-08 18:49:45 +01:00
kim
a561ef3541
[feature] Update media manager to use internal/worker package (#543)
* update media manager to use internal/worker package, update worker with better logging

Signed-off-by: kim <grufwub@gmail.com>

* fix Queue() trace log message format operators

Signed-off-by: kim <grufwub@gmail.com>

* update media manager comment to match updated worker implementation

Signed-off-by: kim <grufwub@gmail.com>
2022-05-07 17:36:01 +02:00
tobi
d7b46a4b63
[chore] Move initial MediaRemotePrune to cmd.server function (#536)
* move initial prune up to the server start function

* don't do initial prune when starting media manager

* add admin/mediaremoteprune in processor
2022-05-03 10:56:18 +01:00
kim
e09958e7f1
[feature] Run initial cache cleanup on gotosocial start in case max-age changed (#533)
Signed-off-by: kim <grufwub@gmail.com>
2022-05-03 11:05:22 +02:00
tobi
eeb78bd141
[bugfix] Allow processing of .png files where checksum is not correct (#487)
* add png stripping code from google/wuffs

* experiment with stripping data from pngs

* add test images

* use StrippedPngDecode for pngs

* add StrippedPngDecode func

* update tests for (no)alphachannel pngs

* nolint on copied function
2022-04-25 14:45:44 +02:00
0hlov3
d5fe729cf4
[chore] Add two new test images for media processing (#424) 2022-04-24 14:49:37 +02:00
tobi
03d7c75ebf
[chore] Update Go version to 1.18 (#444)
* linting with new golangci-lint version

* update go to 1.18

* bump versions in drone.yml

* use new runtime/debug package for version info

* remove Commit build flag from goreleaser

* remove mock commit + version from build script

* go fmt

* add dummy version env flag to test container

* install git in golang container for testing

* only set versionString if Version is defined
2022-04-02 15:40:09 +02:00
tobi
8cc70a14d6
[bugfix] Fix media uploads blocking after a failed upload #438
safer in case of failure
2022-03-22 12:42:34 +01:00
tobi
25cab0e1f4
[bugfix] Fix images not being processed correctly sometimes (#437)
* bump exif-terminator to latest version

* add and test giant turnip from turnip.farm

* don't error if content property is nil
2022-03-21 19:46:51 +01:00
tobi
73e9cca701
[bugfix] Close ReadClosers properly in the media package (#434)
* defer lock reader

* close readers when finished with them

* close the reader in the teereader when finished
2022-03-21 13:41:44 +01:00
tobi
07727753b9
[feature] Clean up/uncache remote media (#407)
* Add whereNotEmptyAndNotNull

* Add GetRemoteOlderThanDays

* Add GetRemoteOlderThanDays

* Add PruneRemote to Manager interface

* Start implementing PruneRemote

* add new attachment + status to tests

* fix up and test GetRemoteOlderThan

* fix bad import

* PruneRemote: return number pruned

* add Cached column to mediaattachment

* update + test pruneRemote

* update mediaTest

* use Cached column

* upstep bun to latest version

* embed structs in mediaAttachment

* migrate mediaAttachment to new format

* don't default cached to true

* select only remote media

* update db dependencies

* step bun back to last working version

* update pruneRemote to use Cached field

* fix storage path of test attachments

* add recache logic to manager

* fix trimmed aspect ratio

* test prune and recache

* return errwithcode

* tidy up different paths for emoji vs attachment

* fix incorrect thumbnail type being stored

* expose TransportController to media processor

* implement tee-ing recached content

* add thoughts of dog to test fedi attachments

* test get remote files

* add comment on PruneRemote

* add postData cleanup to recache

* test thumbnail fetching

* add incredible diagram

* go mod tidy

* buffer pipes for recache streaming

* test for client stops reading after 1kb

* add media-remote-cache-days to config

* add cron package

* wrap logrus so it's available to cron

* start and stop cron jobs gracefully
2022-03-07 11:08:26 +01:00
tobi
55b83bea7c
[feature] Add postDataCallbackFunc to allow cleanup (#408) 2022-02-22 13:50:33 +01:00
tobi
15d1e6b3a1
[bugfix] Fix thumbnail image type (#406)
* fix thumbnail content-type

* test fix thumbnail content-type
2022-02-21 11:26:26 +01:00
tsmethurst
8c0141d103 store and retrieve processState atomically 2022-02-08 13:38:44 +01:00
tsmethurst
d9a778a795 put store lock file in base of storage 2022-01-29 12:16:12 +01:00
tsmethurst
0e7f24ff26 test with disk storage as well 2022-01-24 18:11:27 +01:00
tsmethurst
3c1eb155e4 add file size checks 2022-01-23 15:52:18 +01:00
tsmethurst
c157b1b20b rework data function to provide filesize 2022-01-23 14:41:58 +01:00
tsmethurst
589bb9df02 pass reader around instead of []byte 2022-01-16 18:52:55 +01:00
tsmethurst
723bfe8944 lint, fmt 2022-01-15 17:41:18 +01:00
tsmethurst
6bf39d0fc1 emoji code passing muster 2022-01-15 17:36:15 +01:00
tsmethurst
c4a533db72 start fixing up emoji processing code 2022-01-15 14:33:58 +01:00
tsmethurst
33ca5513ad pin instanceAccountID to in-process emoji 2022-01-11 17:51:45 +01:00
tsmethurst
113f9d9ab4 pass a function into the manager, start work on emoji 2022-01-11 17:49:14 +01:00
tsmethurst
e0f9323b9a test the media manager a bit, add shutdown logic 2022-01-10 18:36:09 +01:00
tsmethurst
dccf21dd87 tests are passing, but there's still much to be done 2022-01-09 18:41:22 +01:00
tsmethurst
f61c3ddcf7 compiling now 2022-01-08 17:17:01 +01:00
tsmethurst
c2ff8f392b further refinements 2022-01-08 13:45:42 +01:00
tsmethurst
7ebe0f6a15 start working on thumb + full funcs 2022-01-04 17:37:54 +01:00
tsmethurst
8abfa7751a return very partial image on first upload 2022-01-03 17:37:38 +01:00
tsmethurst
2f57eb5ece fiddle around with workers 2022-01-02 15:00:53 +01:00
tsmethurst
c4d63d125b more refactoring, media handler => manager 2021-12-28 16:36:00 +01:00
tsmethurst
6803c1682b start refactor of media package 2021-12-27 18:03:36 +01:00
tobi
ef5a9256a8
Extend license notices to 2022 (#354) 2021-12-20 18:42:19 +01:00
tobi
cb8688f429
Remove unnecessary storage config variables (#344)
* rewire config to not use extraneous serve vars

* rename 'file' to 'local' for consistency

* use Type and Size again
2021-12-20 15:19:53 +01:00
tobi
c111b239f7
Add optional syslog logrus hook (#343)
* add optional syslog logrus hook

* document syslog
2021-12-12 18:00:20 +01:00
tobi
0884f89431
Implement Cobra CLI tooling, Viper config tooling (#336)
* start pulling out + replacing urfave and config

* replace many many instances of config

* move more stuff => viper

* properly remove urfave

* move some flags to root command

* add testrig commands to root

* alias config file keys

* start adding cli parsing tests

* reorder viper init

* remove config path alias

* fmt

* change config file keys to non-nested

* we're more or less in business now

* tidy up the common func

* go fmt

* get tests passing again

* add note about the cliparsing tests

* reorganize

* update docs with changes

* structure cmd dir better

* rename + move some files around

* fix dangling comma
2021-12-07 13:31:39 +01:00
tobi
829a934d23
update dependencies (#296) 2021-11-13 12:29:08 +01:00
R. Aidan Campbell
083099a957
reference global logrus (#274)
* reference logrus' global logger instead of passing and storing a logger reference everywhere

* always directly use global logrus logger instead of referencing an instance

* test suites should also directly use the global logrus logger

* rename gin logging function to clarify that it's middleware

* correct comments which erroneously referenced removed logger parameter

* setting log level for tests now uses logrus' exported type instead of the string value, to guarantee error isn't possible
2021-10-11 14:37:33 +02:00
tobi
e04b187702
Refactor/tidy (#261)
* tidy up streaming

* cut down code duplication

* test get followers/following

* test streaming processor

* fix some test models

* add TimeMustParse

* fix uri / url typo

* make trace logging less verbose

* make logging more consistent

* disable quote on logging

* remove context.Background

* remove many extraneous mastodon references

* regenerate swagger

* don't log query on no rows result

* log latency first for easier reading
2021-10-04 15:24:19 +02:00
tobi
ddfd83d0fb
update media processing (#244) 2021-09-23 11:13:11 +02:00
kim (grufwub)
6db57d1487 fix keys used to access storage items
Signed-off-by: kim (grufwub) <grufwub@gmail.com>
2021-09-11 20:28:57 +01:00