Commit graph

2030 commits

Author SHA1 Message Date
tobi
42932f9820
[chore] replace UniqueStrings with Deduplicate (#3154) 2024-07-30 12:20:06 +02:00
kim
47c26818d6
[chore] add some more slice related utility functions + remove duplicated functions (#3149) 2024-07-30 11:29:32 +02:00
Vyr Cossont
a237e2b295
[feature] Implement following hashtags (#3141)
* Implement followed tags API

* Insert statuses with followed tags into home timelines

* Test following and unfollowing tags

* Correct Swagger path params

* Trim conversation caches

* Migration for followed_tags table

* Followed tag caches and DB implementation

* Lint and tests

* Add missing tag info endpoint, reorganize tag API

* Unwrap boosts when timelining based on tags

* Apply visibility filters to tag followers

* Address review comments
2024-07-29 19:26:31 +01:00
kim
368c97f0f8
[bugfix] take into account rotation when generating thumbnail (#3147)
* take into account rotation when generating thumbnail, simplify ffprobe output to show only fields we need

* only show rotation side data

* remove unnecessary comment

* fix code comments

* remove debug logging
2024-07-28 21:10:41 +02:00
kim
58f8082795
[bugfix] moves file rename to earlier in media pipeline so ffmpeg calls ALWAYS have extension (#3146) 2024-07-28 10:31:49 +02:00
tobi
87ee64afa0
[chore] log delivery worker stop/start at debug level (#3145) 2024-07-27 13:43:18 +02:00
kim
ecc114fc00
[chore] replaces nested ifs with switch cases, removes defer 'onFail()' being passed to funcs (#3143) 2024-07-27 11:09:02 +02:00
kim
d9e70b942f
ensure ffmpeg muxer gets set correctly by setting input file extension before clean (#3142) 2024-07-26 16:30:24 +01:00
tobi
ecfea10e35
[bugfix] Use punycode for host part of resource query param when doing webfinger requests (#3133)
* [bugfix] use punycode when webfingering

* account for punycode when checking if final URI matches expected

* hmm

* fix test
2024-07-26 13:11:07 +02:00
tobi
8ab2b19a94
[feature] Federate interaction policies + Accepts; enforce policies (#3138)
* [feature] Federate interaction policies + Accepts; enforce policies

* use Acceptable type

* fix index

* remove appendIRIStrs

* add GetAccept federatingdb function

* lock on object IRI
2024-07-26 12:04:28 +02:00
tobi
f8d399cf6a
[docs] Add separate migration section + instructions for moving to GtS and not just away (#3139)
* [docs] Add separate migration section + instructions for moving to GtS and not just away

* update headings
2024-07-24 19:07:36 +02:00
tobi
c9b6220fef
[chore] Add interaction filter to complement existing visibility filter (#3111)
* [chore] Add interaction filter to complement existing visibility filter

* pass in ptr to visibility and interaction filters to Processor{} to ensure shared

* use int constants for for match type, cache db calls in filterctx

* function name typo 😇

---------

Co-authored-by: kim <grufwub@gmail.com>
2024-07-24 12:27:42 +01:00
kim
63fc9b6c3e
[chore] renames the GTS caches to DB caches (#3127)
* renames the `GTS` caches to `DB` caches, as it better references what they are

* change remaining Caches.GTS uses to Caches.DB
2024-07-24 10:41:43 +02:00
tobi
325b4a2b4a
[feature] Allow user to set "bot" flag; show bot icon on profile (#3135)
* [feature] Allow user to set "bot" flag; show bot icon on profile

* tweak

* update customs
2024-07-24 10:40:56 +02:00
Vyr Cossont
db0a47126e
[bugfix] Serialize empty conversation account list as empty list, not null (#3137) 2024-07-24 10:40:01 +02:00
tobi
6533531cf1
[chore] Add some log lines to recent migrations warning not to interrupt (#3134)
* [chore] Add some log lines to recent migrations warning not to interrupt

* arse
2024-07-23 23:06:20 +02:00
Vyr Cossont
86a59db711
Remove content and related fields from boosts (#3131)
These duplicate the content of the target and aren't necessary for anything.

- Stops copying some fields from target when boosting or processing a remote boost
- Adds a migration to null out existing duplicate data
- Updates tests
2024-07-23 20:51:57 +01:00
Vyr Cossont
2d921d9d7c
Explicitly propagate filter results from statuses to their boosts in API responses (#3130)
Related to #3128
2024-07-23 20:51:07 +01:00
Vyr Cossont
8fdd358f4b
[feature] Conversations API (#3013)
* Implement conversations API

* Sort and page conversations by last status ID

* Appease linter

* Fix deleting conversations and statuses

* Refactor to make migrations automatic

* Lint

* Update tests post-merge

* Fixes from live-fire testing

* Linter caught a format problem

* Refactor tests, fix cache

* Negative test for non-DMs

* Run conversations advanced migration on testrig startup as well as regular server startup

* Document (lack of) side effects of API method for deleting a conversation

* Make not-found check less nested for readability

* Rename PutConversation to UpsertConversation

* Use util.Ptr instead of IIFE

* Reduce cache used by conversations

* Remove unnecessary TableExpr/ColumnExpr

* Use struct tags for both unique constraints on Conversation

* Make it clear how paging with GetDirectStatusIDsBatch should be used

* Let conversation paging skip conversations it can't render

* Use Bun NewDropTable

* Convert delete raw query to Bun

* Convert update raw query to Bun

* Convert latestConversationStatusesTempTable raw query partially to Bun

* Convert conversationStatusesTempTable raw query partially to Bun

* Rename field used to store result of MaxDirectStatusID

* Move advanced migrations to their own tiny processor

* Catch up util function name with main

* Remove json.… wrappers

* Remove redundant check

* Combine error checks

* Replace map with slice of structs

* Address processor/type converter comments

- Add context info for errors
- Extract some common processor code into shared methods
- Move conversation eligibility check ahead of populating conversation

* Add error context when dropping temp tables
2024-07-23 20:44:31 +01:00
kim
31294f7c78
[bugfix] media.Processor{}.GetFile() returning 404s on first call, correctly loading on 2nd (#3129)
* refactor file handling a tiny bit

* whoops

* make processing media / emoji defers a bit clear to see that it's the "on finished processing" path

* some wording

* add some debug logging

* add mutex locks for processing remote media

* try removing freshness check

* fix derefMedia not being allocated

* fix log format string

* handle case of empty file paths (i.e. not stored)

* remove media / emoji once finished processing from dereferencer maps

* whoops, fix the cached / force checks

* move url parsing outside of 'process___Safely()' funcs to prevalidate url

* use emoji.ShortcodeDomain()

* update RefreshEmoji() to also match RefreshMedia() changes

---------

Co-authored-by: tobi <tobi.smethurst@protonmail.com>
2024-07-22 18:45:48 +01:00
dependabot[bot]
5338825d2b
[chore]: Bump github.com/minio/minio-go/v7 from 7.0.73 to 7.0.74 (#3125)
Bumps [github.com/minio/minio-go/v7](https://github.com/minio/minio-go) from 7.0.73 to 7.0.74.
- [Release notes](https://github.com/minio/minio-go/releases)
- [Commits](https://github.com/minio/minio-go/compare/v7.0.73...v7.0.74)

---
updated-dependencies:
- dependency-name: github.com/minio/minio-go/v7
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-07-22 14:12:51 +01:00
kim
c635080a0b
[feature] add flac support (#3121)
* add flac support to the ffprobe format/stream -> filetype parser

* also add audio/flac for flac (not just x-flac)

* update tests

---------

Co-authored-by: tobi <tobi.smethurst@protonmail.com>
2024-07-21 14:42:51 +02:00
tobi
027a93facc
[feature/frontend] Respect prefers-reduced-motion for avatars, headers, and emojis (#3118)
* [feature/frontend] Respect `prefers-reduced-motion` for avatars, headers, and emojis

* go fmt

* fix tests

* use static version of instance thumbnail when appropriate

* use prefers-reduced-motion

* simplify account conversion a bit

* fix c&p error
2024-07-21 14:22:08 +02:00
kim
b415337d40
[bugfix] update common get target account / status doing refresh async (#3124) 2024-07-21 14:04:19 +02:00
tobi
8a2fb48ef4
[chore] Remove duplicate <hr> (#3123)
* [chore] Remove duplicate `<hr>`

* update test
2024-07-21 11:30:22 +02:00
kim
409b3987ac
update go-ffmpreg to v0.2.3 (#3120) 2024-07-20 14:14:58 +01:00
tobi
ad700c5029
[chore/frontend] Use smaller webp images for logo and default avatars (#3119) 2024-07-20 14:02:22 +01:00
kim
50c9b5498b
[feature] use webp for thumbnails (#3116)
* update to use webp for thumbnails

* bump webp quality up to 40% from 12% (it's a bit different to jpeg quality setting)

* update to use yuva colorspace, and use thumbnail=n=10 to select frame

* fix missing comma in ffmpeg args

* add links to appropriate ffmpeg docs

* update tests

* add file size tests for thumbnails

---------

Co-authored-by: tobi <tobi.smethurst@protonmail.com>
2024-07-19 17:28:43 +02:00
kim
f5a44ccad4
[docs] update README with our corporate sponsorship policy (#3112)
* update README with our corporate sponsorship policy

* move location of corporate sponsorship

* tweaks after discussion

* update to maintain consistency
2024-07-19 14:14:07 +02:00
tobi
2b895f822a
[chore] Fix typo (#3113) 2024-07-17 22:04:52 +02:00
kim
72ba5666a6
[chore] media pipeline improvements (#3110)
* don't set emoji / media image paths on failed download, migrate FileType from string to integer

* fix incorrect uses of util.PtrOr, fix returned frontend media

* fix migration not setting arguments correctly in where clause

* fix not providing default with not null column

* whoops

* ensure a default gets set for media attachment file type

* remove the exclusive flag from writing files in disk storage

* rename PtrOr -> PtrOrZero, and rename PtrValueOr -> PtrOrValue to match

* slight wording changes

* use singular / plural word forms (no parentheses), is better for screen readers

* update testmodels with unknown media type to have unset file details, update attachment focus handling converting to frontend, update tests

* store first instance in ffmpeg wasm pool, fill remaining with closed instances
2024-07-17 15:26:33 +00:00
tobi
0aadc2db2a
[feature] Allow users to set default interaction policies per status visibility (#3108)
* [feature] Allow users to set default interaction policies

* use vars for default policies

* avoid some code repetition

* unfuck form binding

* avoid bonkers loop

* beep boop

* put policyValsToAPIPolicyVals in separate function

* don't bother with slices.Grow

* oops
2024-07-17 16:46:52 +02:00
kim
401098191b
give read-only access to /dev for ffmpeg to access /dev/urandom (#3109) 2024-07-16 13:32:48 +01:00
kim
de45c0be60
[feature] more filetype support! (#3107)
* add more supported file types to our media processor that ffmpeg supports, update supported mime type lists

* add code comments to the supported mime types slice

* don't check for zero value string, just parse

* remove some unneeded consts which make the code a bit harder to read

* fix test expected instance media mime types, use compact ffprobe json, simple media processing by type

* final tweaks to media processing code

* don't use safe divide where we don't need to
2024-07-15 15:24:53 +01:00
tobi
9efb11d848
[feature/frontend] Add player for audio files; use thumbnail for poster (#3099)
* [feature/frontend] Audio player for audio media types

* use video preview images for previews instead of video itself

* don't preload

* update tests for new zork status

* collapse media gallery into single row when small
2024-07-15 10:47:57 +01:00
dependabot[bot]
16421f7576
[chore]: Bump github.com/tdewolff/minify/v2 from 2.20.34 to 2.20.37 (#3106) 2024-07-15 09:15:31 +00:00
dependabot[bot]
b8a291c430
[chore]: Bump github.com/coreos/go-oidc/v3 from 3.10.0 to 3.11.0 (#3104) 2024-07-15 09:14:50 +00:00
tobi
9adcce9494
[chore] add step-by-step logging to long migration (#3102) 2024-07-15 11:12:56 +02:00
tobi
e2ed6ef024
[chore] Go mod tidy / vendor (#3100) 2024-07-13 16:23:25 +02:00
tobi
c83e96b8a7
[chore/frontend] Tweak threading a bit, inform about hidden replies (#3097)
* [chore/frontend] Tweak threading a bit, inform about hidden replies

* whoops

* round off bottom of replies col-header if no replies visible
2024-07-13 12:26:16 +02:00
tobi
bbbdf01213
[bugfix] Fix panic in isStatusHomeTimelineable (#3096) 2024-07-13 11:07:01 +02:00
tobi
aeb65bceae
[feature/frontend] Better visual separation between "main" thread and "replies" (#3093)
* [feature/frontend] Better web threading model

* fix test

* bwap

* tweaks

* more tweaks to wording

* typo

* indenting

* adjust wording

* aaa
2024-07-12 20:36:03 +02:00
kim
cde2fb6244
[feature] support processing of (many) more media types (#3090)
* initial work replacing our media decoding / encoding pipeline with ffprobe + ffmpeg

* specify the video codec to use when generating static image from emoji

* update go-storage library (fixes incompatibility after updating go-iotools)

* maintain image aspect ratio when generating a thumbnail for it

* update readme to show go-ffmpreg

* fix a bunch of media tests, move filesize checking to callers of media manager for more flexibility

* remove extra debug from error message

* fix up incorrect function signatures

* update PutFile to just use regular file copy, as changes are file is on separate partition

* fix remaining tests, remove some unneeded tests now we're working with ffmpeg/ffprobe

* update more tests, add more code comments

* add utilities to generate processed emoji / media outputs

* fix remaining tests

* add test for opus media file, add license header to utility cmds

* limit the number of concurrently available ffmpeg / ffprobe instances

* reduce number of instances

* further reduce number of instances

* fix envparsing test with configuration variables

* update docs and configuration with new media-{local,remote}-max-size variables
2024-07-12 09:39:47 +00:00
tobi
5bc567196b
[chore] Add interaction policy gtsmodels (#3075)
* [chore] introduce interaction policy gts models

* update migration a smidge

* fix copy paste typo

* update migration

* use int for InteractionType
2024-07-11 15:44:29 +01:00
snow
8f8093aea4
[bugfix] Don't throw error when parent statuses are missing (#2011) (#3088)
* [bugfix] Don't throw error when parent statuses are missing (#2011)

* Split missing parent status case from error check
2024-07-10 14:38:25 +01:00
Daenney
ad93e57d08
[choore] Update robots.txt (#3092)
Recategorises a pair of scrapers according to their use.
2024-07-10 15:10:34 +02:00
Daenney
1de41f64f2
[chore] Bump ncruces/go-sqlite3 to 0.17.1 (#3085)
More linkanme fixes.
2024-07-08 22:03:00 +02:00
tobi
9a7c8926f5
[chore/frontend] Put ActivityPub logo on index page (#3087)
* [chore/frontend] Put ActivityPub logo on index page

* tidy up

* embiggen the logo a bit

* use light logo instead

* ensmallen link container
2024-07-08 18:12:33 +02:00
tobi
d70f4e166d
[feature/frontend] Allow setting alt-text for avatar + header (#3086) 2024-07-08 15:47:03 +02:00
dependabot[bot]
43c480aec4
[chore]: Bump github.com/minio/minio-go/v7 from 7.0.72 to 7.0.73 (#3083) 2024-07-08 07:59:07 +00:00