Commit graph

649 commits

Author SHA1 Message Date
Mark Felder
ede414094f RichMedia refactor
Rich Media parsing was previously handled on-demand with a 2 second HTTP request timeout and retained only in Cachex. Every time a Pleroma instance is restarted it will have to request and parse the data for each status with a URL detected. When fetching a batch of statuses they were processed in parallel to attempt to keep the maximum latency at 2 seconds, but often resulted in a timeline appearing to hang during loading due to a URL that could not be successfully reached. URLs which had images links that expire (Amazon AWS) were parsed and inserted with a TTL to ensure the image link would not break.

Rich Media data is now cached in the database and fetched asynchronously. Cachex is used as a read-through cache. When the data becomes available we stream an update to the clients. If the result is returned quickly the experience is almost seamless. Activities were already processed for their Rich Media data during ingestion to warm the cache, so users should not normally encounter the asynchronous loading of the Rich Media data.

Implementation notes:

- The async worker is a Task with a globally unique process name to prevent duplicate processing of the same URL
- The Task will attempt to fetch the data 3 times with increasing sleep time between attempts
- The HTTP request obeys the default HTTP request timeout value instead of 2 seconds
- URLs that cannot be successfully parsed due to an unexpected error receives a negative cache entry for 15 minutes
- URLs that fail with an expected error will receive a negative cache with no TTL
- Activities that have no detected URLs insert a nil value in the Cachex :scrubber_cache so we do not repeat parsing the object content with Floki every time the activity is rendered
- Expiring image URLs are handled with an Oban job
- There is no automatic cleanup of the Rich Media data in the database, but it is safe to delete at any time
- The post draft/preview feature makes the URL processing synchronous so the rendered post preview will have an accurate rendering

Overall performance of timelines and creating new posts which contain URLs is greatly improved.
2024-05-07 19:54:56 -04:00
marcin mikołajczak
d415686bb9 Allow to group bookmarks in folders
Signed-off-by: marcin mikołajczak <git@mkljczk.pl>
2024-03-01 11:04:01 +01:00
Mark Felder
fb3eb6e0a4 Fix more Logger warn -> warning 2023-12-20 16:24:27 -05:00
lain
ee15939d33 Skip transaction to generate the index concurrently 2023-12-09 05:54:02 +00:00
Mark Felder
003d3312fd Permit the index creation to run concurrently 2023-12-08 15:48:00 -05:00
lain
35774d44b2 Merge branch 'generate-unset-user-keys-migration' into 'develop'
Fix GenerateUnsetUserKeys migration

See merge request pleroma/pleroma!3976
2023-11-28 12:34:32 +00:00
marcin mikołajczak
fe776d8b33 Fix GenerateUnsetUserKeys migration
Signed-off-by: marcin mikołajczak <git@mkljczk.pl>
2023-11-13 14:18:22 +01:00
lain
752bc168f6 Merge branch 'quotes-count' into 'develop'
Count and display post quotes

See merge request pleroma/pleroma!3956
2023-11-12 13:38:09 +00:00
marcin mikołajczak
9a063deacc Count and display post quotes 2023-11-12 13:38:08 +00:00
Mark Felder
bf426c53b4 Fix digest email processing, consolidate Oban queues
The email related jobs can all share a single Oban queue
2023-11-07 15:14:36 -05:00
Haelwenn
043a00991d Merge branch 'instance-nodeinfo-metadata' into 'develop'
instances: Store some metadata based on NodeInfo

See merge request pleroma/pleroma!3853
2023-06-27 18:58:04 +00:00
Haelwenn
41f2ee69a8 Merge branch 'from/upstream-develop/tusooa/backup-status' into 'develop'
Detail backup states

Closes #3024

See merge request pleroma/pleroma!3809
2023-06-27 12:08:11 +00:00
Lain Soykaf
22878cf84a B Migrations: Don't remove activity_visibility_index for now. 2023-06-11 16:22:16 +04:00
Lain Soykaf
175ee9e6fe Merge branch 'develop' of git.pleroma.social:pleroma/pleroma into unused_indexes 2023-06-11 16:20:30 +04:00
Haelwenn (lanodan) Monnier
fcd49e3985 User: Remove ap_enabled field 2023-05-05 11:11:26 +02:00
Mark Felder
d91a823836 Remove unused indexes
These indexes were always listed as unused and several grow quite large.
The most significant impact is the activities_visibility_index which takes many hours to rebuild when restoring the server from backup even on fast hardware.
2023-04-22 16:43:33 +00:00
Haelwenn (lanodan) Monnier
9a2523a09a instances: Store some metadata based on NodeInfo 2023-03-16 09:02:20 +01:00
lain
e853cfe7c3 Revert "Merge branch 'copyright-bump' into 'develop'"
This reverts merge request !3825
2023-01-02 20:38:50 +00:00
marcin mikołajczak
10886eeaa2 Bump copyright year
Signed-off-by: marcin mikołajczak <git@mkljczk.pl>
2023-01-01 12:13:06 +01:00
tusooa
e4ac2a7cd6
Detail backup states 2022-12-24 00:04:32 -05:00
Lain Soykaf
63d00f8123 Merge branch 'develop' of git.pleroma.social:pleroma/pleroma into update-deps 2022-12-15 17:19:36 -05:00
Mark Felder
8a3b450397 Add migration to remove Quack from ConfigDB 2022-11-11 12:36:34 -05:00
Mark Felder
7c64f705f6 Update to Phoenix 1.6 and chase dependencies
Also bump minimum Elixir to 1.11
2022-11-03 16:13:07 +00:00
Haelwenn
346c130ddc Merge branch 'fix/user-private-key-generation' into 'develop'
User: generate private keys on user creation

See merge request pleroma/pleroma!3737
2022-09-05 15:38:15 +00:00
Hélène
cfb1bc967f
Migrations: generate unset user keys
User keys are now generated on user creation instead of "when needed",
to prevent race conditions in federation and a few other issues. This
migration will generate keys missing for local users.
2022-09-05 03:51:35 +02:00
tusooa
20347898e2 Merge branch 'fix/federation-context-issues' into 'develop'
Fix reply context fixing (Pleroma replies to Misskey threads) and removal of context objects

See merge request pleroma/pleroma!3717
2022-09-04 18:43:36 +00:00
Haelwenn
299255b9bb Merge branch 'from/upstream-develop/tusooa/assoc-object-id' into 'develop'
Add function to calculate associated object id

Closes #2307

See merge request pleroma/pleroma!3692
2022-09-03 02:50:40 +00:00
Haelwenn
e40c221c31 Merge branch 'from/upstream-develop/tusooa/edits' into 'develop'
Editing

Closes #1429, #2859, and #2288

See merge request pleroma/pleroma!3678
2022-09-03 02:16:42 +00:00
Tusooa Zhu
f047088a93
Update thread visibility function 2022-08-20 21:06:12 -04:00
Tusooa Zhu
4e7ed563c0
Lint 2022-08-20 20:43:47 -04:00
Tusooa Zhu
3885ee182a
Switch to associated_object_id index 2022-08-20 20:43:46 -04:00
Tusooa Zhu
06678fb4ad
Add function to calculate associated object id 2022-08-20 20:43:40 -04:00
Hélène
88c1c76d3e
Migrations: delete contexts with BaseMigrator
Due to the lengthiness of this task, the migration has been adapted into
a BaseMigrator migration, running in the background instead.
2022-08-15 01:47:09 +02:00
Hélène
c559c240d1
Migrations: delete context objects
These objects represent from 30 to 70% of the rows on the objects table,
based on numbers from a few live instances (single-user, small, large.)

As those pseudo-objects prevent creating objects with those actual IDs,
deleting them is a better solution. This could have happened if an
object used another object's ID as its context.
2022-08-10 02:29:38 +02:00
Haelwenn
b2ba307f4d Merge branch 'from/upstream-develop/tusooa/2871-fix-local-public' into 'develop'
local only fixes

Closes #2871

See merge request pleroma/pleroma!3660
2022-08-02 05:39:50 +00:00
tusooa
301ce5bc62 Merge branch 'mute-expiration' into 'develop'
MastoAPI: Show mutes expiration date

See merge request pleroma/pleroma!3682
2022-07-23 00:34:15 +00:00
marcin mikołajczak
597f56b4c4 Use :utc_datetime
Signed-off-by: marcin mikołajczak <git@mkljczk.pl>
2022-07-16 16:28:22 +02:00
Haelwenn
29f4ab640b Merge branch 'from/upstream-develop/tusooa/report-too-long' into 'develop'
Fix long report notes giving errors on creation

See merge request pleroma/pleroma!3679
2022-07-04 05:21:20 +00:00
Tusooa Zhu
4edc867b87 Merge branch 'develop' into 'from/upstream-develop/tusooa/edits'
# Conflicts:
#   lib/pleroma/constants.ex
2022-07-03 22:24:57 +00:00
Haelwenn
eefa981e0a Merge branch 'indexing-hotspots' into 'develop'
Add index hotspots

See merge request pleroma/pleroma!3664
2022-07-03 22:01:37 +00:00
Haelwenn
de37583c49 Merge branch 'image_description_from_exif_data' into 'develop'
Use EXIF data of image for image description

See merge request pleroma/pleroma!3535
2022-07-03 21:14:25 +00:00
Haelwenn
6b937d1473 Merge branch 'from/upstream-develop/tusooa/server-announcements' into 'develop'
Server announcements (1st pass)

See merge request pleroma/pleroma!3643
2022-07-03 20:58:20 +00:00
Ilja
75ea766824 Migration failed when no value for Pleroma.Upload was set 2022-07-01 12:15:02 +02:00
Ilja
cc5686bbd6 Migration for exiftool filter
Rename to Exiftool.StripLocation
2022-07-01 12:15:02 +02:00
marcin mikołajczak
b0f83aea29 Store mutes expiration date
Signed-off-by: marcin mikołajczak <git@mkljczk.pl>
2022-06-16 20:38:37 +02:00
Tusooa Zhu
06a3998013
Create Update notifications 2022-06-05 15:02:25 -04:00
Tusooa Zhu
b096fbba1a
Fix long report notes giving errors on creation 2022-06-02 01:28:39 -04:00
Tusooa Zhu
f1722a9f4a
Make lint happy
Ref: fix-local-public
2022-05-09 18:53:32 -04:00
Tusooa Zhu
38444aa92a
Allow authenticated users to access local-only posts in MastoAPI
Ref: fix-local-public
2022-05-09 18:53:18 -04:00
Pete
aa51fd068b Add index hotspots
squash
2022-05-06 18:48:08 -07:00