mirrors/pleroma
1
0
Fork 0
mirror of https://git.pleroma.social/pleroma/pleroma.git synced 2025-04-26 20:54:46 +00:00
Code Issues Projects Releases Wiki Activity
16932 commits 274 branches 86 tags 379 MiB
Commit graph

201 commits

Author SHA1 Message Date
Mark Felder
568819c08a WebPush refactoring: separate build and deliver steps 2024-06-11 17:58:02 -04:00
Mark Felder
dcc50da400 Stream the notifications as part of the job 2024-06-08 22:40:08 -04:00
Mark Felder
0847d9ebaf Oban queue simplification 2024-05-27 13:48:17 -04:00
Mark Felder
ede414094f RichMedia refactor 
Rich Media parsing was previously handled on-demand with a 2 second HTTP request timeout and retained only in Cachex. Every time a Pleroma instance is restarted it will have to request and parse the data for each status with a URL detected. When fetching a batch of statuses they were processed in parallel to attempt to keep the maximum latency at 2 seconds, but often resulted in a timeline appearing to hang during loading due to a URL that could not be successfully reached. URLs which had images links that expire (Amazon AWS) were parsed and inserted with a TTL to ensure the image link would not break.

Rich Media data is now cached in the database and fetched asynchronously. Cachex is used as a read-through cache. When the data becomes available we stream an update to the clients. If the result is returned quickly the experience is almost seamless. Activities were already processed for their Rich Media data during ingestion to warm the cache, so users should not normally encounter the asynchronous loading of the Rich Media data.

Implementation notes:

- The async worker is a Task with a globally unique process name to prevent duplicate processing of the same URL
- The Task will attempt to fetch the data 3 times with increasing sleep time between attempts
- The HTTP request obeys the default HTTP request timeout value instead of 2 seconds
- URLs that cannot be successfully parsed due to an unexpected error receives a negative cache entry for 15 minutes
- URLs that fail with an expected error will receive a negative cache with no TTL
- Activities that have no detected URLs insert a nil value in the Cachex :scrubber_cache so we do not repeat parsing the object content with Floki every time the activity is rendered
- Expiring image URLs are handled with an Oban job
- There is no automatic cleanup of the Rich Media data in the database, but it is safe to delete at any time
- The post draft/preview feature makes the URL processing synchronous so the rendered post preview will have an accurate rendering

Overall performance of timelines and creating new posts which contain URLs is greatly improved.
 2024-05-07 19:54:56 -04:00
Haelwenn (lanodan) Monnier
a299ddb10e
ReceiverWorker: Make sure non-{:ok, _} is returned as {:error, …} 
Otherwise an error like `{:signature, {:error, {:error, :not_found}}}` ends up considered a success.
 2024-04-17 07:43:47 +02:00
marcin mikołajczak
b7c625db0f Merge remote-tracking branch 'origin/develop' into link-verification 2024-03-07 13:13:49 +01:00
Haelwenn (lanodan) Monnier
6af6a9704d
RemoteFetcherWorker: Make sure {:error, _} is returned on failure 
Otherwise jobs are considered a success.
 2024-02-23 01:59:30 +01:00
Mark Felder
91c83a82a0 Fix atom leak in background worker 
The only permitted values are "blocks_import", "follow_import", "mutes_import" of which we already have the equivalent atoms defined.
 2024-02-14 18:03:54 -05:00
Haelwenn
3c65a2899d Merge branch 'handle_object_fetch_failures' into 'develop' 
Handle object fetch failures gracefully

See merge request pleroma/pleroma!4015
 2024-01-19 18:43:00 +00:00
Mark Felder
12c052551b Allow the Remote Fetcher to attempt fetching an unreachable instance 2024-01-14 14:05:43 -05:00
Mark Felder
f17f92105b Oban jobs should be discarded on permanent errors 2023-12-28 23:52:59 -05:00
Mark Felder
3acfdb6f8a Retire the Pleroma.Web.Federator.Publisher module 2023-12-28 14:53:09 -05:00
Mark Felder
ad0a5deb67 Prevent requeuing Remote Fetcher jobs that exceed thread depth 2023-12-27 22:28:41 -05:00
Mark Felder
a2708f7fe3 Leverage existing atoms as return errors for the object fetcher 2023-12-27 22:01:59 -05:00
Mark Felder
9c0040124a Skip remote fetch jobs for unreachable instances 2023-12-26 16:28:05 -05:00
Mark Felder
5f6966cd9f Remove mistaken duplicate fetch 2023-12-26 16:23:41 -05:00
Mark Felder
c6b38441f1 Cancel remote fetch jobs for deleted objects 2023-12-26 16:05:44 -05:00
Mark Felder
d472bafec1 Mark instances as unreachable when returning a 403 from an object fetch 
This is a definite sign the instance is blocked and they are enforcing authorized_fetch
 2023-12-26 15:54:21 -05:00
Mark Felder
223c1bac8d Cancel the job if the signature is still invalid after a refetch of the public key 2023-12-10 12:55:41 -05:00
Mark Felder
82724f6664 Do not retry fetching deleted objects 2023-12-09 17:48:33 -05:00
Mark Felder
4039106500 Fix the req_headers formatting 2023-12-08 21:51:36 -05:00
Mark Felder
074b31d9ab Optimistic Inbox 
Rework inbound federation to accept requests optimistically. The HTTP Signatures Plug will not attempt to fetch the actor or key and will fail early.

If the signature cannot be validated we pass the required data into the Oban job with a reduced priority and increase the timeout to 20 seconds. The Oban job will handle the actor and key fetching before attempting to validate the activity again. This job will be retried 5 times by default.

Another welcome side effect is that actors who change their keys can federate to Pleroma instances immediately instead of needing to wait the default value of 86400s / 24 hours before the key will be fetched again.
 2023-12-08 17:45:20 -05:00
Lain Soykaf
d3f8950588 B MeiliSearch, SearchIndexingWorker: Use Config.Getting, make tests async. 2023-11-12 14:49:50 +04:00
Lain Soykaf
0c5cc51983 Merge branch 'develop' of git.pleroma.social:pleroma/pleroma into pleroma-meilisearch 2023-11-12 13:53:18 +04:00
Mark Felder
bf426c53b4 Fix digest email processing, consolidate Oban queues 
The email related jobs can all share a single Oban queue
 2023-11-07 15:14:36 -05:00
marcin mikołajczak
50e7706b26 Verify link ownership with rel="me" 
Signed-off-by: marcin mikołajczak <git@mkljczk.pl>
 2023-11-02 13:30:46 +01:00
Haelwenn
41f2ee69a8 Merge branch 'from/upstream-develop/tusooa/backup-status' into 'develop' 
Detail backup states

Closes #3024

See merge request pleroma/pleroma!3809
 2023-06-27 12:08:11 +00:00
Haelwenn (lanodan) Monnier
e17265a7a2 TransmogrifierWorker: Remove obsolete worker 2023-05-05 11:11:26 +02:00
anemone
f463b7570e Set background worker timeout to 15 minutes 2023-03-23 23:14:52 -07:00
tusooa
1babd0798f
Stop oban from retrying if validating errors occur when processing incoming data 2023-03-01 21:03:30 -05:00
lain
e853cfe7c3 Revert "Merge branch 'copyright-bump' into 'develop'" 
This reverts merge request !3825
 2023-01-02 20:38:50 +00:00
marcin mikołajczak
10886eeaa2 Bump copyright year 
Signed-off-by: marcin mikołajczak <git@mkljczk.pl>
 2023-01-01 12:13:06 +01:00
tusooa
e4ac2a7cd6
Detail backup states 2022-12-24 00:04:32 -05:00
Ekaterina Vaartis
398141da68 Merge remote-tracking branch 'upstream/develop' into meilisearch 2022-12-20 21:00:07 +03:00
Mark Felder
8be7f87e1f Define sane Oban Worker timeouts 2022-11-11 13:46:27 -05:00
Ekaterina Vaartis
5ac6763238 Make add_to_index and remove_from_index report errors 2022-10-10 20:19:09 +03:00
Ekaterina Vaartis
d89dc5518b Fix meilisearch tests and jobs for oban 2022-10-10 20:19:09 +03:00
Ekaterina Vaartis
2bc21c6f18 Use oban for search indexing 2022-10-10 20:19:09 +03:00
Tusooa Zhu
d487e0160c
Treat containment failure as cancel in ReceiverWorker 2022-08-08 08:41:33 -04:00
Tusooa Zhu
a0166e92fa
Treat MRF rejects as success in Oban worker 2022-08-06 00:33:18 -04:00
Tusooa Zhu
221cb3fb81
Allow users to create backups without providing email address 
Ref: backup-without-email
 2022-05-07 00:23:55 -04:00
Sean King
17aa3644be
Copyright bump for 2022 2022-02-25 23:11:42 -07:00
lain
6e3df11693 Merge branch 'instance-deletion' into 'develop' 
AdminAPI: endpoint to delete all content from a remote instance

See merge request pleroma/pleroma!3483
 2021-11-15 14:24:13 +00:00
Alex Gleason
6a6e42c9bf
PollWorker defensive checks 2021-07-18 11:10:22 -05:00
Alex Gleason
0114754db2
MastodonAPI: Support poll notification 2021-07-17 22:19:38 -05:00
Alex Gleason
f67d00d12b
Add Instance.delete_users_and_activities/1 to delete all content from a remote instance 2021-07-17 14:06:35 -05:00
feld
008499f65a Merge branch 'develop' into 'fix/2412-filters' 
# Conflicts:
#   CHANGELOG.md
 2021-02-03 14:23:23 +00:00
feld
c3dd860a02 Merge branch 'fix/2449-scheduled-poll-bug' into 'develop' 
Fix for scheduled post with poll

Closes #2449

See merge request pleroma/pleroma!3294
 2021-02-03 14:22:23 +00:00
feld
6a2d3fb9a3 Merge branch 'fix/2473-purge-expired-activities-duplicates' into 'develop' 
fix and delete purge activities duplicates

Closes #2473

See merge request pleroma/pleroma!3285
 2021-02-02 17:22:23 +00:00
Alexander Strizhakov
0dc68c157f
fix for scheduled post with poll 2021-02-01 18:22:26 +03:00
Mark Felder
13d79c281f Make attachment cleanup jobs a noop if the setting is disabled. 2021-01-28 14:42:20 -06:00
Alexander Strizhakov
6c987c7670
fix and delete purge activities duplicates 2021-01-28 16:57:41 +03:00
Alexander Strizhakov
39335d4251
fix for unique oban worker option 2021-01-28 16:57:03 +03:00
Alexander Strizhakov
875fbaae35
support for expires_in/expires_at in filters 2021-01-26 08:27:45 +03:00
Mark Felder
ebd7d1365b Make setting user activation status explicit 2021-01-15 11:37:17 -06:00
Haelwenn (lanodan) Monnier
c4439c630f
Bump Copyright to 2021 
grep -rl '# Copyright © .* Pleroma' * | xargs sed -i 's;Copyright © .* Pleroma .*;Copyright © 2017-2021 Pleroma Authors <https://pleroma.social/>;'
 2021-01-13 07:49:50 +01:00
Mark Felder
86dcfb4eb9 More places we should be using Upload.base_url 2021-01-08 17:32:42 -06:00
Alexander Strizhakov
8d218ebaf5
Moving some background jobs into simple tasks 
- fetching activity data
- attachment prefetching
- using limiter to prevent overload
 2020-11-11 13:39:49 +03:00
lain
dd2b3a8da9 Merge branch 'develop' of git.pleroma.social:pleroma/pleroma into feature/expire-mutes 2020-11-04 16:51:42 +01:00
Egor Kislitsyn
ad605e3e16
Rename Pleroma.Backup to Pleroma.User.Backup 2020-10-20 17:16:58 +04:00
Egor Kislitsyn
7c22c9afb4
Allow admins request user backups 2020-10-07 18:34:31 +04:00
Egor Kislitsyn
7fdd81d000
Add "Your backup is ready" email 2020-10-07 18:34:30 +04:00
Egor Kislitsyn
739cb1463b
Add backups deletion 2020-10-07 18:34:29 +04:00
Egor Kislitsyn
a0ad9bd734
Add BackupWorker 2020-10-07 18:34:28 +04:00
Roman Chvanikov
8f5589cf66 Merge develop 2020-09-23 13:56:50 +03:00
Roman Chvanikov
28d0986f83 Refactor mutes removing in CommonAPI and User 2020-09-20 20:58:17 +03:00
Maksim Pechnikov
5a8ea0a5b0 small refactoring 2020-09-17 09:13:43 +03:00
Maksim Pechnikov
b867f9d7ae Merge branch 'develop' into issue/2099 2020-09-17 08:41:35 +03:00
Roman Chvanikov
c5830ac037 Merge develop 2020-09-13 12:24:57 +03:00
Roman Chvanikov
527afb813a Remove unused require 2020-09-13 12:23:45 +03:00
Alexander Strizhakov
15aece7238 remove validate_expires_at from enqueue method 2020-09-10 21:52:31 +03:00
Alexander Strizhakov
2c2094d4b2 configurable lifetime for ephemeral activities 2020-09-10 21:51:59 +03:00
Alexander Strizhakov
eb5ff715f7 pin/unpin for activities with expires_at option 2020-09-10 21:50:41 +03:00
Alexander Strizhakov
b3485a6dbf little clean up 2020-09-10 21:50:41 +03:00
Alexander Strizhakov
93e1c8df9d reject activity creation 
if passed expires_at option and expiring activities are not configured
 2020-09-10 21:50:41 +03:00
Alexander Strizhakov
4981b5a1a3 copyright header 2020-09-10 21:50:41 +03:00
Alexander Strizhakov
629a8de9cb deprecation warning 
changed namespace for activity expiration configuration
 2020-09-10 21:50:41 +03:00
Alexander Strizhakov
de4c935071 don't expire pinned posts 2020-09-10 21:50:41 +03:00
Alexander Strizhakov
9bf1065a06 schedule activity expiration in Oban 2020-09-10 21:50:40 +03:00
Alexander Strizhakov
7dd986a563
expire mfa tokens through Oban 2020-09-10 16:01:19 +03:00
Alexander Strizhakov
3ce658b930
schedule expired oauth tokens deletion with Oban 2020-09-10 16:01:19 +03:00
Roman Chvanikov
880a92ccc7 Merge branch 'develop' into feature/expire-mutes 2020-09-08 15:16:43 +03:00
Roman Chvanikov
e3f845b243 Add expiring mutes for activities 2020-09-08 15:13:50 +03:00
Roman Chvanikov
c56e3d4f3b Add expires_in param for account mutes 2020-09-08 13:26:44 +03:00
Alexander Strizhakov
ee67c98e55
removing Stats worker from Oban cron jobs 2020-09-07 19:16:14 +03:00
Maksim Pechnikov
5ae56aafb2 added import mutes 2020-09-06 21:42:51 +03:00
Maksim Pechnikov
9b055f7211 fix cron wroker 2020-08-13 21:03:15 +03:00
Maksim Pechnikov
3515cb5c3a fix Cron.PurgeExpiredActivitiesWorker 2020-08-13 21:01:21 +03:00
Maksim Pechnikov
a8d967762e migrate to oban 2.0-rc1 2020-06-23 15:09:01 +03:00
Maksim Pechnikov
b023110799 fixed a visibility of functions 2020-06-15 20:48:22 +03:00
Maksim Pechnikov
2e8a236cef fix invalidates media url's 2020-06-14 21:02:57 +03:00
Egor Kislitsyn
2069ec5006
Fix Oban warnings 2020-05-26 16:11:42 +04:00
Maksim Pechnikov
cb40602a16 added media proxy invalidation 2020-05-15 21:39:42 +03:00
Egor Kislitsyn
7803a85d2c
Add OpenAPI spec for StatusController 2020-05-13 00:25:21 +04:00
Egor Kislitsyn
6db52c3b36 Fix Oban warning 
Warning example:

     [warn] Expected Elixir.Pleroma.Workers.BackgroundWorker.perform/2 to return :ok, {:ok, value}, or {:error, reason}. Instead received:  [error: "not found @user@server.party", error: "not found "]  The job will be considered a success.
 2020-04-22 19:27:28 +04:00
rinpatch
e87a32bcd7 rip out fetch_initial_posts 
Every time someone tries to use it, it goes mad and tries to scrape the
entire fediverse for no visible reason, it's better to just remove it
than continue shipping it in it's current state.

idea acked by lain and feld on irc

Closes #1595 #1422
 2020-03-15 15:59:17 +03:00
lain
6f7a8c43a2 Merge branch 'fix/no-email-no-fail' into 'develop' 
Do not fail when user has no email

See merge request pleroma/pleroma!2249
 2020-03-04 12:43:06 +00:00
Mark Felder
05da5f5cca Update Copyrights 2020-03-03 16:44:49 -06:00
Haelwenn (lanodan) Monnier
6da6540036
Bump copyright years of files changed after 2020-01-07 
Done via the following command:
git diff fcd5dd259a --stat --name-only | xargs sed -i '/Pleroma Authors/c# Copyright © 2017-2020 Pleroma Authors <https:\/\/pleroma.social\/>'
 2020-03-02 06:08:45 +01:00
Egor Kislitsyn
cb60a9c42f
Do not fail when user has no email 2020-02-27 17:27:49 +04:00