Commit graph

203 commits

Author SHA1 Message Date
Nutomic
165b19e75c
Optimize federated language updates to avoid unnecessary db writes (#2786)
* Optimize federated language updates to avoid unnecessary db writes (fixes #2772)

* fix tests

* fix test, rename functions

---------

Co-authored-by: Dessalines <dessalines@users.noreply.github.com>
2023-03-30 11:03:13 -04:00
Nutomic
6f513793cb
Activitypub crate rewrite (#2782)
* update activitypub-federation crate to 0.4.0

* fixes

* apub compiles!

* everything compiling!

* almost done, federated follow failing

* some test fixes

* use release

* add code back in
2023-03-21 16:03:05 +01:00
Nutomic
d9e7f0100a
Dont upsert Instance row every apub fetch (#2771)
This is not necessary because the domain cant change, so we only
need to insert if no row exists for this domain.

Also fetch instance actor when parsing person, not only community

Co-authored-by: Dessalines <dessalines@users.noreply.github.com>
2023-02-28 21:36:57 -05:00
Nutomic
62663a9f2e
Implement separate mod activities for feature, lock post (#2716)
* Implement separate mod activities for feature, lock post

Also includes collection for featured posts. Later we also need
to do the same for Comment.distinguished

* some changes

---------

Co-authored-by: Dessalines <dessalines@users.noreply.github.com>
2023-02-18 09:50:28 -05:00
Dessalines
36aaa50644
Organize utils into separate files. Fixes #2295 (#2736)
* Organize utils into separate files. Fixes #2295

* Moving tests.

* Fix test.

* Fix test 2
2023-02-16 05:05:14 +01:00
Nutomic
3bb98fcc64
Fix post titles being cut off (fixes #2718) (#2720)
I didnt notice that this limit would also apply to Lemmy and other
software which has the title field. So gonna change it to be
identical to db limit
2023-02-10 13:35:23 -05:00
Kradyz
ab9b60e86d
Convert HTML from titles into markdown (#2709)
Co-authored-by: Kradyz <k@radiz.nl>
2023-02-07 03:27:17 +01:00
Nutomic
7c1a37d584
In apub post receive, check posting restricted to mods (ref #2702) (#2704)
Co-authored-by: Dessalines <dessalines@users.noreply.github.com>
2023-02-05 12:31:09 -05:00
Nutomic
7e3d3839b6
Post creation from Mastodon (fixes #2590) (#2651)
* Post creation from Mastodon (fixes #2590)

* better logic for page title

* add deserialize helper

Co-authored-by: Dessalines <dessalines@users.noreply.github.com>
2023-01-20 12:43:23 -05:00
Anon
9dfd819691
Add support for Featured Posts (#2585)
* Add support for Featured Posts

* Fix rebase

* More fixes
2022-12-12 11:17:10 +00:00
Nutomic
0ecf256ce3
Remove federation backwards compatibility with 0.16.x (#2183)
* Breaking: remove compatiblity with page.url field (ref #2182)

* Breaking: change type of Instance to `Application` (ref #2200)

Co-authored-by: Dessalines <dessalines@users.noreply.github.com>
2022-12-09 11:21:17 -05:00
Nutomic
2732a5bf07
Rework websocket (#2598)
* Merge websocket crate into api_common

* Add SendActivity trait so that api crates compile in parallel with lemmy_apub

* Rework websocket code

* fix websocket heartbeat
2022-12-09 10:31:47 -05:00
Felix Ableitner
c6c52ab9cc Add SendActivity trait so that api crates compile in parallel with lemmy_apub 2022-12-02 10:46:49 +01:00
Felix Ableitner
201fa97769 Move code to generate apub urls into lemmy_api_common 2022-12-02 10:46:49 +01:00
Felix Ableitner
030afbc2e7 Merge websocket crate into api_common 2022-12-02 10:46:49 +01:00
Nutomic
bcf5c91f81
Use audience field to federate items in groups (fixes #2464) (#2584) 2022-12-01 15:52:49 -05:00
Nutomic
41d4852efc
Federate group moderators using attributedTo field (#2588)
* Federate group moderators using attributedTo field

* fix tests

Co-authored-by: Dessalines <dessalines@users.noreply.github.com>
2022-11-25 09:01:58 -05:00
Nutomic
d20d2b9218
Implement federated user following (fixes #752) (#2577)
* Implement federated user following (fixes #752)

* rewrite send_activity_in_community and add docs, remove default for column pending

* improve migration

* replace null values in db migration
2022-11-23 18:40:47 -05:00
Nutomic
6f3bf4634b
Various pedantic clippy fixes (#2568)
* Various pedantic clippy fixes

* more clippy pedantic fixes

* try to fix ci

* add fix clippy script, use rust 1.65

* fix clippy
2022-11-19 04:33:54 +00:00
Nutomic
24756af84b
Live reload settings (fixes #2508) (#2543)
* Live reload rate limit settings (fixes #2508)

* fix tests
2022-11-16 14:06:22 -05:00
Nutomic
e3bb43542c
Mobilizon federation (#2544)
* Mobilizon federation

* Also accept document attachments from mobilizon
2022-11-12 08:51:08 -05:00
Dessalines
5d837780f5
Add diesel_async, get rid of blocking function (#2510)
* Moving settings to Database.

- Moves many settings into the database. Fixes #2285
- Adds a local_site and instance table. Fixes #2365 . Fixes #2368
- Separates SQL update an insert forms, to avoid runtime errors.
- Adds TypedBuilder to all the SQL forms, instead of default.

* Fix weird clippy issue.

* Removing extra lines.

* Some fixes from suggestions.

* Fixing apub tests.

* Using instance creation helper function.

* Move forms to their own line.

* Trying to fix local_site_data, still broken.

* Testing out async

* Testing out async 2

* Fixing federation tests.

* Trying to fix check features 1.

* Starting on adding diesel async. 1/4th done.

* Added async to views and schema.

* Adding some more async

* Compiling now.

* Added diesel async. Fixes #2465

* Running clippy --fix

* Trying to fix cargo test on drone.

* Trying new muslrust.

* Trying a custom dns

* Trying a custom dns 2

* Trying a custom dns 3

* Trying a custom dns 4

* Trying a custom dns 5

* Trying a custom dns 6

* Trying a custom dns 7

* Addressing PR comments.

* Adding check_apub to all verify functions.

* Reverting back drone.

* Fixing merge

* Fix docker images.

* Adding missing discussion_languages.

* Trying to fix federation tests.

* Fix site setup user creation.

* Fix clippy

* Fix clippy 2

* Test api faster

* Try to fix 1

* Try to fix 2

* What are these lines about

* Trying to fix 3

* Moving federation test back to top.

* Remove logging cat.
2022-11-09 10:05:00 +00:00
Nutomic
b5cd732372
Mark objects as not deleted when received via apub (fixes #2507) (#2528) 2022-11-03 09:39:30 -04:00
Nutomic
6aa9bdebae
Fix limit_languages to operate on correct instance (fixes #2496) (#2518)
* Fix limit_languages to operate on correct instance (fixes #2496)

* cargo fmt
2022-10-28 10:42:05 -04:00
Dessalines
235cc8b228
Moving settings to Database. (#2492)
* Moving settings to Database.

- Moves many settings into the database. Fixes #2285
- Adds a local_site and instance table. Fixes #2365 . Fixes #2368
- Separates SQL update an insert forms, to avoid runtime errors.
- Adds TypedBuilder to all the SQL forms, instead of default.

* Fix weird clippy issue.

* Removing extra lines.

* Some fixes from suggestions.

* Fixing apub tests.

* Using instance creation helper function.

* Move forms to their own line.

* Trying to fix local_site_data, still broken.

* Fixing federation tests.

* Trying to fix check features 1.

* Addressing PR comments.

* Adding check_apub to all verify functions.
2022-10-27 09:24:07 +00:00
Nutomic
2ef0f8f5f8
implement language tags for site/community in db and api (#2434)
* implement language tags for site/community in db and api

* add api checks for valid languages

* during db migration, update existing users, sites, communities to have all languages enabled

* init new users/communities with site languages (not all languages)

* federate site/community languages

* fix tests

* when updating site languages, limit community languages to this subset

also, when making a new post and subset of user lang, community lang
contains only one item, use that as post lang

* add tests for actor_language db functions

* include language list in siteview/communityview

* Fix some of the review comments

* Some more review changes

* Add todo about boxed query

* Add default_post_language to GetCommunityResponse
2022-10-06 14:27:58 -04:00
Dessalines
c9f1407429
Diesel 2.0.0 upgrade (#2452)
* Initial commit to bump diesel to 2.0.0-rc.0 and see what happens

* Add chrono feature from diesel

* db_schema crate is close to building?

* Upgrade diesel-derive-newtype

* Mostly modifying references to connections to be mutable ones; also used
new way to do migrations as suggested by the migration guide; a lot more
compiles now, though I can't figure out this tricky ToSql issue at the
moment

* Running clippy --fix

* Trying to fix drone clippy 1

* Fix clippy

* Upgrade clux-musl

* Trying to fix drone clippy 2

* Trying to fix drone clippy 3

* Trying to fix drone clippy 5

* Adding diesel table aliases, removing sql view hack. Fixes #2101

Co-authored-by: Steven Chu <stevenc1@gmail.com>
Co-authored-by: Nutomic <me@nutomic.com>
2022-09-26 14:09:32 +00:00
Felix Ableitner
3d0d8796ad Add language tags for comments 2022-09-02 10:49:54 -04:00
Nutomic
e4a49b6eab
Tag posts and comments with language (fixes #440) (#2269)
* Tag posts and comments with language (fixes #440)

* Untangle PostView tests

* Implement test for PostView language query

* Store languages directly in database

* finish moving languages into db, it compiles

* update post_view

* serde skip Language.id field

* add local_user_language table, other changes suggested in review

* add code for local_user_discussion_language_view

* Remove unnecessary clones in db view converteres

* Fixing up some table and join issues.

* Clearing the current languages.

* Fix formatting.

* update user languages in single transaction

* proper test for user language queries

* Some fixes for all / missing user languages. (#2404)

* Some fixes for all / missing user languages.

* Adding back in transaction.

* fix test

Co-authored-by: Dessalines <tyhou13@gmx.com>
Co-authored-by: Dessalines <dessalines@users.noreply.github.com>
2022-08-18 15:11:19 -04:00
Nutomic
c89006c94a
Rejected federated pm from blocked users (fixes #2398) (#2408) 2022-08-17 16:53:19 -04:00
Dessalines
870abf8442
Adding distinguish comment. Fixes #2002 (#2391)
* Adding distinguish comment. Fixes #2002

* Adding lemmy:distinguished
2022-08-17 11:38:52 +00:00
vpzomtrrfrt
08a797c986
Accept Image objects in attachments (#2394) 2022-08-16 10:12:43 +00:00
Dessalines
9c3efe32e7
First pass at adding comment trees. (#2362)
* First pass at adding comment trees.

- Extracted comment replies into its own table.
- Added ltree column to comment
- Added parent_id param to GetComments to fetch a tree branch
- No paging / limiting yet

* Adding child_count to comment_aggregates.

* Adding parent comment update counts

* Fix unit tests.

* Comment tree paging mostly done.

* Fix clippy

* Fix drone tests wrong postgres version.

* Fix unit tests.

* Add back in delete in unit test.

* Add postgres upgrade script.

* Fixing some PR comments.

* Move update ltree into Comment::create

* Updating based on comments.

* Fix send soft fail.
2022-07-30 05:55:59 +02:00
Dessalines
8bfeb4b627
Fix a few form options for diesel. Fixes #2287 (#2376)
* Fix a few form options for diesel. Fixes #2287

* Adding TODO comment.
2022-07-29 15:04:21 +02:00
Nutomic
63fff96275
Fix problem where actors can have empty public key (fixes #2347) (#2348) 2022-07-11 18:25:33 +00:00
Luna
e33736ae47
remove parking_lot (#2350) 2022-07-11 17:12:12 +00:00
Dessalines
a745fa6f43
Remove update and read site config. Fixes #2306 (#2329)
* Remove update and read site config. Fixes #2306

* Removing lazy_static, removing Settings::get()
2022-06-22 20:24:54 +00:00
Nutomic
fcaf7a084c
Upgrade activitypub_federation to 0.2.0, add setting federation.debug (#2300) 2022-06-08 11:45:39 -04:00
Nutomic
5387c262c1
Forbid outgoing requests in activitypub tests (fixes #2289) (#2294) 2022-06-03 11:31:22 -04:00
Nutomic
339eab01fd
Embed Peertube videos (#2261)
* Use og:video attribute for embeds, change Post.embed_html to embed_url

* fix clippy
2022-06-02 21:44:47 +00:00
Nutomic
a2a594b763
Extract Activitypub logic into separate library (#2288)
* Create example for apub lib

* some rewriting of apub lib

* Add LocalInstance struct for apub lib to avoid using Lemmy Settings

* Move ActorType trait to lemmy_apub, because its not needed in library

* Use reqwest_retry instead of custom impl, dont specify timeout on every send()

* Some improvements to example

* Moved inbox handling to library

* bug fixes

* Move context and serde helpers into library

* wip: example changes

* Add lemmy_utils feature to build only LemmyError

* Rename to activitypub_federation

* Remove lemmy_utils dep from activitypub_federation using generic error type

* Finish activitypub example

* Cleanup and fix tests

* Reorganize library files

* Remove ApubObject.to_tombstone()

* Extract activitypub library into separate git repository
2022-06-02 16:33:41 +02:00
Nutomic
f0ec3015cd
Remove check that avatars/banners are locally hosted (fixes #2254) (#2255) 2022-05-10 17:08:13 +00:00
Nutomic
7b86441bab
Federate with Peertube (#2244) 2022-05-06 23:53:33 +00:00
Nutomic
3aa3d75a1e
Add cargo feature for building lemmy_api_common with mininum deps (#2243) 2022-05-03 17:44:13 +00:00
Nutomic
7058cfa1cf
Implement restricted community (only mods can post) (fixes #187) (#2235)
* Implement restricted community (only mods can post) (fixes #187)

* review fixes

* fix tests
2022-04-28 20:32:32 +00:00
Nutomic
ae84258c41
Federation: dont overwrite local object from Announce activity (#2232)
* Federation: dont overwrite local object from Announce activity (fixes #2143)

* add missing form fields

* refactoring

* add ap_id, updated fields

* fix
2022-04-25 21:11:34 +00:00
Nutomic
bbd739af9b
Change Person, Instance types (#2200) 2022-04-12 17:10:35 +00:00
Nutomic
fed73a72c1
Write mod log for federated sticky/lock post actions (#2203) 2022-04-11 23:03:31 +00:00
Nutomic
f9d563d80a
Add missing mod log entries for federated actions (fixes #1489) (#2198) 2022-04-07 20:44:28 +00:00
Nutomic
bfae246734
Migrate towards using page.attachment field for url (ref #2144) (#2182) 2022-04-01 18:25:19 +00:00
Dessalines
4cf0da7b60 Clippy fixes. 2022-03-30 10:58:03 -04:00
Dessalines
f2a0841586
Rate limit websocket joins. (#2165)
* Rate limit websocket joins.

* Removing async on mutex lock fn.

* Removing redundant ip

* Return early if check fails.
2022-03-27 00:29:05 +00:00
Nutomic
dfb0938738
GNU social compatibility (#2100)
* Use SourceCompat everywhere (better compat with other software)

* Name field should not be mandatory in Group

* also check page.cc field for community id

* add gnu social tests

* better to use option<sourcecompat>

* update gnu social tests, marked vote as "unlisted"
2022-03-24 16:33:42 +00:00
Nutomic
8112816e99
If viewed actor isnt in db, fetch it from other instance (#2145) 2022-03-23 21:27:51 +00:00
Nutomic
bcf7ec6109
Forbid remote URLs for avatars/banners (fixes #1618) (#2132) 2022-03-18 15:46:58 +00:00
Nutomic
ef1e164cc5
Make activity queue worker count configurable, log stats (#2113) 2022-03-03 18:54:33 +00:00
Dessalines
a5433ebf77 Fixing unit tests. 2022-03-03 12:51:55 -05:00
Nutomic
762b85b27e
Reorganize federation tests (#2092)
* Reorganize apub protocol tests

* Reorder apub protocol struct members to clarify mandatory/optional fields
2022-02-17 22:04:01 +00:00
Nutomic
dd865c5af5
Implement instance actor (#1798)
* Implement instance actor

* wip: make site bans federate

* finish implementation and unit tests for federated bans

* start adding api tests

* fix api test

* remve site from GetCommunityResponse

* only federate site bans originating from user's home instance

* dont expose site.private_key in api
2022-02-07 19:23:12 +00:00
Nutomic
4a23ee4d8b
Dont make webfinger request when viewing community/user profile (fixes #1896) (#2049) 2022-01-27 16:39:22 +00:00
Nutomic
f23fed70bc
Add tombstone tests, better test errors (#2046) 2022-01-20 14:13:29 +00:00
Nutomic
eea3308906
Add tests for lotide federation, make lotide groups fetchable (#2035)
* Add tests for lotide federation, make lotide groups fetchable

* Accept posts using Note type (and better error messages for tests)
2022-01-17 09:40:47 -05:00
Dessalines
e65c45f152
Adding temporary bans. Fixes #1423 (#1999)
* Adding temporary bans. Fixes #1423

* Adding ban_expires to person

* Fix clippy

* Removing ban_expires from federated fields.

* Trying to add expires to blockcommunity apub.
2022-01-08 12:37:07 +00:00
Aode (lion)
97ebf2f6f3 Consolidate reqwest clients, use reqwest-middleware for tracing 2021-12-06 16:54:34 -06:00
Riley
35cbae61bc
Don't drop error context when adding a message to errors (#1958)
* Respond directly with LemmyError

Instrument Perform implementations for more precise traces
Use ApiError to format JSON errors when messages are present
Keep SpanTrace output in LemmyError Display impl

* Hide SpanTrace debug output from LemmyError

* Don't log when entering spans, only when leaving

* Update actix-web

* Update actix-rt

* Add newline after error info in LemmyError Display impl

* Propogate span information to blocking operations

* Instrument apub functions

* Use skip_all for more instrument attributes, don't skip 'self' in some api actions

* Make message a static string

* Send proper JSON over websocket

* Add 'message' to LemmyError display if present

* Use a quieter root span builder, don't pretty-print logs

* Keep passwords and emails out of logs

* Re-enable logging Login

* Instrument feeds

* Emit our own errors

* Move error log after status code recording

* Make Sensitive generic over the inner type

* Remove line that logged secrets
2021-12-06 09:54:47 -05:00
Riley
1579ee566f
background-jobs 0.11 (#1943) 2021-11-23 12:20:01 +00:00
Riley
3fea5645f8
Add tracing (#1942) 2021-11-23 12:16:47 +00:00
Dessalines
05d43150bc Making public key required. Fixes #1934 2021-11-22 10:10:18 -05:00
Felix Ableitner
afccd5cf69 Replace activitystreams crate with activitystreams-kinds 2021-11-22 14:28:57 +01:00
Felix Ableitner
da89ea22fb Fix Smithereen webfinger, remove duplicate webfinger impl (fixes #1916) 2021-11-18 19:28:53 +01:00
Aode (lion)
20cddf5e81 Support mastodon deletes 2021-11-16 18:07:47 +01:00
Felix Ableitner
7f4a773b88 Create and Note always need to tag parent creator, for mastodon notifications 2021-11-16 18:07:47 +01:00
Felix Ableitner
c1f1b8aa0f Activities in community should also be sent to actors in cc 2021-11-16 18:07:45 +01:00
Felix Ableitner
252d87d332 Fix clippy error 2021-11-08 13:16:14 +01:00
Felix Ableitner
888e683856 For verify_is_public() we also need to check cc field 2021-11-06 18:44:34 +01:00
Felix Ableitner
a5a674a270 Add method ApubObject.verify() 2021-11-06 18:35:14 +01:00
Felix Ableitner
02ce7bdc7d Remove unused ActorType methods 2021-11-06 15:01:00 +01:00
Felix Ableitner
8ea21c39b7 Reduce stack memory usage in apub code
- use our own, smaller Endpoints struct
- wrap ObjectId.url in Box
- adjust usage of Box in different places
2021-11-06 14:53:39 +01:00
Felix Ableitner
c725514841 Change to_apub and from_apub to take by value and avoid cloning 2021-11-06 14:47:58 +01:00
Felix Ableitner
2edf8ba157 Move ObjectId to library 2021-11-05 21:37:46 +01:00
Felix Ableitner
969a7f2d1b Refactoring apub code 2021-11-05 21:12:10 +01:00
Dessalines
a83113935d
Check if post or comment are deleted first. Fixes #1864 (#1867)
* Check if post or comment are deleted first. Fixes #1864

* Refactoring delete apub.

* Revert "Refactoring delete apub."

This reverts commit ba2c3d06cfb870efe792f4b2541036265b425156.
2021-11-03 17:47:24 +00:00
Felix Ableitner
41f7bcc0d2 Correctly use and document check_is_apub_id_valid() param use_strict_allowlist 2021-11-02 21:39:06 +01:00
Felix Ableitner
b396344eae Merge two functions into one 2021-11-02 21:39:06 +01:00
Felix Ableitner
1c6f74cad5 Convert note.content and chat_message.content to html (fixes #1871) 2021-11-02 21:39:02 +01:00
Felix Ableitner
0bde2d595e Add tests for parsing activities and collections 2021-11-02 12:44:51 +01:00
Felix Ableitner
aaaf039779 Move apub test files into tree structure 2021-10-29 16:54:19 +02:00
Felix Ableitner
5ff044346f Move object and collection structs to protocol folder 2021-10-29 12:50:32 +02:00
Felix Ableitner
d89156810d Move @context out of object/activity definitions 2021-10-29 12:50:32 +02:00
Felix Ableitner
271785b7fb When receiving activity, dont read community from cc (for pleroma compat and better verification) 2021-10-29 12:50:30 +02:00
Felix Ableitner
74523fb534 Make Note.context optional for pleroma compat (hack) 2021-10-29 12:45:54 +02:00
Felix Ableitner
6792e376b4 Rewrite community outbox to use new fetcher 2021-10-29 12:45:53 +02:00
Felix Ableitner
bb085189e0 Merge traits ToApub and FromApub into ApubObject 2021-10-29 12:45:53 +02:00
Nutomic
61189efe72
Rewrite collections to use new fetcher (#1861)
* Merge traits ToApub and FromApub into ApubObject

* Rewrite community outbox to use new fetcher

* Rewrite community moderators collection

* Rewrite tombstone
2021-10-27 12:03:07 -04:00
Nutomic
149a4e0de8
Breaking apub changes (#1859)
* Remove CommentInReplyToMigration

* Remove compat for RemovePostCommentOrCommunity

* Remove PublicUrlMigration

* Change type of pm to ChatMessage from Pleroma, make pm.to array

* Use person.summary instead of person.content for pleroma compat

* Also change group.content to summary

* Rewrite apub object test json to serve as nice examples

* Also add test case for parsing pleroma private message
2021-10-22 12:21:26 -04:00
Nutomic
dd0ba10b44
Pleroma federation2 (#1855)
* Allow fetching person from Pleroma, including test case (ref #1461)

* Added test case for parsing community from apub json

- fixed a bug with objectid (de)serialization
- fixed a bug with outbox fetching (ref #1582)

* Added apub test for post

* Ignore errors when reading community outbox (fixes #1582)

* Dont fetch community outbox/moderators during tests

* added test for lemmy comment

* Added federation test for pleroma comment

* Added html2md crate to parse comment html from pleroma (fixes #1461)

* some fixes for update_apub_test_files.sh

* Add tests for ToApub, private message, remove update script

* Delete objects from db at the end of each test
2021-10-21 13:25:35 -04:00
Felix Ableitner
1aa0e1997b Major refactor, adding newtypes for apub crate
- this allows moving FromApub/ToApub traits into apub lib
2021-10-20 12:28:09 +02:00
Felix Ableitner
f24999027e Merge crates db_schema and db_queries 2021-10-20 12:28:08 +02:00
Dessalines
2402515fcc
Dont allow posts to deleted / removed communities. Fixes #1827 (#1828)
* Dont allow posts to deleted / removed communities. Fixes #1827

* Fixing couldnt find community error.

* Adding check in createorupdate post and comment.

* make sure post wasn't deleted or removed.

* Adding a post not deleted or removed check to creatorupdatecomment.

* Using pub(crate)
2021-10-14 16:33:19 +00:00
Nutomic
b96ce81f89
Move code to apub library (#1795)
* Remove dependency of apub_lib on LemmyContext

* Move ApubObject trait to library

* Reorganize files in apub lib

* Move ActorType, signatures, activity_queue to apub library
2021-10-06 16:20:05 -04:00