Merge pull request 'Add logging to find bug (ref #1283)' (#146) from debug-1283 into main

Reviewed-on: https://yerbamate.ml/LemmyNet/lemmy/pulls/146
This commit is contained in:
dessalines 2020-11-30 18:49:11 +00:00
commit 9435994405
21 changed files with 113 additions and 88 deletions

View file

@ -1,4 +1,4 @@
# Contributing
See [here](https://dev.lemmy.ml/docs/contributing.html) for contributing Instructions.
See [here](https://lemmy.ml/docs/contributing.html) for contributing Instructions.

1
Cargo.lock generated
View file

@ -1761,6 +1761,7 @@ dependencies = [
"async-trait",
"awc",
"background-jobs",
"backtrace",
"base64 0.13.0",
"bcrypt",
"chrono",

View file

@ -20,7 +20,7 @@
<br />
<a href="https://join.lemmy.ml">Join Lemmy</a>
·
<a href="https://dev.lemmy.ml/docs/index.html">Documentation</a>
<a href="https://lemmy.ml/docs/index.html">Documentation</a>
·
<a href="https://github.com/LemmyNet/lemmy/issues">Report Bug</a>
·
@ -65,7 +65,7 @@ Each Lemmy server can set its own moderation policy; appointing site-wide admins
- Open source, [AGPL License](/LICENSE).
- Self hostable, easy to deploy.
- Comes with [Docker](https://dev.lemmy.ml/docs/administration_install_docker.html) and [Ansible](https://dev.lemmy.ml/docs/administration_install_ansible.html).
- Comes with [Docker](https://lemmy.ml/docs/administration_install_docker.html) and [Ansible](https://lemmy.ml/docs/administration_install_ansible.html).
- Clean, mobile-friendly interface.
- Only a minimum of a username and password is required to sign up!
- User avatar support.
@ -100,8 +100,8 @@ Each Lemmy server can set its own moderation policy; appointing site-wide admins
## Installation
- [Docker](https://dev.lemmy.ml/docs/administration_install_docker.html)
- [Ansible](https://dev.lemmy.ml/docs/administration_install_ansible.html)
- [Docker](https://lemmy.ml/docs/administration_install_docker.html)
- [Ansible](https://lemmy.ml/docs/administration_install_ansible.html)
## Lemmy Projects
@ -134,9 +134,9 @@ Lemmy is free, open-source software, meaning no advertising, monetizing, or vent
## Contributing
- [Contributing instructions](https://dev.lemmy.ml/docs/contributing.html)
- [Docker Development](https://dev.lemmy.ml/docs/contributing_docker_development.html)
- [Local Development](https://dev.lemmy.ml/docs/contributing_local_development.html)
- [Contributing instructions](https://lemmy.ml/docs/contributing.html)
- [Docker Development](https://lemmy.ml/docs/contributing_docker_development.html)
- [Local Development](https://lemmy.ml/docs/contributing_local_development.html)
### Translations

View file

@ -20,7 +20,7 @@ Here are some of the bigger changes:
- The first **federation public beta release**, woohoo :fireworks:
- All Lemmy functionality now works over ActivityPub (except turning remote users into mods/admins)
- Instance allowlist and blocklist
- Documentation for [admins](https://dev.lemmy.ml/docs/administration_federation.html) and [devs](https://dev.lemmy.ml/docs/contributing_federation_overview.html) on how federation works
- Documentation for [admins](https://lemmy.ml/docs/administration_federation.html) and [devs](https://lemmy.ml/docs/contributing_federation_overview.html) on how federation works
- Upgraded to newest versions of @asonix activitypub libraries
- Full local federation setup for manual testing
- Automated testing for nearly every federation action
@ -54,18 +54,18 @@ Here are some of the bigger changes:
## Contributors
We'd also like to thank both the [NLnet foundation](https://nlnet.nl/) for their support in allowing us to work full-time on Lemmy ( as well as their support for [other important open-source projects](https://nlnet.nl/project/current.html) ), [those who sponsor us](https://dev.lemmy.ml/sponsors), and those who [help translate Lemmy](https://weblate.yerbamate.ml/projects/lemmy/). Every little bit does help. We remain committed to never allowing advertisements, monetizing, or venture-capital in Lemmy; software should be communal, and should benefit humanity, not a small group of company owners.
We'd also like to thank both the [NLnet foundation](https://nlnet.nl/) for their support in allowing us to work full-time on Lemmy ( as well as their support for [other important open-source projects](https://nlnet.nl/project/current.html) ), [those who sponsor us](https://lemmy.ml/sponsors), and those who [help translate Lemmy](https://weblate.yerbamate.ml/projects/lemmy/). Every little bit does help. We remain committed to never allowing advertisements, monetizing, or venture-capital in Lemmy; software should be communal, and should benefit humanity, not a small group of company owners.
## Upgrading
- [with manual Docker installation](https://dev.lemmy.ml/docs/administration_install_docker.html#updating)
- [with Ansible installation](https://dev.lemmy.ml/docs/administration_install_ansible.html)
- [with manual Docker installation](https://lemmy.ml/docs/administration_install_docker.html#updating)
- [with Ansible installation](https://lemmy.ml/docs/administration_install_ansible.html)
## Testing Federation
Federation is finally ready in Lemmy, pending possible bugs or other issues. So for now we suggest to enable federation only on test servers, or try it on our own test servers ( [enterprise](https://enterprise.lemmy.ml/), [ds9](https://ds9.lemmy.ml/), [voyager](https://voyager.lemmy.ml/) ).
If everything goes well, after a few weeks we will enable federation on dev.lemmy.ml, at first with a limited number of trusted instances. We will also likely change the domain to https://lemmy.ml . Keep in mind that changing domains after turning on federation will break things.
If everything goes well, after a few weeks we will enable federation on lemmy.ml, at first with a limited number of trusted instances. We will also likely change the domain to https://lemmy.ml . Keep in mind that changing domains after turning on federation will break things.
To enable on your instance, edit your [lemmy.hjson](https://github.com/LemmyNet/lemmy/blob/main/config/defaults.hjson#L60) federation section to `enabled: true`, and restart.
@ -147,7 +147,7 @@ Overall, since our last major release in January (v0.6.0), we have closed over
Before starting the upgrade, make sure that you have a working backup of your
database and image files. See our
[documentation](https://dev.lemmy.ml/docs/administration_backup_and_restore.html)
[documentation](https://lemmy.ml/docs/administration_backup_and_restore.html)
for backup instructions.
**With Ansible:**
@ -203,4 +203,4 @@ This is the biggest release by far:
Another major announcement is that Lemmy now has another lead developer besides me, [@felix@radical.town](https://radical.town/@felix). Theyve created a better documentation system, implemented RSS feeds, simplified docker and project configs, upgraded actix, working on federation, a whole lot else.
https://dev.lemmy.ml
https://lemmy.ml

View file

@ -1,6 +1,6 @@
{
# for more info about the config, check out the documentation
# https://dev.lemmy.ml/docs/administration_configuration.html
# https://lemmy.ml/docs/administration_configuration.html
# settings related to the postgresql database
database: {
@ -9,7 +9,7 @@
# host where postgres is running
host: "postgres"
}
# the domain name of your instance (eg "dev.lemmy.ml")
# the domain name of your instance (eg "lemmy.ml")
hostname: "{{ domain }}"
# json web token for authorization between server and client
jwt_secret: "{{ jwt_password }}"
@ -26,7 +26,7 @@
# whether to enable activitypub federation.
enabled: false
# Allows and blocks are described here:
# https://dev.lemmy.ml/docs/administration_federation.html#instance-allowlist-and-blocklist
# https://lemmy.ml/docs/administration_federation.html#instance-allowlist-and-blocklist
#
# comma separated list of instances with which federation is allowed
# allowed_instances: ""

View file

@ -25,7 +25,7 @@
# maximum number of active sql connections
pool_size: 5
}
# the domain name of your instance (eg "dev.lemmy.ml")
# the domain name of your instance (eg "lemmy.ml")
hostname: null
# address where lemmy should listen for incoming requests
bind: "0.0.0.0"
@ -65,7 +65,7 @@
# whether to enable activitypub federation.
enabled: false
# Allows and blocks are described here:
# https://dev.lemmy.ml/docs/administration_federation.html#instance-allowlist-and-blocklist
# https://lemmy.ml/docs/administration_federation.html#instance-allowlist-and-blocklist
#
# comma separated list of instances with which federation is allowed
allowed_instances: ""

View file

@ -1,6 +1,6 @@
{
# for more info about the config, check out the documentation
# https://dev.lemmy.ml/docs/administration_configuration.html
# https://lemmy.ml/docs/administration_configuration.html
setup: {
# username for the admin user
@ -11,7 +11,7 @@
site_name: "lemmy-test"
}
# the domain name of your instance (eg "dev.lemmy.ml")
# the domain name of your instance (eg "lemmy.ml")
hostname: "my_domain"
# address where lemmy should listen for incoming requests
bind: "0.0.0.0"

View file

@ -1,16 +1,17 @@
# Trending / Hot / Best Sorting algorithm
## Goals
An expected feature in link aggregators is a kind of "Trending" sort which shows users a mixture of new posts / comments and popular ones, making for a display order which highlights the most currently active parts of the site / thread. This keeps the experience fresh and makes sure the site stays moving. Various flaws can be found in the ways that popular link aggregators like Reddit have implemented "Hot" or "Trending" sorts, so Lemmy has its own algorithm.
## Goals and Considerations
- During the day, new posts and comments should be near the top, so they can be voted on.
- After a day or so, the time factor should go away.
- Use a log scale, since votes tend to snowball, and so the first 10 votes are just as important as the next hundred.
## Reddit Sorting
[Reddit's comment sorting algorithm](https://medium.com/hacking-and-gonzo/how-reddit-ranking-algorithms-work-ef111e33d0d9), the wilson confidence sort, is inadequate, because it completely ignores time. What ends up happening, especially in smaller subreddits, is that the early comments end up getting upvoted, and newer comments stay at the bottom, never to be seen. Research showed that nearly all top comments are just the [first ones posted.](https://minimaxir.com/2016/11/first-comment/)
| Reddit | Hacker News | Lemmy |
|-|-|-|
| Does not take the lifetime of the thread into account, [giving early comments an overwhelming advantage over later ones,](https://minimaxir.com/2016/11/first-comment/) with the effect being even worse in small communities. New comments pool at the bottom of the thread, effectively killing off discussion and making each thread a race to comment early. This lowers the quality of conversation and rewards comments that are repetitive and spammy. | While far superior to Reddit's implementation for its decay of scores over time, [Hacker News' ranking algorithm](https://medium.com/hacking-and-gonzo/how-hacker-news-ranking-algorithm-works-1d9b0cf2c08d) does not use a logarithmic scale for scores. | Counterbalances the snowballing effect of votes over time with a logarithmic scale. Negates the inherent advantage of early comments while still ensuring that votes still matter in the long-term, not nuking older popular comments. |
## Hacker News Sorting
The [Hacker New's ranking algorithm](https://medium.com/hacking-and-gonzo/how-hacker-news-ranking-algorithm-works-1d9b0cf2c08d) is great, but it doesn't use a log scale for the scores.
## My Algorithm
## Additional Details
```
Rank = ScaleFactor * log(Max(1, 3 + Score)) / (Time + 2)^Gravity

View file

@ -6,17 +6,17 @@ To enable federation, change the setting `federation.enabled` to `true` in `lemm
Federation does not start automatically, but needs to be triggered manually through the search. To do this you have to enter a reference to a remote object, such as:
- `!main@dev.lemmy.ml` (Community)
- `@nutomic@dev.lemmy.ml` (User)
- `https://dev.lemmy.ml/c/programming` (Community)
- `https://dev.lemmy.ml/u/nutomic` (User)
- `https://dev.lemmy.ml/post/123` (Post)
- `!main@lemmy.ml` (Community)
- `@nutomic@lemmy.ml` (User)
- `https://lemmy.ml/c/programming` (Community)
- `https://lemmy.ml/u/nutomic` (User)
- `https://lemmy.ml/post/123` (Post)
For an overview of how federation in Lemmy works on a technical level, check out our [Federation Overview](contributing_federation_overview.md).
## Instance allowlist and blocklist
The federation section of Lemmy's config has two variables `allowed_instances` and `blocked_instances`. These control which other instances Lemmy will federate with. Both settings take a comma separated list of domains, eg `dev.lemmy.ml, example.com`. You can either change those settings via `/admin`, or directly on the server filesystem.
The federation section of Lemmy's config has two variables `allowed_instances` and `blocked_instances`. These control which other instances Lemmy will federate with. Both settings take a comma separated list of domains, eg `lemmy.ml,example.com`. You can either change those settings via `/admin`, or directly on the server filesystem.
It is important to note that these settings only affect sending and receiving of data between instances. If allow federation with a certain instance, and then remove it from the allowlist, this will not affect previously federated data. These communities, users, posts and comments will still be shown. They will just not be updated anymore. And even if an instance is blocked, it can still fetch and display public data from your instance.

View file

@ -96,7 +96,7 @@ Receives activities from user: `Follow`, `Undo/Follow`, `Create`, `Update`, `Lik
|---|---|---|
| `preferredUsername` | yes | Name of the actor |
| `name` | yes | Title of the community |
| `category` | yes | Hardcoded list of categories, see https://dev.lemmy.ml/api/v1/categories |
| `category` | yes | Hardcoded list of categories, see https://lemmy.ml/api/v1/categories |
| `sensitive` | yes | True indicates that all posts in the community are nsfw |
| `attributedTo` | yes | First the community creator, then all the remaining moderators |
| `content` | no | Text for the community sidebar, usually containing a description and rules |
@ -689,4 +689,4 @@ Sent to: User
"to": "https://enterprise.lemmy.ml/u/riker/inbox",
"object": ...
}
```
```

View file

@ -11,11 +11,11 @@ This section describes all the aspects of Lemmy where the council has decision m
- Priorities / Emphasis
- Controversial features (For example, an unpopular feature should be removed)
- Moderation and conflict resolution on:
- [dev.lemmy.ml](https://dev.lemmy.ml/)
- [lemmy.ml](https://lemmy.ml/)
- [github.com/LemmyNet/lemmy](https://github.com/LemmyNet/lemmy)
- [yerbamate.ml/LemmyNet/lemmy](https://yerbamate.ml/LemmyNet/lemmy)
- [weblate.yerbamate.ml/projects/lemmy/](https://weblate.yerbamate.ml/projects/lemmy/)
- Technical administration of dev.lemmy.ml
- Technical administration of lemmy.ml
- Official Lemmy accounts
- [Mastodon](https://mastodon.social/@LemmyDev)
- [Liberapay](https://liberapay.com/Lemmy/)
@ -67,13 +67,13 @@ stay open for at least two days.
## 7. Communication
- A private Matrix chat for all council members.
- (Once private communities are done) A private community on dev.lemmy.ml for issues.
- (Once private communities are done) A private community on lemmy.ml for issues.
## 8. Member List / Contact Info
General Contact [@LemmyDev Mastodon](https://mastodon.social/@LemmyDev)
- [Dessalines](https://dev.lemmy.ml/u/dessalines)
- [Nutomic](https://dev.lemmy.ml/u/nutomic)
- [AgreeableLandscape](https://dev.lemmy.ml/u/AgreeableLandscape)
- [fruechtchen](https://dev.lemmy.ml/u/fruechtchen)
- [kixiQu](https://dev.lemmy.ml/u/kixiQu)
- [Dessalines](https://lemmy.ml/u/dessalines)
- [Nutomic](https://lemmy.ml/u/nutomic)
- [AgreeableLandscape](https://lemmy.ml/u/AgreeableLandscape)
- [fruechtchen](https://lemmy.ml/u/fruechtchen)
- [kixiQu](https://lemmy.ml/u/kixiQu)

View file

@ -247,7 +247,7 @@ impl Perform for GetPosts {
let community_name = data.community_name.to_owned();
let posts = match blocking(context.pool(), move |conn| {
PostQueryBuilder::create(conn)
.listing_type(type_)
.listing_type(&type_)
.sort(&sort)
.show_nsfw(show_nsfw)
.for_community_id(community_id)

View file

@ -333,8 +333,6 @@ impl Perform for Search {
) -> Result<SearchResponse, LemmyError> {
let data: &Search = &self;
dbg!(&data);
match search_by_apub_id(&data.q, context).await {
Ok(r) => return Ok(r),
Err(e) => debug!("Failed to resolve search query as activitypub ID: {}", e),

View file

@ -46,3 +46,4 @@ anyhow = "1.0"
thiserror = "1.0"
background-jobs = " 0.8"
reqwest = { version = "0.10", features = ["json"] }
backtrace = "0.3"

View file

@ -48,16 +48,14 @@ pub(in crate::objects) fn check_object_domain<T, Kind>(
where
T: Base + AsBase<Kind>,
{
let actor_id = if let Some(url) = expected_domain {
check_is_apub_id_valid(&url)?;
let object_id = if let Some(url) = expected_domain {
let domain = url.domain().context(location_info!())?;
apub.id(domain)?.context(location_info!())?
} else {
let actor_id = apub.id_unchecked().context(location_info!())?;
check_is_apub_id_valid(&actor_id)?;
actor_id
apub.id_unchecked().context(location_info!())?
};
Ok(actor_id.to_string())
check_is_apub_id_valid(&object_id)?;
Ok(object_id.to_string())
}
pub(in crate::objects) fn set_content_and_source<T, Kind1, Kind2>(

View file

@ -17,6 +17,7 @@ use activitystreams::{
};
use activitystreams_ext::Ext1;
use anyhow::Context;
use backtrace::Backtrace;
use lemmy_db::{
community::Community,
post::{Post, PostForm},
@ -32,6 +33,7 @@ use lemmy_utils::{
LemmyError,
};
use lemmy_websocket::LemmyContext;
use log::error;
use url::Url;
#[async_trait::async_trait(?Send)]
@ -130,6 +132,15 @@ impl FromApub for PostForm {
let community =
get_or_fetch_and_upsert_community(community_actor_id, context, request_counter).await?;
if community.local && creator.local {
let page_id = page.id_unchecked().context(location_info!())?;
let bt = Backtrace::new();
error!(
"Lemmy is parsing a local post as remote, page id: {}, stack trace: {:?}",
page_id, bt
);
}
let thumbnail_url = match &page.inner.image() {
Some(any_image) => Image::from_any_base(
any_image

View file

@ -156,7 +156,7 @@ pub enum SortType {
TopAll,
}
#[derive(EnumString, ToString, Debug, Serialize, Deserialize)]
#[derive(EnumString, ToString, Debug, Serialize, Deserialize, Clone)]
pub enum ListingType {
All,
Local,

View file

@ -160,7 +160,7 @@ pub struct PostView {
pub struct PostQueryBuilder<'a> {
conn: &'a PgConnection,
query: BoxedQuery<'a, Pg>,
listing_type: ListingType,
listing_type: &'a ListingType,
sort: &'a SortType,
my_user_id: Option<i32>,
for_creator_id: Option<i32>,
@ -184,7 +184,7 @@ impl<'a> PostQueryBuilder<'a> {
PostQueryBuilder {
conn,
query,
listing_type: ListingType::All,
listing_type: &ListingType::All,
sort: &SortType::Hot,
my_user_id: None,
for_creator_id: None,
@ -200,7 +200,7 @@ impl<'a> PostQueryBuilder<'a> {
}
}
pub fn listing_type(mut self, listing_type: ListingType) -> Self {
pub fn listing_type(mut self, listing_type: &'a ListingType) -> Self {
self.listing_type = listing_type;
self
}
@ -497,7 +497,7 @@ mod tests {
};
let read_post_listings_with_user = PostQueryBuilder::create(&conn)
.listing_type(ListingType::Community)
.listing_type(&ListingType::Community)
.sort(&SortType::New)
.for_community_id(inserted_community.id)
.my_user_id(inserted_user.id)
@ -505,7 +505,7 @@ mod tests {
.unwrap();
let read_post_listings_no_user = PostQueryBuilder::create(&conn)
.listing_type(ListingType::Community)
.listing_type(&ListingType::Community)
.sort(&SortType::New)
.for_community_id(inserted_community.id)
.list()

View file

@ -172,7 +172,7 @@ impl Settings {
}
}
/// Returns something like `http://localhost` or `https://dev.lemmy.ml`,
/// Returns something like `http://localhost` or `https://lemmy.ml`,
/// with the correct protocol and hostname.
pub fn get_protocol_and_hostname(&self) -> String {
format!("{}://{}", self.get_protocol_string(), self.hostname)

View file

@ -4,9 +4,9 @@ set -e
declare -a arr=(
"https://mastodon.social/"
"https://peertube.social/"
"https://dev.lemmy.ml/"
"https://dev.lemmy.ml/feeds/all.xml"
"https://dev.lemmy.ml/.well-known/nodeinfo"
"https://lemmy.ml/"
"https://lemmy.ml/feeds/all.xml"
"https://lemmy.ml/.well-known/nodeinfo"
"https://fediverse.blog/.well-known/nodeinfo"
"https://torrents-csv.ml/service/search?q=wheel&page=1&type_=torrent"
)

View file

@ -42,7 +42,8 @@ enum RequestType {
pub fn config(cfg: &mut web::ServiceConfig) {
cfg
.route("/feeds/{type}/{name}.xml", web::get().to(get_feed))
.route("/feeds/all.xml", web::get().to(get_all_feed));
.route("/feeds/all.xml", web::get().to(get_all_feed))
.route("/feeds/local.xml", web::get().to(get_local_feed));
}
lazy_static! {
@ -61,34 +62,43 @@ async fn get_all_feed(
context: web::Data<LemmyContext>,
) -> Result<HttpResponse, Error> {
let sort_type = get_sort_type(info).map_err(ErrorBadRequest)?;
let rss = blocking(context.pool(), move |conn| {
get_feed_all_data(conn, &sort_type)
})
.await?
.map_err(ErrorBadRequest)?;
Ok(
HttpResponse::Ok()
.content_type("application/rss+xml")
.body(rss),
)
Ok(get_feed_data(&context, ListingType::All, sort_type).await?)
}
fn get_feed_all_data(conn: &PgConnection, sort_type: &SortType) -> Result<String, LemmyError> {
let site_view = SiteView::read(&conn)?;
async fn get_local_feed(
info: web::Query<Params>,
context: web::Data<LemmyContext>,
) -> Result<HttpResponse, Error> {
let sort_type = get_sort_type(info).map_err(ErrorBadRequest)?;
Ok(get_feed_data(&context, ListingType::Local, sort_type).await?)
}
let posts = PostQueryBuilder::create(&conn)
.listing_type(ListingType::All)
.sort(sort_type)
.list()?;
async fn get_feed_data(
context: &LemmyContext,
listing_type: ListingType,
sort_type: SortType,
) -> Result<HttpResponse, LemmyError> {
let site_view = blocking(context.pool(), move |conn| SiteView::read(&conn)).await??;
let listing_type_ = listing_type.clone();
let posts = blocking(context.pool(), move |conn| {
PostQueryBuilder::create(&conn)
.listing_type(&listing_type_)
.sort(&sort_type)
.list()
})
.await??;
let items = create_post_items(posts)?;
let mut channel_builder = ChannelBuilder::default();
channel_builder
.namespaces(RSS_NAMESPACE.to_owned())
.title(&format!("{} - All", site_view.name))
.title(&format!(
"{} - {}",
site_view.name,
listing_type.to_string()
))
.link(Settings::get().get_protocol_and_hostname())
.items(items);
@ -96,7 +106,12 @@ fn get_feed_all_data(conn: &PgConnection, sort_type: &SortType) -> Result<String
channel_builder.description(&site_desc);
}
Ok(channel_builder.build().map_err(|e| anyhow!(e))?.to_string())
let rss = channel_builder.build().map_err(|e| anyhow!(e))?.to_string();
Ok(
HttpResponse::Ok()
.content_type("application/rss+xml")
.body(rss),
)
}
async fn get_feed(
@ -150,7 +165,7 @@ fn get_feed_user(
let user_url = user.get_profile_url(&Settings::get().hostname);
let posts = PostQueryBuilder::create(&conn)
.listing_type(ListingType::All)
.listing_type(&ListingType::All)
.sort(sort_type)
.for_creator_id(user.id)
.list()?;
@ -176,7 +191,7 @@ fn get_feed_community(
let community = Community::read_from_name(&conn, &community_name)?;
let posts = PostQueryBuilder::create(&conn)
.listing_type(ListingType::All)
.listing_type(&ListingType::All)
.sort(sort_type)
.for_community_id(community.id)
.list()?;
@ -206,7 +221,7 @@ fn get_feed_front(
let user_id = Claims::decode(&jwt)?.claims.id;
let posts = PostQueryBuilder::create(&conn)
.listing_type(ListingType::Subscribed)
.listing_type(&ListingType::Subscribed)
.sort(sort_type)
.my_user_id(user_id)
.list()?;